Non-Date types that store Date values cannot be rendered as a time-based Chart

Expected behavior

In our case, we use Unix (Epoch) time stored as an Int64 on MongoDB for all our Date/Time fields like created_on, updated_on, etc. We expect these fields to be converted to Date objects by fores-express-mongoose. These Date objects should then be easily searchable, sort-able, and just generally something we can manipulate. Furthermore, when using these fields (created_on, updated_on) in the Dashboard view to create charts, it should work normally and function correctly.

Actual behavior

What actually happens is that we see some errors when using the fields created_on and updated_on when trying to create Charts in the Dashboard view. This error says [forest] 🌳🌳🌳 Unexpected error: can't convert from BSON type long to Date MongoError: can't convert from BSON type long to Date.

Failure Logs

Mongoose: order.aggregate([ { '$match': { created_on: { '$ne': null } } }, { '$group': { _id: { year: { '$year': [ { '$subtract': [ '$created_on', -18000000 ] } ] }, month: { '$month': [ { '$subtract': [ '$created_on', -18000000 ] } ] }, day: { '$dayOfMonth': [ { '$subtract': [ '$created_on', -18000000 ] } ] } }, created_on: { '$first': '$created_on' }, count: { '$sum': 1 } } }, { '$sort': { created_on: 1 } }, { '$project': { values: { key: '$_id', value: '$count' } } }], {})
[forest] 🌳🌳🌳  Unexpected error: can't convert from BSON type long to Date
MongoError: can't convert from BSON type long to Date
    at Connection.<anonymous> (/Users/anshnanda/Developer/KiranaKart/KiranaKart-Admin/node_modules/mongodb/lib/core/connection/pool.js:466:61)
    at Connection.emit (events.js:315:20)
    at processMessage (/Users/anshnanda/Developer/KiranaKart/KiranaKart-Admin/node_modules/mongodb/lib/core/connection/connection.js:384:10)
    at TLSSocket.<anonymous> (/Users/anshnanda/Developer/KiranaKart/KiranaKart-Admin/node_modules/mongodb/lib/core/connection/connection.js:553:15)
    at TLSSocket.emit (events.js:315:20)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:223:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
POST /forest/stats/orderTracking 500 80 - 142.329 ms

Context

TODO: Please provide any relevant information about your setup.

Package Version: 6.7.2
Express Version: 4.17.1
Mongoose Version: 5.8.2
MongoDB Version: 4.4.3

After some debugging, I was able to fix the raw query that was in the logs before forest-express-mongoose failed.

Original query:

order.aggregate(
  [
    {$match: {created_on: {$ne: null}}},
    {
      $group: {
        _id: {
          year: {$year: [{$subtract: ['$created_on', -18000000]}]},
          month: {$month: [{$subtract: ['$created_on', -18000000]}]},
          day: {$dayOfMonth: [{$subtract: ['$created_on', -18000000]}]},
        },
        created_on: {$first: '$created_on'},
        count: {$sum: 1},
      },
    },
    {$sort: {created_on: 1}},
    {$project: {values: {key: '$_id', value: '$count'}}},
  ],
  {},
);

Fixed query:

db.order.aggregate(
  [
    {$match: {created_on: {$ne: null}}},
    {
      $group: {
        _id: {
          year: {$year: [{$toDate: {$subtract: ['$created_on', -18000000]}}]},
          month: {$month: [{$toDate: {$subtract: ['$created_on', -18000000]}}]},
          day: {$dayOfMonth: [{$toDate: {$subtract: ['$created_on', -18000000]}}]},
        },
        created_on: {$first: '$created_on'},
        count: {$sum: 1},
      },
    },
    {$sort: {created_on: 1}},
    {$project: {values: {key: '$_id', value: '$count'}}},
  ],
  {},
);

Clearly, the fix was adding $toDate in front of $year, $month, and $dayOfMonth. This worked because all 3 of those commands take in only a Date object according to MongoDB. Therefore, we needed to convert our Int64 Epoch time to a Date before passing it into the query.

The question still remains: What is the problem in the Source Code and how to fix it?

ForestAdmin / forest-express-mongoose