agmen-hu / node-datapumps

Node.js ETL (Extract, Transform, Load) toolkit for easy data import, export or transfer between systems.
MIT License
291 stars 38 forks source link

Using the group method on MongoDbMixin #30

Open sfhorton01 opened 8 years ago

sfhorton01 commented 8 years ago

When using the MongoDbMixin to read data, I need to be able to perform a group or aggregate first. So what I have is something like:

eventsPump.mixin(MongodbMixin(...))
      .useCollection('sites')
      .from(eventsPump.group(['status.description'], {type: 'Master'},
        { sites : 0, transactions: 0 },
        function (curr, result) {
          result.sites++;
          if (curr.metricsSummary) {
            result.transactions += curr.metricsSummary.transactions.count;
          }
        }
      ))
...

However, I am getting an error on the .from line:

Error: Argument must be datapumps.Buffer or stream
    at Pump.module.exports.Pump.from (/web/gstv-server-apps-reports/node_modules/datapumps/lib/Pump.js:94:15)

Am I doing something wrong? Or is this not really supported?

novaki commented 8 years ago

Unfortunately the .group() method is not wrapped in the pump. A quick workaround could be to call group on mongo collection directly:

eventsPump._mongo.collection.group(<your args here>).stream()

Please note the stream() call after grouping. Input buffers of pumps must be streams, arrays or datapumps Buffers.

rafaneri commented 7 years ago

Is it possible to do the same for aggregation?

eventsPump._mongo.collection.aggregate(<your args here>).stream()
novaki commented 7 years ago

This should work .aggregate too. But the method will only exist in .from call if the mixin receives a mongodb connection, not a connection string.