search

eherve / mongoose-datatable

Server side dataTable request support for mongoose
MIT License
42 stars 29 forks source link

Deep populate solution. #32

Closed Fonz001 closed 8 years ago

Fonz001 commented 8 years ago

Hi!

First of all, thanks for this plugin and all the hard work you put into it.

I have a question regarding to the deep populate in mongoose. This is the client-side scenario:

"aoColumns": [
          {"mData": "_id"},
          {"mData": "product"},  //<==== Reference for autopopulate
          {"mData": "product.manufactor"}, //<=== Reference for deep autopopulate
]

Now when I request new data from the server, I have the following (default) implementation:

Item.dataTable(req.query, {}, function(err, data) {
    if (err) return next(err);
    res.send(data);
});

The problem is that the "product.manufactor" field only contains the ObjectID, not the deep population.

My solution to this problem is (using the async library):

Item.dataTable(req.query, {}, function(err, data) {
    if (err) return next(err);
    async.map(data.data, function(item, done){
      item.product.populate('manufactor', done);
    }, function(err){
      if(err) return next(err);
      res.send(data);
    });
});

Is there a better solution to this problem? (I'd prefer not to call this populate() on 10/25/50/100 items)

Greetings!

eherve commented 8 years ago

Hi,

Thank you for using this module.

Did you try to specify the field you need instead of just the manufactor ? Like: product.manufactor.name or something like that ?

Regards

Fonz001 commented 8 years ago

Hi!

That doesnt help. It says: PATH_SEPARATOR not found.

Replacing PATH_SEPARATOR for "." doesn't help either. The field is not populated (the objectId is still there)

Greetings!

eherve commented 8 years ago

Ok can you give me the debug trace please

Fonz001 commented 8 years ago

Hi!

I got it figured out; However, I'm working with company sensitive data, so I had to create a fresh project just to solve this bug. See: https://github.com/Fonz001/mongoose-datatables-test

Installation:

  1. (clone repo)
  2. npm install
  3. node bin/db-seed.js (assuming you have a local mongodb installation)
  4. node bin/www
  5. goto: http://localhost:3000/

The crucial part how I got it to work is located in: view/index.jade:

This works

          {"mData": "name"},
          {"mData": "product"},
          {"mData": "product.manufacturer.name"}

This fails:

          {"mData": "name"},
          {"mData": "product.name"},
          {"mData": "product.manufacturer.name"}

Now for the debug info during the fail:

Query: { draw: '1',
  columns:
   [ { data: 'name',
       name: '',
       searchable: 'true',
       orderable: 'true',
       search: [Object] },
     { data: 'product.name',
       name: '',
       searchable: 'true',
       orderable: 'true',
       search: [Object] },
     { data: 'product.manufacturer.name',
       name: '',
       searchable: 'true',
       orderable: 'true',
       search: [Object] } ],
  order: [ { column: '0', dir: 'asc' } ],
  start: '0',
  length: '10',
  search: { value: '', regex: 'false' },
  _: '1437507698573' }
Search Criteria builded: { options: {},
  pageStart: 0,
  pageSize: 10,
  nbColumns: 3,
  search: undefined,
  fields:
   [ { index: 0,
       path: 'name',
       searchable: true,
       search: undefined,
       sortable: true,
       sort: { dir: 'asc', precedence: 0 },
       selectable: true,
       type: 'String',
       ref: undefined,
       refType: undefined,
       arrayType: undefined,
       base: undefined,
       arrayPath: undefined },
     { index: 1,
       path: 'product.name',
       searchable: true,
       search: undefined,
       sortable: true,
       sort: undefined,
       selectable: true,
       type: 'String',
       ref: undefined,
       refType: undefined,
       arrayType: undefined,
       base: [ 'product' ],
       arrayPath: undefined },
     { index: 2,
       path: 'product.manufacturer.name',
       searchable: true,
       search: undefined,
       sortable: true,
       sort: undefined,
       selectable: true,
       type: 'String',
       ref: undefined,
       refType: undefined,
       arrayType: undefined,
       base: [ 'product', 'manufacturer' ],
       arrayPath: undefined } ],
  select: { name: 1, product: 1 },
  sort: { name: 'asc' },
  conditions: undefined,
  populate:
   [ { path: 'product', sort: {}, select: 'name' },
     { path: 'product.manufacturer', sort: {}, select: 'name' } ] }
Data: { draw: '1',
  recordsTotal: 4,
  recordsFiltered: 4,
  data:
   [ { _id: 55ae9ebd60da88e82766f159,
       name: 'Item 1',
       product: { _id: 55ae9ebd60da88e82766f156, name: 'Product 1' } },
     { _id: 55ae9ebd60da88e82766f15a,
       name: 'Item 2',
       product: { _id: 55ae9ebd60da88e82766f156, name: 'Product 1' } },
     { _id: 55ae9ebd60da88e82766f15b,
       name: 'Item 3',
       product: { _id: 55ae9ebd60da88e82766f157, name: 'Product 2' } },
     { _id: 55ae9ebd60da88e82766f15c,
       name: 'Item 4',
       product: { _id: 55ae9ebd60da88e82766f158, name: 'Product 3' } } ] }

Conclusions:

  1. This problem still needs some sort of fix
  2. A quick fix is to not specify a field but an object (see working/failing case)
  3. I replaced the PATH_SEPARATOR for "." (temporary fix)

And finally, since you are already digging into the code. Do you have any suggestions what a better implementation might be? Note that I need the object_id for link to the item/product/manufacturer.

Greetings,

Patrick

Fonz001 commented 8 years ago

I apologise! It seems that the test case from above is not a quick fix. I had 2 node servers running by accident...

Still, the project can easily be used for debugging.

The settings:

        "aoColumns": [
          {"mData": "name"},
          {"mData": "product.name"},
          {"mData": "product.manufacturer.name"}
        ],

The debug trace:

Query: { draw: '1',
  columns:
   [ { data: 'name',
       name: '',
       searchable: 'true',
       orderable: 'true',
       search: [Object] },
     { data: 'product.name',
       name: '',
       searchable: 'true',
       orderable: 'true',
       search: [Object] },
     { data: 'product.manufacturer.name',
       name: '',
       searchable: 'true',
       orderable: 'true',
       search: [Object] } ],
  order: [ { column: '0', dir: 'asc' } ],
  start: '0',
  length: '10',
  search: { value: '', regex: 'false' },
  _: '1437551971756' }
Search Criteria builded: { options: {},
  pageStart: 0,
  pageSize: 10,
  nbColumns: 3,
  search: undefined,
  fields:
   [ { index: 0,
       path: 'name',
       searchable: true,
       search: undefined,
       sortable: true,
       sort: { dir: 'asc', precedence: 0 },
       selectable: true,
       type: 'String',
       ref: undefined,
       refType: undefined,
       arrayType: undefined,
       base: undefined,
       arrayPath: undefined },
     { index: 1,
       path: 'product.name',
       searchable: true,
       search: undefined,
       sortable: true,
       sort: undefined,
       selectable: true,
       type: 'String',
       ref: undefined,
       refType: undefined,
       arrayType: undefined,
       base: [ 'product' ],
       arrayPath: undefined },
     { index: 2,
       path: 'product.manufacturer.name',
       searchable: true,
       search: undefined,
       sortable: true,
       sort: undefined,
       selectable: true,
       type: 'String',
       ref: undefined,
       refType: undefined,
       arrayType: undefined,
       base: [ 'product', 'manufacturer' ],
       arrayPath: undefined } ],
  select: { name: 1, product: 1 },
  sort: { name: 'asc' },
  conditions: undefined,
  populate:
   [ { path: 'product', sort: {}, select: 'name' },
     { path: 'product.manufacturer', sort: {}, select: 'name' } ] }
Data: { draw: '1',
  recordsTotal: 4,
  recordsFiltered: 4,
  data:
   [ { _id: 55ae9ebd60da88e82766f159,
       name: 'Item 1',
       product: { _id: 55ae9ebd60da88e82766f156, name: 'Product 1' } },
     { _id: 55ae9ebd60da88e82766f15a,
       name: 'Item 2',
       product: { _id: 55ae9ebd60da88e82766f156, name: 'Product 1' } },
     { _id: 55ae9ebd60da88e82766f15b,
       name: 'Item 3',
       product: { _id: 55ae9ebd60da88e82766f157, name: 'Product 2' } },
     { _id: 55ae9ebd60da88e82766f15c,
       name: 'Item 4',
       product: { _id: 55ae9ebd60da88e82766f158, name: 'Product 3' } } ] }
eherve commented 8 years ago

Hi,

Thanks for the trace, so both cases are not working ? The debug shows that the populate should have been done... I'll try tonight to see what is the request that is send to the database.

Fonz001 commented 8 years ago

Both of the cases fail indeed.

Thank you for your time!

eherve commented 8 years ago

I have pushed a solution for deep populate but I have to test it more and add one more feature before releasing it. This version does the deep populate but you need to request all intermediate fields. I'm adding the auto intermediate fields request for the release.

Now

1.0.3a 
"aoColumns": [
          {"mData": "name"},
          {"mData": "product.name"},
          {"mData": "product.manufacturer"},
          {"mData": "product.manufacturer.name"}
],

Next

1.0.3 
"aoColumns": [
          {"mData": "name"},
          {"mData": "product.name"},
          {"mData": "product.manufacturer.name"}
],
Fonz001 commented 8 years ago

Thanks! Great that you took your time to solve the issue!

eherve commented 8 years ago

Version 1.0.4 solve the deep populqtion