offshore not populating if I don't .limit() results

[japel]

Hey there!

I am using sails-hook-orm-offshore with sails 0.12.4 and I have 2 models:

ModelA containing ~2000 records and ModelB containing ~80000 I'm associating ModelA with ModelB with a one-to-many association via an ID field. When I:

ModelA
    .find()
    .populate("data")
    .exec(function (error, found) {
      if (error) {
        console.error(error);
      }

      for (var i = 0; i < found.length; i++) {
        if (found[i].data.length>0) {
          console.log(util.inspect(found[i], {depth: 10}));
        }
      }
    });

nothing is logged to console, but if I:

ModelA
    .find()
    .limit(30)
    .populate("data")
    .exec(function (error, found) {
      if (error) {
        console.error(error);
      }

      for (var i = 0; i < found.length; i++) {
        if (found[i].data.length>0) {
          console.log(util.inspect(found[i], {depth: 10}));
        }
      }
    });

records are logged... The limitation seems to work until weird 1005 records, so if I .limit(1006) it again stops populating, and nothing gets logged.

What could be happening here?

[atiertant]

hi japel, i think your adapter doesn't answer... which one do you use? or this could be the database performance problem...

[japel]

I am using sails-mysql... I will check the server logs and see if anything points to performance

[japel]

It was a performance issue... Thanks for pointing me into that direction. I'll close this.

[atiertant]

sails-mysql use union for populate, you could try using offshore-sql who use left join ... i would be curious to see performance diff

[japel]

populate_offshore-sql: 55265ms populate_sails-mysql: 3339517ms

:flushed:

[atiertant]

@japel offshore-sql would be 60x faster than sails-mysql in this case ! Amazing !

[japel]

yeah, crazy!

[atiertant]

@japel which query did you used for your test ? did you used skip or limit on populate ?

[japel]

@atiertant no, no skip and no limit

[atiertant]

thanks for your tests