Unexpected result using .stream() in sails-postgresql

[eashaw]

Node version: 14.17.0 Sails version (sails): 1.4.4 ORM hook version (sails-hook-orm): ^3.0.2 Sockets hook version (sails-hook-sockets): ^2.0.0 Organics hook version (sails-hook-organics): ^2.0.0 Grunt hook version (sails-hook-grunt): 1.0.4 Uploads hook version (sails-hook-uploads): DB adapter & version (e.g. sails-mysql@5.55.5): sails-postgresql 2.0.0

---

When using .stream() with a model using sails-postgresql the second batch was not iterating through the models correctly.

We are looking into this behavior. To ensure that .stream() is working correctly, you can specify an attribute to sort by in the .stream() and the models will iterate in the correct order.

User.stream().sort('id').eachRecord( async (record)=>{
   // ...
});

[sailsbot]

@eashaw Thanks for posting! We'll take a look as soon as possible.

In the mean time, there are a few ways you can help speed things along:

• look for a workaround. (Even if it's just temporary, sharing your solution can save someone else a lot of time and effort.)
• tell us why this issue is important to you and your team. What are you trying to accomplish? (Submissions with a little bit of human context tend to be easier to understand and faster to resolve.)
• make sure you've provided clear instructions on how to reproduce the bug from a clean install.
• double-check that you've provided all of the requested version and dependency information. (Some of this info might seem irrelevant at first, like which database adapter you're using, but we ask that you include it anyway. Oftentimes an issue is caused by a confluence of unexpected factors, and it can save everybody a ton of time to know all the details up front.)
• read the code of conduct.
• if appropriate, ask your business to sponsor your issue. (Open source is our passion, and our core maintainers volunteer many of their nights and weekends working on Sails. But you only get so many nights and weekends in life, and stuff gets done a lot faster when you can work on it during normal daylight hours.)
• let us know if you are using a 3rd party plugin; whether that's a database adapter, a non-standard view engine, or any other dependency maintained by someone other than our core team. (Besides the name of the 3rd party package, it helps to include the exact version you're using. If you're unsure, check out this list of all the core packages we maintain.)

  ---

Please remember: never post in a public forum if you believe you've found a genuine security vulnerability. Instead, disclose it responsibly.

For help with questions about Sails, click here.

[ochicf]

I also noted some repeated records being returned when using stream and fixed that with the sort clause workaround.

Now I just realised that when checking for null values in the query criteria some records are skipped, and the sort clause does not seem to have any effect.

For example:

const criteria = { someField: null };

const userCount = await User.count(criteria);
let userStreamCount = 0;

await User.stream(criteria).sort('id').eachRecord( async (record)=>{
  userStreamCount += 1;
});

In the above example, userCount is greater than userStreamCount. I've tested with a dataset of 159, the stream just returns 90 records. Also tested with and without a transaction in usingConnection.

[eashaw]

Hi @ochicf, I wasn't able to reproduce the results you were seeing. If you want to reproduce it in a github repo, I'd be happy to take a look.