OpenFn / adaptors

The new home for OpenFn adaptors; re-usable connectors for the most common DPGs and DPI building blocks.
GNU General Public License v3.0
4 stars 8 forks source link

salesforce: autofetch is broken #644

Open josephjclark opened 1 week ago

josephjclark commented 1 week ago

The autofetch feature on the salesforce query sort of works but also not really.

This PR #478 introduced a few regressions:

Basically autofetch needs to push the RECORDS into a single flat results array

Not to mention that the code is a bit quirky, to say the least.

I'd also ask whether callback should be triggered per batch, or at the end result. I think that when you're talking about a batch processing API it make sense to chunkup the callback or else you're likely to hit a problem downstream. Maybe this is an option. This is something that comes up on many APIs so I'd want to think carefully about what we do here.

josephjclark commented 1 week ago

And there are no unit tests on any of this functionality