search

eltoroit / ETCopyData

SFDX Plugin to populate your scratch org and/or developer sandbox with data extracted from multiple sObjects.
113 stars 23 forks source link

INVALID_QUERY_LOCATOR error during export #36

Closed OscarScholten closed 4 years ago

OscarScholten commented 5 years ago

When exporting many SObjects that each have many records, it is possible to run into 'INVALID_QUERY_LOCATOR' errors. An explanation of the general problem can be found here:

https://help.salesforce.com/articleView?id=000323582&language=en_US&type=1&mode=1

If you are exporting 20 SObjects, with 10.000 records each, you will likely run into this problem. I have implemented a fix in this commit, for which I'd gladly create a pull request, but would like your feedback on the following:

My fix introduces a limiter on the total number of open connections/queries. It will work for 100% of the cases, but on the other hand exports of 50+ SObjects of just a handful of records are likely to be a bit slower due to a lower concurrency; the slowdown that I've measured is around 20%.

So my question is: what do you prefer: a setting to switch the limiter on/off, or use the default limiter for each and every project? My two cent: don't make this configurable and use a default limiter for all projects, this makes sure new users don't run into this problem. In our case, the "default" export went from 1 minute to 1m10 sec, which is not a noticeable difference in my opinion.

Cheers, Oscar

eltoroit commented 5 years ago

I would like to see a switch on ETCopyData.json (similar to the timeout in Bulk API). The switch could be named "maxConcurrent" and has a default value of 5. Maybe Salesforce increases this value in a future release and the ETCopyData.json is the only thing that would need to be updated.

eltoroit commented 4 years ago

@OscarScholten I am sorry I never followed up on this conversation. Has this been working for you? I guess that from 1 minute to 1m10 sec is not that critical... we should do put this into the plugin.

OscarScholten commented 4 years ago

@eltoroit I have not found time yet to create a pull request for this issue. Me and my colleagues are running ETCopyData from a branch in my cloned repo. That branch is a few commits behind master by now unfortunately. It's on my longer todo list to get that straightened out and as part of that make sure this improvement finds its way upstream too, but I don't expect to work on this the coming few weeks.

I noticed in #17 you're about to make a release, cool! Don't wait for this feature ;-)

eltoroit commented 4 years ago

Thanks for letting me know, I will not be waiting ;-)