Closed OscarScholten closed 4 years ago
I would like to see a switch on ETCopyData.json (similar to the timeout in Bulk API). The switch could be named "maxConcurrent" and has a default value of 5. Maybe Salesforce increases this value in a future release and the ETCopyData.json is the only thing that would need to be updated.
@OscarScholten I am sorry I never followed up on this conversation. Has this been working for you? I guess that from 1 minute to 1m10 sec is not that critical... we should do put this into the plugin.
@eltoroit I have not found time yet to create a pull request for this issue. Me and my colleagues are running ETCopyData from a branch in my cloned repo. That branch is a few commits behind master by now unfortunately. It's on my longer todo list to get that straightened out and as part of that make sure this improvement finds its way upstream too, but I don't expect to work on this the coming few weeks.
I noticed in #17 you're about to make a release, cool! Don't wait for this feature ;-)
Thanks for letting me know, I will not be waiting ;-)
When exporting many SObjects that each have many records, it is possible to run into 'INVALID_QUERY_LOCATOR' errors. An explanation of the general problem can be found here:
https://help.salesforce.com/articleView?id=000323582&language=en_US&type=1&mode=1
If you are exporting 20 SObjects, with 10.000 records each, you will likely run into this problem. I have implemented a fix in this commit, for which I'd gladly create a pull request, but would like your feedback on the following:
My fix introduces a limiter on the total number of open connections/queries. It will work for 100% of the cases, but on the other hand exports of 50+ SObjects of just a handful of records are likely to be a bit slower due to a lower concurrency; the slowdown that I've measured is around 20%.
So my question is: what do you prefer: a setting to switch the limiter on/off, or use the default limiter for each and every project? My two cent: don't make this configurable and use a default limiter for all projects, this makes sure new users don't run into this problem. In our case, the "default" export went from 1 minute to 1m10 sec, which is not a noticeable difference in my opinion.
Cheers, Oscar