Only pulling 25 records at a time?

[synthetiv]

Describe the bug I'm using this plugin on a client's site to sync Salesforce Contact objects to WordPress Users. 17 fields are mapped, it's set to pull every 5 minutes and to push whenever a WP user is created or updated, and the "Process asynchronously" checkbox is checked. When more than 25 records are updated in Salesforce at once (e.g. by bulk editing a list in SF), only 25 records are updated in WordPress.

To Reproduce Steps to reproduce the behavior:

1. Log in to the WordPress site and map Salesforce Contacts to WordPress Users. Schedule a pull every 5 minutes. You can set the field map to push to SF when a WP user is created or updated, but that doesn't appear to be necessary to reproduce this issue.
2. Enable debug logging in the Object Sync for Salesforce plugin settings.
3. Log in to Salesforce, view Contacts, and select 26 or more records.
4. Edit a field (change everyone's state to MN, for instance) to trigger a bulk update.
5. Watch the debug log and/or user list in WordPress.

25 new users will appear (after 5 minutes or so, once the scheduled pull has been performed). The debug log will show the initial query retrieving 25 records, those records being added to the Action Scheduler queue, and the corresponding records being created or updated in WordPress. If the plugin is set to push to Salesforce as well, it will show some or all records being pushed back to SF (which isn't really necessary, but isn't a problem either). Then it will show a new query for users with a different LastModifiedDate cutoff that returns 0 results. None of the other records updated in SF will be synced, unless they're updated separately.

Note: early on in my troubleshooting I suspected that the 5-minute pulls were too frequent, that maybe one pull didn't have enough time to finish before the next pull began, but based on the logs I've examined, the pull and subsequent push take less than one minute to complete.

Expected behavior All updated records are synced to WordPress. Debug log should show a query that fetches the first 25 records, then another that fetches the next 25, until they've all been retrieved. Subsequent pushes back to SF are (again) unnecessary but acceptable.

Environment (please complete the following information):

• WordPress Version: 5.1.1
• Other relevant plugins: Object Sync for Salesforce 1.8.12, WP Members 3.2.6.1
• PHP Version: 7.3.15

Additional context I'm assuming the limit of 25 comes from here, but my understanding from looking at the surrounding code there is that that's not supposed to be a "hard" limit -- that after pulling 25 records, the plugin should run another query to sync the next 25, and so on, rather than ignoring subsequent records. But please correct me if I'm wrong!

Thanks so much.

[jonathanstegall]

Your assumption is correct, at least. What it's supposed to do is save the query that it runs, and then offset it by that value. So each time it runs, the limit increments and the query is saved in the database.

Are you able to see if the query is in fact being saved in your wp_options table? If it is, is it failing to increment the limit value?

[synthetiv]

Thanks for writing back so quickly, that was amazing.

So a bunch of testing and extra logging later, I can say that the query is being saved to the options table, but it seems like this call to Object_Sync_Sf_Salesforce_Pull::increment_current_type_datetime() is changing the LastModifiedDate for the next query (because it sets the object_sync_for_salesforce_pull_last_sync_Contact option, which is then used by get_pull_date_value()), and that's causing the next query to return no results.

I'm thinking increment_current_type_datetime() should only be called after checking for further results, and then only if none are found. In fact, it's already called by clear_current_type_query() -- is it necessary to call it directly from get_updated_records() at all?

[jonathanstegall]

I think I originally had it run the increment_current_type_datetime method before checking for further results in case something in Salesforce had changed in between. Their API can be quite slow at times, as you probably know.

I'm open to your idea as a solution. I'm curious about the best way to test it, other than just running a bunch of large queries. Have you already tested it? Did it fix the problem?

[jonathanstegall]

I ran a quick local test, and so far this seems to skip over batches of records. I'm not sure why yet. It might be a bit before I can go deep enough to figure out if there's an issue with this solution, and what it is, if so.

[synthetiv]

Thanks for looking! So you mean removing the call to increment_current_type_datetime is causing records to be dropped? Or just that you were able to replicate the issue to begin with?

I may have some time this week to look into this more, and I'll let you know what I learn -- and I'll certainly submit a PR if I get that far.

[jonathanstegall]

Sure. I tested it with removing the call to increment_current_type_datetime. I had it do an error_log with the SOQL it was running. It would increment by 25 a couple times, then it would increment by 75 and miss a bunch, or 50.

I might need to do more logging with removing it, or tweak other things of course, but that was what happened on the first try.

[jonathanstegall]

I've also tried running the same log on the plugin's current version. It increments the offset by 25 records, as it should. Whatever the issue is, it seems to be inconsistent, and presumably not fixable just by removing that method.

[synthetiv]

Hmm, OK -- sounds like I need to do some more extensive testing to make sure this isn't somehow caused by a plugin conflict, which I really didn't think it could be... but who knows! Thanks again. I probably won't be able to do that testing this week, but hopefully I'll have time next week.

[synthetiv]

Hi again -- well, many very eventful 'next weeks' have come and gone but I did finally get the chance to re-test using an unmodified version of the plugin with the settings I had been using, both on a staging copy of this site with all other plugins deactivated, and on a clean install of WordPress, and I seem to still be running into the same problem: when I update more than 25 records at a time, only the first 25 are pulled by WP.

I'll try to make more time to test without that call to increment_current_type_datetime, but in the meantime, I'd welcome any other ideas as to what might be going on. I think the strangest part of all this is that our client has reported that they can bulk-update any number of records at once on production and they all get pulled to WP, which obviously casts a lot of doubt on my theory about that function call being the problem -- but as far as I can tell, the plugin settings on production and staging are functionally the same. Are you aware of any settings on the Salesforce end (like in the connected app) that might change the number of records that can be queried at once, or anything like that?

[jonathanstegall]

To clarify, are you saying that on a staging site, it only pulls 25 records, but on the production site it pulls any number of records?

If so, that is very odd. I'm not aware of anything that would cause such a thing.

[synthetiv]

Exactly. Yeah, it doesn't make a lot of sense. Different web servers and different SF orgs, but as far as I know, the settings are the same. I'm going to spend some time today comparing them side-by-side, though.