One of the problems in keeping data in-sync is that we often can't detect when something changes in CiviCRM that means Mailchimp is no longer up-to-date.
e.g. if you have a smart group that's tied to an interest grouping at Mailchimp, and you change something that means a contact now matches (or doesn't) the smart group criteria.
Such changes can only be fixed by doing a Push from CiviCRM to Mailchimp sync.
However this opens the possibility that any bulk upload (for example) done at Mailchimp that has not yet been Pulled down to CiviCRM would get ruined as those subscribers would be unsubscribed.
What about this for an algorithm for a new type of sync:
Keep a log of the last time a pull/smart sync was done.
Download all records from Mailchimp that have been updated since the last pull/smart sync and pull these into CiviCRM. This is different to a pull sync which also includes removing people in CiviCRM that are not found among the Mailchimp subscribers.
Then do a general push sync.
I think this could reduce the chance of data loss due to simultaneous bulk changes in CiviCRM and Mailchimp. What do you think?
A second idea is to abandon support for Smart Groups, since they're what causes the trouble. Without smart groups we could detect changes to group membership accurately and then queue those Contacts for a frequently-run background sync operation. This would be much more efficient, at the cost of making users manually update groups. I suppose there's nothing to stop the implementation of this idea anyway, then it works for non-smart groups.
One of the problems in keeping data in-sync is that we often can't detect when something changes in CiviCRM that means Mailchimp is no longer up-to-date.
e.g. if you have a smart group that's tied to an interest grouping at Mailchimp, and you change something that means a contact now matches (or doesn't) the smart group criteria.
Such changes can only be fixed by doing a Push from CiviCRM to Mailchimp sync.
However this opens the possibility that any bulk upload (for example) done at Mailchimp that has not yet been Pulled down to CiviCRM would get ruined as those subscribers would be unsubscribed.
What about this for an algorithm for a new type of sync:
Keep a log of the last time a pull/smart sync was done.
Download all records from Mailchimp that have been updated since the last pull/smart sync and pull these into CiviCRM. This is different to a pull sync which also includes removing people in CiviCRM that are not found among the Mailchimp subscribers.
Then do a general push sync.
I think this could reduce the chance of data loss due to simultaneous bulk changes in CiviCRM and Mailchimp. What do you think?
A second idea is to abandon support for Smart Groups, since they're what causes the trouble. Without smart groups we could detect changes to group membership accurately and then queue those Contacts for a frequently-run background sync operation. This would be much more efficient, at the cost of making users manually update groups. I suppose there's nothing to stop the implementation of this idea anyway, then it works for non-smart groups.