Design migration for followups in the person model

[danse]

I think that we had more troubles than expected with old-style versus new-style source cases. Let us try to plan and design more this time. I will write a small doc and we could keep it in this repo as a reference until the migration will be over. Both the document and this issue could be a reference for people stumbling upon that part of the code.

Goals:

• have a clear idea about how to adapt the code, for a better implementation
• anticipate possible problems
• have a single reference in dashboard, followup, search
• keep some doc for the poor people who might step into the adaptation logic

[danse]

There was a meeting with me, @rmehner, @jo and @obdulia-losantos. The biggest emerged problems involve results of Elastic full text searches, or searches on specific fields.

Problems

Elastic can return just the whole matching document, this means that, for instance, in case of a search on the followup date it could return new-style followup documents or old-style person documents with matching and not matching followups. This would require an adaptation layer and a double filtering layer in the Sense Dashboard in order to give the user consistent results. We would then have complexity in both the search logic and the querying client logic in Sense Dashboard.

Solution

@jo proposed the brilliant solution of writing a migration deamon. Such a service would continuously run in the background, converting old-style documents to new-style documents. All other components would then assume to have all data available with the new version.

Specifically, the daemon does not need to delete the content of contact.followUps when creating the new ones, because the date of visit will be used as an id of followup documents. Therefore Sense Followup will not get any conflict

@jo and @rmehner estimate writing the daemon to take less than three days

Schedule

We realised that with a similar architecture the rollout of the breaking data model can happen quite seamlessly. The migration daemon will have to be deployed first, this will create new-style followups that will be ignored by other components. After the migration daemon will be in production, all other components will be able to asynchronously switch to the new data model. First, the consumers should switch, that means the dashboard and the search. Lastly also Sense Followup can switch to the new model, after this there will be no old-style documents written anymore

Conclusion

If this works, it will be the smartest migration we ever did