Separate transform from load in the ETL

[Nelly-Barret]

For now, the Transform and the Load steps are really mixed because:

• every BATCH_SIZE Resource instances created in memory,
• they are inserted in the database
• then they are deleted from the main memory
• and so on until all Resource instances are created and inserted

This does not allow a good separation between the data transformation and the loading in the database.

Instead, I was thinking of the folowing:

• create the Resource instances in memory
• write them in a JSON file every BATCH_SIZE
• write the JSON file in the database after all Resource instances have been created.

MongoDB supports data loading from a JSON file: https://www.mongodb.com/resources/languages/json-to-mongodb

[Nelly-Barret]

Recall that, during the Transform step, when we create ExaminationRecord and DiseaseRecord instances we need to have references to Examination, Hospital and Disease instances. This is curently done by inserting them in the database and then to retrieve them in memory to keep track of the mappings, e.g., column name <-> examination id, disease name <-> disease id, etc.

Therefore, I can:

• call Load class within the Transform step to insert only xamination, Hospital and Disease instances
• call Load after Transform to load the rest (Patient, Sample, ExaminationRecord and DiseaseRecord instances) in the database

[Nelly-Barret]

Done at https://github.com/Nelly-Barret/BETTER-fairificator/pull/14