Compare shortlisted ETL pipeline tools

[shaarful]

• [x] Create a subtask for each tool
• [x] #7
• [x] #8
• [x] #9
• [x] Create a test database - Use AMR Data (remember that in practice the format will be different)
• [ ] Put them through the process without editing the data to see if they load
• [ ] Put the database through the process and add a condition. See if the database loads with the set condition (deleting patient names)
• [ ] Aggregate the data according to Specimen Category
• [ ] Compare the three

[WaliZaman]

All three tools had their shortcomings that were described in the comments before closing the individual issues. Luigi:Made models according to the tutorials and codes provided by the documentation and other sites where people have used online sql databases. One model which used a single python code was successful, but n could not be replicated. The others that had multiple python codes all ended with the task failing. My own code ran partially, only for saving a xlsx file in destination folders. When made into multiple python codes, this task also failed. Prefect: Prefect could not be run/tested due to lack of documentation of codes. The codes provided to test the models did not run and returned an error. DBT: Followed the documentation to setup everything required to run DBT on python. An online account is needed to access and create the environment required to run each project/model. A lot of setting up is required to run the pipelines.

So for now, I will try to set up a pipeline using python, from scratch.