alepietrobon
commented
1 year ago @huruizverastegui we're update the readme and restructured the code to be more modular. This should improve on your first 2 points and the last one hopefully.
'How to update' the code is tricky: it's hard to say how people will change the data collection forms. As I have mentioned, the dashboards are tightly coupled with the data sources, so it's hard to predict what changes will be required.
Variabilize is a great idea, we usually do that in our DBT scripts. The variable language differs per SQL dialect though , so to keep this as generic as possible we did not use that here. https://www.mssqltips.com/sqlservertip/7161/sql-variables-sql-server-oracle-postgresql/ We might review this in future versions (i think requiring dbt will be a good step forward as it handles cross-db language)
We're going to make another few updates by next week to make this as clean and as readable as possible, especially since we might have better connectors coming up. So watch this space.
Meanwhile, please have a look and see if it feels more easy to navigate now. Thanks
As discussed during our call on May 16th, I'm concerned with the maintenance and the sustainability of the ETL code. I also understand you have little time left on the project and that building flexibility on forms analysis is not an easy task.
Here are a couple of proposals to make the code more self explanatory and easier for anyone to update in the future, without changing the strategy you agreed on.
Feel free to reach out if you have any questions Hugo Ruiz