Review data preparation pipelines

open-innovations / jrf-insight

JRF North England Insight Finder

https://open-innovations.github.io/jrf-insight/

MIT License

1 stars 0 forks source link

Review data preparation pipelines #57

Open gilesdring opened 11 months ago

gilesdring commented 11 months ago

Consider structure of pipelines and efficiency of running

luke-strange commented 10 months ago

I'd say this is high prioirty now and something Giles and I (mainly) should discuss soon. We need to come up with more consistent approach as we currently have the dvc pipelines with place_Data and then duckDB for time series. Need consitency in:

[ ] naming conventions
[ ] data storage
[ ] acessing datat within lume
[ ] wther or not we do any post-processing through nunjucks or make sure everything is ready to go beforehand.