search

theonesp / p_zero-code

MIT License
2 stars 1 forks source link

discuss the best etl extraction #8

Open theonesp opened 2 years ago

theonesp commented 2 years ago

every 24 hours several tables will be parsed from stage_sild, then transformed from wide to long and finally inserted into pzero.chartevents, ¿what is the best strategy for doing this?

theonesp commented 2 years ago

Now the chartevents view takes 8 seconds to run, it is transforming 88259 wide rows from r_monitor_parsed into 558153 long rows. The table contains data from a year period, that means it will take 10 seconds more to run every year if we don't improve the ETL workflow.

xborrat commented 2 years ago

what about auto refresh materialized view with a cron?

xborrat commented 2 years ago

We will use nifi with scheduled periodic execution.