search

inigohidalgo / prefect-polygon-etl

ELT for Polygon.io financial data using Prefect
2 stars 0 forks source link

feat: use kedro datasets #3

Open inigohidalgo opened 4 months ago

inigohidalgo commented 4 months ago

One possibility:

setup flow, run every X time from a centralized config repo: config.yml -> OmegaConfigLoader -> JSON Block

runtime flow: DataCatalog.from_config(JSON Block)