search

hofstadter-io / hof

Framework that joins data models, schemas, code generation, and a task engine. Language and technology agnostic.
https://docs.hofstadter.io
Apache License 2.0
522 stars 37 forks source link

hof/flow -> airflow #389

Open verdverm opened 5 months ago

verdverm commented 5 months ago

It would be great to transform workflows to other frameworks, airflow as an example, to scale out. There by, hof flow becomes a way to quickly develop the ETL, re: better CUE ergonomics for the ETL twiddling / iteration

Probably helpful if we could snapshot task i/o too, for both iteration and determining what code to generate. Ideally flow can have a standardize record format to feed into hof generators for target workflow frameworks. Might also be helpful for non-workflow functions, like API handlers when generating CRUD+

gedw99 commented 1 month ago

Totally agree …

have a look at:

conduit

https://github.com/ConduitIO/conduit

anyquery

https://github.com/julien040/anyquery/

these are golang based CDC that track changes in many data stores …

I am looking at the best way to integrate these with hof

would appreciate any feedback you have .

see my github profile whee we can chat on signal