Refactor/modular design

sungchun12 / fst

fst: flow state tool | smooth where you want it, friction where you need it when data engineering

https://www.loom.com/share/ecfbdfb981e4443d94d2c95f16176118

Apache License 2.0

32 stars 1 forks source link

Refactor/modular design #10

Closed sungchun12 closed 1 year ago

sungchun12 commented 1 year ago

Let's make it easier to evolve building blocks together!

I made a TON of breaking changes to this to do the below.

Modular file design
fst start now works simply with an optional --path flag
Rerun with dbt test if first dbt build command did not run tests
Kept in the platform agnostic polling mechanism
No need to use nested git clones once released to pypi

sungchun12 commented 1 year ago

To note: The dbt build time is big, at ~20 seconds. Although this is likely my local issue as it persists when going back to earlier commits as well.

At some point I've had new yaml file be generated for models which were already described in schema.yml, but I couldn't reproduce it. It must be some edge case

Yep, I've had this happen rarely, but I can see that. For the dbt build, that feels like a WSL issue because compilation and runtime shouldn't be that long.