Performance question: over an hour for `dbt-osmosis document` ?

z3z1ma / dbt-osmosis

Provides automated YAML management, a dbt server, streamlit workbench, and git-integrated dbt model output diff tools

https://z3z1ma.github.io/dbt-osmosis/

Apache License 2.0

422 stars 45 forks source link

Performance question: over an hour for `dbt-osmosis document` ? #38

Closed jaredx435k2d0 closed 1 year ago

jaredx435k2d0 commented 1 year ago

Running dbt-osmosis document took over an hour, and eventually froze iTerm2 (I'd guess due to memory issues?)

In terms of the size of our Snowflake instance, I think I'd say we're small to medium.

Is this expected? I have a total of 2051 models

z3z1ma commented 1 year ago

@jaredx435k2d0 I have now taken a first stab at parallelizing the yaml stuff. I would expect a massive speed increase. Closing this for now. We can revisit this once we re-assess performance profile on more projects in the wild.

jaredx435k2d0 commented 1 year ago

This is awesome news! Thank you so much.