amedia / dbtwiz

Python package with CLI helper tool for dbt
MIT License
0 stars 0 forks source link

Check if manifest must be rebuilt before parsing #14

Open lhz opened 1 month ago

lhz commented 1 month ago

We skip running dbt parse before reading the manifest, since the parsing is quite slow.

When new models have been added, we need to update the manifest before they will be available in interactive selectors and other parts of dbtwiz, so we should do a fast recursive scan of the model folder to check for files that have been updated since the manifest was last built and automatically do a dbt parse before reading the manifest when needed.

lhz commented 1 month ago

Need to implement this in Python: find models -newer ~/.config/dbtwiz/models-cache.json

lhz commented 1 month ago
from pathlib import Path

cache_mtime = (Path.home() / ".config" / "dbtwiz" / "models-cache.json").stat().st_mtime

all_models = (Path.cwd() / "models").glob("**/*.sql")

modified_models = list(filter(lambda p: p.stat().st_mtime > cache_mtime, all_models))