z3z1ma / dbt-osmosis

Provides automated YAML management, a dbt server, streamlit workbench, and git-integrated dbt model output diff tools
https://z3z1ma.github.io/dbt-osmosis/
Apache License 2.0
422 stars 45 forks source link

KeyError: 'sources' #115

Closed awoehrl closed 6 months ago

awoehrl commented 7 months ago

Hi!

I'm regularly seeing the following error, when trying to run dbt-osmosis. I'm not quite sure, but I think this happens, when I change the folder structure of my models, after I already generated yaml files. When I run dbt-osmosis again, I will see the error from time to time:

Traceback (most recent call last):
  File "/home/projects/.envs/dbt_env/bin/dbt-osmosis", line 8, in <module>
    sys.exit(cli())
  File "/home/projects/.envs/dbt_env/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/projects/.envs/dbt_env/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/projects/.envs/dbt_env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/projects/.envs/dbt_env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/projects/.envs/dbt_env/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/projects/.envs/dbt_env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/projects/.envs/dbt_env/lib/python3.10/site-packages/dbt_osmosis/main.py", line 80, in wrapper
    return func(*args, **kwargs)
  File "/home/projects/.envs/dbt_env/lib/python3.10/site-packages/dbt_osmosis/main.py", line 216, in refactor
    if runner.commit_project_restructure_to_disk():
  File "/home/projects/.envs/dbt_env/lib/python3.10/site-packages/dbt_osmosis/core/osmosis.py", line 681, in commit_project_restructure_to_disk
    if structure.output["sources"]:
KeyError: 'sources'
ArtemCLime commented 7 months ago

@awoehrl Hey! I also faced with this error - it happens usually when dbt-osmosis faces with duplicated model. Your dbt-osmosis configuration sets a specific folder/files structure. When you generate YAML files, it places them according to your configuration path in your dbt_project.yml. If you change folder structure and run osmosis again - it will probably output in such an error - when trying to create a doc that already exists, but in a different location. When you change the file structure of you models, remember to:

To resolve this error try to look into the model file at which exception appeared and check if there are any duplicated YAML model exists already for it.