What's the supported dbt version?

[danielsepulvedab]

Hey, thanks for sharing you work on this.

I'm having some problems using the library to parse my project. Neither the the manifest parser nor the DbtFolderReader is able to parse it without failing. For my project I'm using dbt 0.20.0, is this version supported by this package? The manifest schema has slightly changed the last couple of versions and I'm wondering if you might be working with another version.

[z3z1ma]

Hey @danielsepulvedab

I use dbt version 0.20.x myself and have no issue. Is there any more specific information you can share on the error you are seeing?

Warm Regards,

[danielsepulvedab]

Hey @z3z1ma, when I use the --dbt_path option I'm getting:

Traceback (most recent call last):
  File "/Users/daniel.sepulveda/work/dbt-models/venv/bin/dbt-metabase", line 6, in <module>
    dbtmetabase.main(sys.argv[1:])
  File "/Users/daniel.sepulveda/work/dbt-models/venv/lib/python3.7/site-packages/dbtmetabase/__init__.py", line 364, in main
    dbt_include_tags=parsed.dbt_include_tags,
  File "/Users/daniel.sepulveda/work/dbt-models/venv/lib/python3.7/site-packages/dbtmetabase/__init__.py", line 83, in models
    docs_url=dbt_docs_url,
  File "/Users/daniel.sepulveda/work/dbt-models/venv/lib/python3.7/site-packages/dbtmetabase/parsers/dbt_folder.py", line 79, in read_models
    model_key=ModelKey.nodes,
  File "/Users/daniel.sepulveda/work/dbt-models/venv/lib/python3.7/site-packages/dbtmetabase/parsers/dbt_folder.py", line 130, in _read_model
    mb_columns.append(self._read_column(column, schema))
  File "/Users/daniel.sepulveda/work/dbt-models/venv/lib/python3.7/site-packages/dbtmetabase/parsers/dbt_folder.py", line 174, in _read_column
    for test in column.get("tests", []):
TypeError: 'NoneType' object is not iterable

When trying to parse the manifest, I'm getting:

Traceback (most recent call last):
  File "/Users/daniel.sepulveda/work/dbt-models/venv/bin/dbt-metabase", line 6, in <module>
    dbtmetabase.main(sys.argv[1:])
  File "/Users/daniel.sepulveda/work/dbt-models/venv/lib/python3.7/site-packages/dbtmetabase/__init__.py", line 364, in main
    dbt_include_tags=parsed.dbt_include_tags,
  File "/Users/daniel.sepulveda/work/dbt-models/venv/lib/python3.7/site-packages/dbtmetabase/__init__.py", line 83, in models
    docs_url=dbt_docs_url,
  File "/Users/daniel.sepulveda/work/dbt-models/venv/lib/python3.7/site-packages/dbtmetabase/parsers/dbt_manifest.py", line 99, in read_models
    self._read_model(node, include_tags=include_tags, docs_url=docs_url)
  File "/Users/daniel.sepulveda/work/dbt-models/venv/lib/python3.7/site-packages/dbtmetabase/parsers/dbt_manifest.py", line 196, in _read_model
    fk_target_table_alias = self.manifest[model_key][depends_on_id]["alias"]
KeyError: 'source.project.model'

I'm using dbt-metabase==0.8.0rc4

[z3z1ma]

So for the folder reader, Is it possible you have tests key somewhere in your project on a column with nothing defined afterwards or null? I know it sounds strange but its the first sanity check I'd look towards. In either case its an edge case we should account for.

In the manifest reader, source.project.model seems like a very odd model id but it definitely was derived from your manifest since depends_on_id can only come directly from a manifest node. In either case- I know exactly where we can catch that edge case too.

Lets keep this issue open and I'll put through a PR to catch these cases. Why they exist during the parsing of your project I am not entirely sure though so open to any more info if its relevant to this package.

[danielsepulvedab]

I'll check for occurrences of the cases that you mentioned.

The source.project.model key looks odd because I masked. I forgot to mention that or to be more obvious about it, sorry. The actual value of the Key is a valid model in my project.

Let me know if I can provide you with more data to debug this. Thank you!

[z3z1ma]

@danielsepulvedab

The last PR introduced some more resiliency for these kinds of edge cases. If you give it another run with the latest master, it should at a minimum run through. I'd still be curious on the tests keys anywhere with null values which seems to be what the above tracebacks boil down to on the surface at least.

[danielsepulvedab]

@z3z1ma Thanks for the pointers and the fix in the code. There was indeed a column that had a some like:

      - name: <column_name>
        description: <column_description>
        test:

By removing that field the run with the --dbt_path ran. After compiling the project the version with the --dbt_manifest_path still prompted an error.

Tomorrow I'll download the master version and run it with that to see the results and I'll update this issue with what I get.

[danielsepulvedab]

Hey @z3z1ma, with the current version on master, I could successfully run my project with both parsers. Thank you so much. I think we could close this issue, since it was solved on #64