Open joshuaLW opened 3 years ago
jtcohen6
commented
3 years ago Hey @joshuaLW, this is indeed a cryptic error. The question of removing the Snowplow package seems like a good place to start debugging.
packages.yml?snowplow package in dbt_project.yml?
joshuaLW
commented
3 years ago @jtcohen6 three things... I never actually checked that the snowplow modules were removed from packages.yml when i ran "dbt clean" but now having reversed these changes with "dbt deps" I've only this in that file: packages:
I did comment out all variables and models referencing the snowplow package in the project.yml
I didn't remove all dependencies upon snowplow from my models but i expected to get an error of the kind model x refers to model y that does not exist.
I'm going to go at it that way now actually: i'm going to remove references to snowplow models in my models first and then try removing snowplow module again.
jtcohen6
commented
3 years ago I did comment out all variables and models referencing the snowplow package in the project.yml
If I had to pick one of the three as the likeliest cause, I'd say it's this one. Try removing snowplow from packages.yml, run dbt clean, leave the vars in place, and then see what error you get back. (I hope it will be of the missing-ref kind.)
joshuaLW
commented
3 years ago @jtcohen6 good shout. I've just tried emptying the packages.yml then running dbt clean and i'm getting the errors i expected eg. "model x depends on y" so i'll keep going with cleaning up my models till all dependancies are removed... and hopefully it was just that... :) FWIW I thought dbt clean would have cleaned my packages.yml too... but computers can't read my mind can they?
joshuaLW
commented
3 years ago @jtcohen6 So the plot thickens: the error returns even with packages.yml empty. It seems to be connected to one of the models I pinched from the package.
so to recap:
1) i installed snowplow with dbt deps
2) copied three models from the module into my models
3) cleared packages.yml of its content
4) ran dbt clean
5) dbt docs generate (any other cmd get's the same result) and i get the "object of type 'NoneType' has no len()"
joshuaLW
commented
3 years ago @jtcohen6 yup i found the line that was causing the problem. It was a reference to a variable that was nested in the package i'd removed.
jtcohen6
commented
3 years ago There it is! Well found.
I think it would be an improvement to the snowplow package if its models handled better the case when var('snowplow:pass_through_columns') is undefined in the project (None), rather than merely empty ([]).
Perhaps by rewriting that line to be:
{% if var('snowplow_pass_through_columns', []) | length > 0 %}That way, if you do want to copy-paste the model code into your own project, and (sensibly) remove the snowplow-specific vars from your project config, you won't be stymied by a cryptic error.
I'm going to transfer this issue to the snowplow package repository, since that's where the change should take place.
Describe the bug
I'm getting a cryptic error message after removing a module.: TypeError: object of type 'NoneType' has no len()
Steps To Reproduce
dbt clean dbt run (I'm sure the problem is something to do with my project.yml but i'm not sure what exactly is throwing the error.... will report back when i can.)
Expected behavior
An error message that will help me fix my configs/project.yml
Screenshots and log output
2021-01-15 11:19:42.774812 (MainThread): Traceback (most recent call last): File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/main.py", line 124, in main results, succeeded = handle_and_check(args) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/main.py", line 202, in handle_and_check task, res = run_from_args(parsed) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/main.py", line 255, in run_from_args results = task.run() File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/task/runnable.py", line 399, in run self._runtime_initialize() File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/task/runnable.py", line 118, in _runtime_initialize super()._runtime_initialize() File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/task/runnable.py", line 78, in _runtime_initialize self.load_manifest() File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/task/runnable.py", line 65, in load_manifest self.manifest = get_full_manifest(self.config) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/perf_utils.py", line 31, in get_full_manifest adapter.connections.set_query_header, File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/manifest.py", line 678, in load_manifest return ManifestLoader.load_all(config, macro_manifest, macro_hook) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/manifest.py", line 348, in load_all loader.load(macro_manifest=macro_manifest) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/manifest.py", line 204, in load self.parse_project(project, macro_manifest, old_results) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/manifest.py", line 177, in parse_project self.parse_with_cache(path, parser, old_results) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/manifest.py", line 133, in parse_with_cache parser.parse_file(block) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/base.py", line 429, in parse_file self.parse_node(file_block) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/base.py", line 402, in parse_node self.render_update(node, config) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/base.py", line 377, in render_update self.render_with_context(node, config) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/base.py", line 289, in render_with_context parsed_node.raw_sql, context, parsed_node, capture_macros=True File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/clients/jinja.py", line 577, in get_rendered return render_template(template, ctx, node) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/clients/jinja.py", line 528, in render_template return template.render(ctx) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render self.environment.handle_exception() File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception reraise(*rewrite_traceback_stack(source=source)) File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise raise value.with_traceback(tb) File "", line 111, in top-level template code TypeError: object of type 'NoneType' has no len()
System information
Which database are you using dbt with?
The output of
dbt --version:The operating system you're using: MAC OS The output of
python --version: 3.7.3Additional context
I've recently removed the snowplow module and copied some of the models into my own models folder. reversing these steps gets rid of the problem.