search

dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.76k stars 1.62k forks source link

[Bug] fixtures in fixtures/subfolders throwing parsing error #9570

Closed graciegoheen closed 7 months ago

graciegoheen commented 8 months ago

Is this a new bug in dbt-core?

Current Behavior

I have a unit test definition using csv fixtures:

unit_tests:  
  - name: test_cloud_account_mappings
    model: int_cloud_account_mappings__stepped 
    given: 
      - input: ref('int_account_domains__ranked')
        format: csv
        fixture: cloud_account_mappings_mock_input
    expect: 
      format: csv
      fixture: cloud_account_mappings_mock_output

Everything works as expected when I nest my fixture files directly in the fixtures folder:

If i instead nest my fixtures in a subfolder, I get a parsing error:

23:21:34 Unable to do partial parsing because an error occurred. Switching to full reparse.
23:21:34 Encountered an error:
not enough values to unpack (expected 3, got 1)
23:21:34 Traceback (most recent call last):
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/parser/manifest.py", line 392, in load
    project_parser_files = self.partial_parser.get_parsing_files()
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/parser/partial.py", line 191, in get_parsing_files
    self.delete_from_saved(file_id)
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/parser/partial.py", line 288, in delete_from_saved
    self.delete_fixture_node(saved_source_file)
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/parser/partial.py", line 609, in delete_fixture_node
    unit_test = self.saved_manifest.unit_tests.pop(unique_id)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'unit_test.fishtown_internal_analytics.int_cloud_account_mappings__stepped.test_cloud_account_mappings'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/cli/requires.py", line 109, in wrapper
    result, success = func(*args, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/cli/requires.py", line 94, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/cli/requires.py", line 187, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/cli/requires.py", line 216, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/cli/requires.py", line 263, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/cli/requires.py", line 289, in wrapper
    ctx.obj["manifest"] = parse_manifest(
                          ^^^^^^^^^^^^^^^
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/parser/manifest.py", line 1846, in parse_manifest
    manifest = ManifestLoader.get_full_manifest(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/parser/manifest.py", line 324, in get_full_manifest
    manifest = loader.load()
               ^^^^^^^^^^^^^
  File "/venv/dbt-versionless/lib/python3.11/site-packages/dbt/parser/manifest.py", line 407, in load
    (_, line, method) = formatted_lines[-3].split(", ")
    ^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 3, got 1)
23:21:15 not enough values to unpack (expected 3, got 1)

Expected Behavior

I should be able to nest my fixture files in subfolders for better repo organization

Which database adapter are you using with dbt?

snowflake

Additional Context

No response

gshank commented 7 months ago

A pull request, #9714, fixed the secondary exception reported in this ticket, but not the primary error.

emmyoop commented 7 months ago

Upon further investigation, this no longer seems to be an issue. Closing.