search

medic / cht-pipeline

Data models for transforming the raw data from Couch DB into models that can then be queried to build dashboards
GNU General Public License v3.0
1 stars 4 forks source link

Debug macros not executing in dbt testing environment #154

Open lorerod opened 2 months ago

lorerod commented 2 months ago

I tried implementing unit tests for incremental tables as part of Enhance DBT models testing work. But the incremental conditions depend on the max_existing_timestamp macro, which is not working in the test environment.

Investigate why macros don't work in the testing environment.

andrablaj commented 2 months ago

@witash any chance this is solved in the issue https://github.com/medic/cht-pipeline/issues/144 you are working on?

lorerod commented 2 months ago

The test I tried to implement for the person incremental model:

- name: test_person_model_transformation_and_data_integrity_incremental
    model: person
    overrides:
      macros:
        # unit test this model in "incremental" mode
        is_incremental: true
    given:
      - input: ref('contact')
        format: csv
        fixture: person_contact_initial
      - input: source('couchdb', "{{ env_var('POSTGRES_TABLE') }}")
        format: csv
        fixture: person_source_table_initial
      - input: this
        # contents of current my_incremental_model
        rows:
          - { uuid: p1,saved_timestamp: 2024-08-01 00:00:00,date_of_birth: 1980-01-01,sex: M}
          - { uuid: p2,saved_timestamp: 2024-08-01 00:00:00,date_of_birth: 1990-05-15,sex: F}
    expect:
      # what will be inserted/merged into my_incremental_model
      rows:
        - { uuid: p1,saved_timestamp: 2024-08-01 00:00:00,date_of_birth: 1980-01-01,sex: M}
        - { uuid: p2,saved_timestamp: 2024-08-01 00:00:00,date_of_birth: 1990-05-15,sex: F}

The output in dbt/logs/dbt.log:

[0m16:04:27.495212 [debug] [Thread-1 (]: Began running node unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental
16:04:27.496010 [info ] [Thread-1 (]: 33 of 35 START unit_test person::test_person_model_transformation_and_data_integrity_incremental  [RUN]
16:04:27.497186 [debug] [Thread-1 (]: Re-using an available connection from the pool (formerly unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity, now unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental)
16:04:27.498039 [debug] [Thread-1 (]: Began compiling node unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental
16:04:27.498479 [debug] [Thread-1 (]: Began executing node unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental
16:04:27.518147 [debug] [Thread-1 (]: Using postgres connection "unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental"
16:04:27.518832 [debug] [Thread-1 (]: On unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental: BEGIN
16:04:27.519298 [debug] [Thread-1 (]: Opening a new connection, currently in state closed
16:04:27.528900 [debug] [Thread-1 (]: SQL status: BEGIN in 0.009 seconds
16:04:27.529879 [debug] [Thread-1 (]: Using postgres connection "unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental"
16:04:27.530839 [debug] [Thread-1 (]: On unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental: /* {"app": "dbt", "dbt_version": "1.8.6", "profile_name": "default", "target_name": "test", "node_id": "unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental"} */
SELECT max(saved_timestamp) as max_existing
        FROM "".""
16:04:27.532267 [debug] [Thread-1 (]: Postgres adapter: Postgres error: zero-length delimited identifier at or near """"
LINE 3:         FROM "".""
                     ^

16:04:27.533124 [debug] [Thread-1 (]: On unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental: ROLLBACK
16:04:27.537942 [debug] [Thread-1 (]: On unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental: Close
16:04:27.543425 [debug] [Thread-1 (]: Database Error in unit_test test_person_model_transformation_and_data_integrity_incremental (models/contacts/tests/persons.yml)
  zero-length delimited identifier at or near """"
  LINE 3:         FROM "".""
                       ^
16:04:27.544162 [error] [Thread-1 (]: 33 of 35 ERROR person::test_person_model_transformation_and_data_integrity_incremental  [ERROR in 0.05s]
16:04:27.546269 [debug] [Thread-1 (]: Finished running node unit_test.cht_pipeline_base.person.test_person_model_transformation_and_data_integrity_incremental

The error might be related to this context in the macro not being correctly resolved or improperly formatted in the test env.