dbt-labs / dbt-adapter-tests

a pytest plugin for dbt adapter test suites
19 stars 11 forks source link

Quoting in dbt_project_yml not used correctly #14

Closed aurany closed 3 years ago

aurany commented 3 years ago

I'm trying to create an adapter for IBM DB2. In my spec-file i have

projects:
  - overrides: base
    paths:
      models/view_model.sql: |
        {{ config(materialized='view') }}
        select * from sysibm.sysdummy1
    dbt_project_yml:
      quoting:
        database: false
        schema: true
        identifier: true

This is working in some cases/tests, for example

CREATE VIEW testdb."dbt_test_210210204959161102758925"."view_model__dbt_tmp" AS

select * from sysibm.sysdummy1

_EDIT: The reason for overriding the viewmodel is to get rid of the default quoting.

...but in some cases/tests it goes wrong. This case/test will fail because database is quoted and the identifier is not.

select count(*) as num_rows from "testdb"."dbt_test_210210204959161102758925".base

I assume something goes wrong in the _relation_from_name function but i'm not sure how to fix it, and I do not understand why this happens only in some tests.

// R

jtcohen6 commented 3 years ago

Hey @aurany, thanks for trying out the adapter testing suite! Cool to hear you're building an IBM DB2 adapter.

You're right, the issue seems to be cropping up in step_relations_equal or step_relation_rows, which (to be honest) are doing a little more in python string compilation than they probably should, to avoid requiring the addition of many additional test-only adapter methods just for checking equality/cardinality. Rather than having dbt render the relation name, these methods instead call a _relation_from_name method bespoke to this test suite:

https://github.com/fishtown-analytics/dbt-adapter-tests/blob/a411336e9493a38c228a876e19af24d70ec53d86/pytest_dbt_adapter/spec_file.py#L167-L210

The _relation_from_name method does not consider the quoting set in dbt_project.yml. It does, however, consider the adapter's default quote_policy. If IBM DB2 never supports database quoting, but always supports schema and identifier quoting, you're much better off setting that within your adapter, rather than needing to override it within the specfile (and, by extension, needing to set it manually in every project that uses your adapter).

E.g. Here is where dbt-snowflake sets a default quote policy of False across the board; the SnowflakeAdapter then pulls in the SnowflakeRelation here.

aurany commented 3 years ago

Thanks @jtcohen6, it's a much better option to set default quoting in the adapter.