search

dbt-labs / dbt-adapter-tests

a pytest plugin for dbt adapter test suites
19 stars 11 forks source link

Adapter that does not have a concept of 'database' #27

Closed samecrowder closed 3 years ago

samecrowder commented 3 years ago

Hi team,

I'm developing the new dbt Rockset adapter, which lives here https://github.com/rockset/dbt-rockset. We have basic functionality working on the adapter, and I am now trying to get some adapter tests working.

The problem is that Rockset does not support the notion of a database. There are simply workspaces (which are schemas) and collections (which are tables). I had to write some overrides in the adapter to get the standard ref() model to work so that references would resolve without including the database field (so the SQL query just says SELECT * FROM <schema>.<relation>). That override is made here: https://github.com/rockset/dbt-rockset/blob/f16b8a3e783488e645362726296e43a2eca4358a/dbt/include/rockset/macros/adapters.sql#L13.

The problem is the adapter tests are trying to select from <db>.<schema>.<identifier>. The database name that is being used is what I configured here https://github.com/rockset/dbt-rockset/blob/f16b8a3e783488e645362726296e43a2eca4358a/dbt/adapters/rockset/connections.py#L41. I had to write this hack bc it seems that many places in dbt-core require a database to be present in the adapter class.

How can I allow the adapter tests to write queries that do no include database? All queries are failing with the current structure.

Thanks!! Sam

samecrowder commented 3 years ago

It appears that creating a custom include_policy in class RocksetRelation has resolved the issue

jtcohen6 commented 3 years ago

@samecrowder Right on. I'm very glad to hear that setting the includ_policy within the adapter got this working for the test suite as well. (FWIW it does sound like there's an issue with the current test suite's inheritance of the adapter's quote policy: https://github.com/dbt-labs/dbt-adapter-tests/issues/16.)

In general, I'd point you to dbt-spark as the canonical implementation of a "no database" adapter. In Apache Spark, schema and database are interchangeable names for the second-level hierarchical thing; to avoid confusion with other adapters, where database means "third-level thing," we insist on calling it schema.