datafold / data-diff

Compare tables within or across databases
https://docs.datafold.com
MIT License
2.95k stars 272 forks source link

Support Postgres materialized views #866

Closed rubenhelsloot closed 7 months ago

rubenhelsloot commented 9 months ago

Describe the bug I want to compare two materialized views built using DBT, but when I run data-diff --dbt --select <my model>, I get "New model or no access to prod table." The reason is that is that in Postgres, materialized views are not stored in the information_schema.columns schema, which is what is queried by data-diff to find the column:

SELECT column_name, data_type, datetime_precision,                                                                                 
    -- see comment for DEFAULT_NUMERIC_PRECISION                                                                   
    CASE                                                                                                           
        WHEN data_type = 'numeric'                                                                                 
            THEN coalesce(numeric_precision, 131072 + 16383)                                                       
        ELSE numeric_precision                                                                                     
    END AS numeric_precision,                                                                                      
    CASE                                                                                                           
        WHEN data_type = 'numeric'                                                                                 
            THEN coalesce(numeric_scale, 16383)                                                                    
        ELSE numeric_scale                                                                                         
    END AS numeric_scale                                                                                           
    FROM information_schema.columns
    WHERE table_name = '<my model>' AND table_schema = 'intermediate_prod'

Instead, we'd need to query pg_matviews, as noted in this StackOverflow answer.

I figure we could solve this issue by unioning the first query with another one that queries pg_matviews.

Describe the environment I'm running data-diff v0.11.0 with dbt-core and dbt-postgres 1.7.7 on Postgres 15.

github-actions[bot] commented 7 months ago

This issue has been marked as stale because it has been open for 60 days with no activity. If you would like the issue to remain open, please comment on the issue and it will be added to the triage queue. Otherwise, it will be closed in 7 days.

github-actions[bot] commented 7 months ago

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment and it will be reopened for triage.