Bug: Disabled sources are cleaned up if source location is set in dbt_project vars

[kokorin]

Intro

In our project we have to support different set of sources for dev and prod envs. We found a bug with dbt-osmosis yaml refactor: disabled sources are corrupted if source file location is set in dbt_project.yml like this:

vars:
  dbt-osmosis:
    dev: "staging/_dev_sources.yml"
    prod: "staging/_prod_sources.yml"

Here is a file containing minimal DBT project reproducing behavior described above: dbt-osmosis-disabled-source-bug.zip

How to reproduce

1. python -m venv .venv
2. .\.venv\Scripts\activate or source .venv/bin/activate
3. pip install -r requirements.txt
4. dbt build
5. dbt build --target prod
6. check _prod_sources.yml content, it should be

version: 2
sources:
  - name: prod
    database: prod
    schema: main
    config:
      enabled: "{{ target.name == 'prod' }}"
    tables: 
      - name: users

7. dbt-osmosis yaml refactor
8. check _prod_sources.yml content, it now contains no tables and config is wrong:

version: 2
sources:
  - name: prod
    database: dev
    schema: prod
    tables: []

9. check _dev_sources.yml content, it's updated correctly:

   version: 2
   sources:
   - name: dev
   database: dev
   schema: main
   config:
     enabled: "{{ target.name == 'dev' }}"
   tables:
     - name: users
       columns:
         - name: id
           description: ''
           data_type: INT
         - name: name
           description: ''
           data_type: TEXT
         - name: nickname
           description: ''
           data_type: TEXT

[z3z1ma]

I think you would need to replicate the dynamism in the osmosis config.

vars:
  dbt-osmosis:
    dev: "{{ 'staging/_dev_sources.yml' if target.name == 'dev' else None }}"
    prod: "{{ 'staging/_prod_sources.yml' if target.name == 'prod' else None }}"

Not 100% sure this would work but something along these lines, might need to point a gitignored file. The source synchronization is meant to both add new tables from a source but also remove stuff that no longer exists. I don't see why we couldn'e have a flag to opt out of any behavior like that though.