Question / request: Customize name of files

[jaredx435k2d0]

The docs mention I have a few different config options I can use for the name of my yml files, such as:

+dbt-osmosis: "folder.yml" +dbt-osmosis: "schema/model.yml"

Is there any way to further configure this? For example, our project has the convention of having __sources.yml as the end of our sources file names.

I'm not sure if there's already a way to do it. I tried a few things but was unsuccessful.

[jaredx435k2d0]

Actually, it sort of seems like it's already doing this if that file already exists? Not totally sure on how this is working, but it did update my existing file that ended in __sources.yml, so I think that's good enough for me!

[jaredx435k2d0]

I've found that if I have a yml file for my staging models, that file name is not preserved, but rather "superseded" (as stated in the console output):

Before: _stg_my_schema.yml

After: my_schema.yml

I'd like to request that the file names not be changed

[jaredx435k2d0]

I'm definitely a Python rookie, but I tried adding schema = "_stg_" + schema immediately before the return in get_target_schema_path() and that seemed to do the trick for me for now.

[volderette]

Hi,

I love dbt-osmosis and I would be interested in having a feature like this as well. Right now it's not possible to follow dbt's best practices:

models
├── intermediate
│   └── finance
│       ├── _int_finance__models.yml
│       └── int_payments_pivoted_to_orders.sql
├── marts
│   ├── finance
│   │   ├── _finance__models.yml
│   │   ├── orders.sql
│   │   └── payments.sql
│   └── marketing
│       ├── _marketing__models.yml
│       └── customers.sql
├── staging
│   ├── jaffle_shop
│   │   ├── _jaffle_shop__docs.md
│   │   ├── _jaffle_shop__models.yml
│   │   ├── _jaffle_shop__sources.yml
│   │   ├── base
│   │   │   ├── base_jaffle_shop__customers.sql
│   │   │   └── base_jaffle_shop__deleted_customers.sql
│   │   ├── stg_jaffle_shop__customers.sql
│   │   └── stg_jaffle_shop__orders.sql
│   └── stripe
│       ├── _stripe__models.yml
│       ├── _stripe__sources.yml
│       └── stg_stripe__payments.sql
└── utilities
    └── all_dates.sql

https://docs.getdbt.com/guides/best-practices/how-we-structure/5-the-rest-of-the-project

[z3z1ma]

I think I can support f-strings so users can really do whatever they want so dbt-osmosis: _{model} or dbt-osmosis: /{folder}/_{model}, etc

[jaredx435k2d0]

This would be great!

[z3z1ma]

This is done!

@volderette To achieve this:

models
├── intermediate
│   └── finance
│       ├── _int_finance__models.yml
│       └── int_payments_pivoted_to_orders.sql
├── marts
│   ├── finance
│   │   ├── _finance__models.yml
│   │   ├── orders.sql
│   │   └── payments.sql
│   └── marketing
│       ├── _marketing__models.yml
│       └── customers.sql
├── staging
│   ├── jaffle_shop
│   │   ├── _jaffle_shop__docs.md
│   │   ├── _jaffle_shop__models.yml
│   │   ├── _jaffle_shop__sources.yml
│   │   ├── base
│   │   │   ├── base_jaffle_shop__customers.sql
│   │   │   └── base_jaffle_shop__deleted_customers.sql
│   │   ├── stg_jaffle_shop__customers.sql
│   │   └── stg_jaffle_shop__orders.sql
│   └── stripe
│       ├── _stripe__models.yml
│       ├── _stripe__sources.yml
│       └── stg_stripe__payments.sql
└── utilities
    └── all_dates.sql

You would have this:

models:
  your_project:
    intermediate:
      +dbt-osmosis: _int_{parent}__models.yml

    marts:
      +dbt-osmosis: _{parent}__models.yml

    staging:
      +dbt-osmosis: _{parent}__models.yml
      jaffle_shop:
        base:
          +dbt-osmosis: _{parent}_{node.fqn[-3]}__models.yml
          # or hard coding is now valid!
          # +dbt-osmosis: _base_jaffle_shop__models.yml

[volderette]

Thank you so much! This looks great and super flexible!

[ryantimjohn]

@z3z1ma when I use a variable like this in the dbt-osmosis config {node.fqn[-3]}, as you suggest in: +dbt-osmosis: _{parent}_{node.fqn[-3]}__models.yml I get this error: TypeError: list indices must be integers or slices, not str

Wonder if this might be a python version error or a dbt version error if this is working for everyone else. Doubt it's dbt-osmosis version error as I just installed. Cheers! -tim

[ryantimjohn]

here's full stack trace:

  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/bin/dbt-osmosis", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/dbt_osmosis/main.py", line 80, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/dbt_osmosis/main.py", line 172, in refactor
    if runner.commit_project_restructure_to_disk():
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/dbt_osmosis/core/osmosis.py", line 593, in commit_project_restructure_to_disk
    blueprint = self.draft_project_structure_update_plan()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/dbt_osmosis/core/osmosis.py", line 569, in draft_project_structure_update_plan
    for unique_id, schema_file in self.build_schema_folder_mapping().items():
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/dbt_osmosis/core/osmosis.py", line 456, in build_schema_folder_mapping
    osmosis_schema_path = self.get_target_schema_path(dbt_node)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tim/.local/pipx/.cache/cc6f81ab30b23c6/lib/python3.11/site-packages/dbt_osmosis/core/osmosis.py", line 269, in get_target_schema_path
    schema = osmosis_path_spec.format(node=node, model=node.name, parent=node.fqn[-2])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: list indices must be integers or slices, not str