search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
774 stars 170 forks source link

[Feature] dbt source rendering: Use `.` instead of `_` between schema & table #1308

Open corsettigyg opened 2 weeks ago

corsettigyg commented 2 weeks ago

Description

Currently in cosmos, sources are rendered as emptyOperators with a taskid of `{schema}{table}_source`

Image

The main issue with this naming convention is that the usual convention of naming tables with the schema uses dot (.) instead of underscore (_), and by using underscore in the naming of our tasks, it is not possible to properly identify where the schema name ends and where the table name begins.

Use case/motivation

By having a logical separation of the schema and the table, it is easy to extend cosmos and add extra features to the sources. At my company we are adding sensors before the sources, but the process becomes way harder because of something as simple as using the _ instead of .in the name of the task.

Related issues

No response

Are you willing to submit a PR?

tatiana commented 2 weeks ago

@corsettigyg, thanks for your feedback! Would this address the issue if we were to use __ instead of _ as a separator between schema & table?

corsettigyg commented 1 week ago

@tatiana totally

tatiana commented 5 days ago

I just discussed this with @pankajkoti and @pankajastro, and I was concerned about making this change backwards compatible.

The preferred approach is to make it backwards compatible by having yet another configuration—and having a ticket to make the default __ in Cosmos 2.0. We'll squeeze in this change in Cosmos 1.8. The configuration could be within RenderConfig.

@corsettigyg, would you by any chance have bandwith of contributing to this?