rexledesma
commented
3 months ago We have a new way of collecting this column level metadata without the need for the Dagster dbt package. Mind trying it out to see if this resolves your issue?
Instead of using the log_column_level_metadata macro, you can just specify that you want the column level metadata in python instead, by specifying .fetch_column_metadata().
You'll need dagster>=1.7.9 and dagster-dbt>=0.23.9 to try this out.
from dagster import AssetExecutionContext
from dagster_dbt import dbt_assets, DbtCliResource
@dbt_assets(manifest=...)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["build"], context=context).stream().fetch_column_metadata()
maxfirman
Dagster version
1.7.8
What's the issue?
This line of code raises the following exception when running against the dbt-dremio adaptor:
The problem arises because the
refmacro has been overridden in the dbt-dremio adaptor (see here), and thelog_column_level_metadatamacro calls the ref macro with a "package" keyword argument that doesn't exist in the overridden macro.What did you expect to happen?
The
log_column_level_metadatamacro should not callrefusing the "package" keyword argument. This method of calling ref is not documented as part of the public api. See the dbt documentation for examples for how ref is expected to be called.How to reproduce?
Might be hard to reproduce without standing up a Dremio instance.
Deployment type
Dagster Cloud
Deployment details
No response
Additional information
I'm happy to work on a fix for this issue, although I won't be able to make a start for the next week or so.
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.