search

astronomer / astronomer-cosmos

Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code
https://astronomer.github.io/astronomer-cosmos/
Apache License 2.0
657 stars 170 forks source link

[Feature] Publish dbt run result to XCom #1073

Open ghjklw opened 4 months ago

ghjklw commented 4 months ago

Description

It seems that the build_and_run_cmd for all operators except ACI return something. I am unsure whether this is consistently the result of the dbt runner across all operators but these are data that could be interesting to publish through XCom.

I have limited experience with Airflow custom operators, but I believe that if AbstractDbtBaseOperator.execute simply returned the output of self.build_and_run_cmd or pushed it using context['ti'].xcom_push , it would do the trick.

This could leverage Airflow's BaseOperator do_xcom_push attribute.

If this feature is implemented, it might be useful to document the return value depending on the execution mode being used.

Use case/motivation

I would be interested in, for example, persisting these data for monitoring purposes. They could also be really useful when using a custom node converter.

Related issues

https://github.com/astronomer/astronomer-cosmos/issues/976

Are you willing to submit a PR?

tatiana commented 4 months ago

@ghjklw as you mentioned, I think this issue relates to both of the following

I believe Cosmos should not return this as part of execute (we actually cleared this up in a previous version, with #304). The reason is that we can trash the Airflow metadata database with data that users may not need or care for.

I prefer the approaches described in #867 and #976, from the perspective that users who want to output this to the Airflow metadata would be allowed to - but this would not be enforced to all users.

Would you be interested in working on this feature?

ghjklw commented 4 months ago

@tatiana, I understand the reasoning behind not doing it that way. I can indeed take a look at how that's achieved for on_warning_callback.

I'd be happy to give it a go, just be aware that I'm on vacation right now and will have a limited bandwidth in August + I will have to get familiar with Cosmos' development environment for my first contribution, du or bought take a bit of time. If that's ok, I'll get in touch to clarify how this could be implemented.

pankajkoti commented 1 week ago

Reopening as "dosubot" marked it as stale & closed it prematurely.