search

astronomer / airflow-provider-kafka

A provider package for kafka
Apache License 2.0
37 stars 16 forks source link

Push data from ConsumeFromTopicOperator to XCom #33

Closed manojmanivannan closed 1 year ago

manojmanivannan commented 1 year ago

Can the messages consumed from ConsumeFromTopicOperator be pushed to XCom ?

dylanbstorey commented 1 year ago

Should be possible but as a rule I wouldn't suggest it.

manojmanivannan commented 1 year ago

Any suggestions on how to move records from Kafka topics ? I'm trying to find a solution for this case https://stackoverflow.com/questions/76372178/how-to-push-data-records-from-consumefromtopicoperator-to-xcom

dylanbstorey commented 1 year ago

I'd start to see if

https://airflow.apache.org/docs/apache-airflow-providers-apache-kafka/1.1.0/

Helps you out. That's the officially supported provider now.

Specifically

https://airflow.apache.org/docs/apache-airflow-providers-apache-kafka/1.1.0/sensors.html#awaitmessagetriggerfunctionsensor

Might be of interest.

Also, why can't you apply the work you're doing within the consume from topic operator directly ?

manojmanivannan commented 1 year ago

@dylanbstorey thanks so much. After more digging, I was able to simply the approach and implement using Kafka consumer from kafka-python.