search

kedro-org / kedro-plugins

First-party plugins maintained by the Kedro team.
Apache License 2.0
91 stars 85 forks source link

Logging seems to freeze or break airflow #603

Open noklam opened 6 months ago

noklam commented 6 months ago

Description

Short description of the problem here.

  1. https://github.com/kedro-org/kedro-plugins/issues/13#issuecomment-1088710755 It was first found that logging is breaking kedro-airflow sometimes, this causing the airflow nodes fail immediately during the execution of first node. The workaround is documented in the link.

  2. A new symptoms seem to be rich is interacting with airflow. @DimedS : When deploying a Kedro project to Airflow using Astronomer, as detailed in this manual, an issue was encountered where the first node would fail after a few minutes, culminating in a{task_context_logger.py:104} ERROR - Detected zombie job error message.

The problem was resolved by modifying the logging.yml file to utilize custom logging settings. from

root:
  handlers: [rich, info_file_handler]

to 

root:
  handlers: [console]
Screenshot 2024-03-12 at 12 23 12

Context

How has this bug affected you? What were you trying to accomplish?

Steps to Reproduce

  1. [First Step]
  2. [Second Step]
  3. [And so on...]

Expected Result

Tell us what should happen.

Actual Result

Tell us what happens instead.

-- If you received an error, place it here.
-- Separate them if you have more than one.

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

astrojuanlu commented 6 months ago

The pressure is mounting to do tackle https://github.com/kedro-org/kedro/issues/2928

astrojuanlu commented 6 months ago

Also I'm almost sure I've been affected by this and the only workaround I found was using a Dockerized version of Airflow instead of the pip install I tried initially.

noklam commented 6 months ago

@astrojuanlu Do you know why would a dockerized version help? Good to know this as a workaround though.

noklam commented 6 months ago

The pressure is mounting to do tackle kedro-org/kedro#2928

This may solve problem 2, but not 1 (not sure if it still persists), from experience it also depends on the airflow version. I tried to look at both rich or airflow Github repo and couldn't find any related problem, is it worth to do some investigation to see if this is an issue created by Kedro instead?

em-pe commented 5 months ago

@noklam Not sure how much of help this would be but I managed to fix both of the issues disabling kedro logging at all by commenting out following line.

rich made the pipelinie stuck and "disable_existing_loggers": True made the configure_project fail with -1 error code and no error logs. When you disable logging changes done by kedro everything seems to work fine, including logging information from airflow tasks.

noklam commented 5 months ago

@em-pe I think you are right I found that previously though the root cause is still unknown.