search

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
9.92k stars 901 forks source link

Configuring a packaged Kedro project while using a custom logging config with File Handlers raises error #4222

Open cariveroco opened 1 week ago

cariveroco commented 1 week ago

Description

Importing configure_project from kedro.framework.project fails when using a custom Kedro logging config with File Handlers, if the target files of the handlers do not exist yet. The log files do not get automatically created.

The FileNotFoundError occurs when importing project, session, or startup from kedro.framework.

Context

I intend to run pipelines from a packaged Kedro Project in a "clean" directory (eventually: in a docker container) using a custom logger with Rotating File Handlers. I installed the packaged Kedro project and tried to run pipelines via script, which includes first configuring the project using kedro.framework.project.configure_project. Since the working directory is "clean" (i.e., does not have log files yet), I expected the log files to be automatically created, as is the observed behavior when I run pipelines via CLI in my local machine (where I have direct access to the source code in the Project Directory). Instead, I'm getting FileNotFoundError.

Steps to Reproduce

  1. Start with a clean directory/environment.
  2. Set up a custom_logging_config.yml file with a Rotating File Handler (e.g. level: INFO, class: logging.handlers.RotatingFileHandler, filename: logs/info.log)
  3. Point the environment variable KEDRO_LOGGING_CONFIG to the previously created file.
  4. Import session or project or startup from kedro.framework

Expected Result

The log files automatically get created and the execution proceeds to running the pipelines.

Actual Result

FileNotFoundError encountered.

INFO:kedro.framework.project:Using 'custom_logging_config.yml' as logging configuration. 
--- Logging error ---
Traceback (most recent call last):
  File "/Users/PQ27WN/.pyenv/versions/3.11.9/lib/python3.11/logging/handlers.py", line 73, in emit
    if self.shouldRollover(record):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/PQ27WN/.pyenv/versions/3.11.9/lib/python3.11/logging/handlers.py", line 194, in shouldRollover
    self.stream = self._open()
                  ^^^^^^^^^^^^
  File "/Users/PQ27WN/.pyenv/versions/3.11.9/lib/python3.11/logging/__init__.py", line 1213, in _open
    return open_func(self.baseFilename, self.mode,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Users/PQ27WN/AFv311/exp-build-package/logs/info.log'

Your Environment

rashidakanchwala commented 1 week ago

Hi @cariveroco,

Thank you for raising this issue! We appreciate your input. We’ll take a closer look and get back to you shortly with any updates or follow-up questions.