Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
When having parameters in conf/base/sub_folder/parameters_A.yml structure where sub_folder is symlinked, the pipeline gives the following error, however, it is able to read the parameters from the following structure: conf/base/parameters_A.yml
ValueError: Pipeline input(s) {'params:length', 'params:width'} not found in the DataCatalog
Context
The issue is that fsspec.filesystem.glob() which we use to find the paths recursively doesn’t find symlinks.
Description
When having parameters in
conf/base/sub_folder/parameters_A.ymlstructure wheresub_folderis symlinked, the pipeline gives the following error, however, it is able to read the parameters from the following structure:conf/base/parameters_A.ymlValueError: Pipeline input(s) {'params:length', 'params:width'} not found in the DataCatalogContext
The issue is that
fsspec.filesystem.glob()which we use to find the paths recursively doesn’t find symlinks.https://github.com/kedro-org/kedro/blob/adfc593bcd2f1b74676e7ab7c1a3b9c168b7257f/kedro/config/omegaconf_config.py#L295
Steps to Reproduce
For a default spaceflights-pandas project create a symlink folder and place
parameters_data_science.ymlin the linked folder.Run the pipeline.
Expected Result
Symlinks are found when loading configuration files.
Your Environment
pip show kedroorkedro -V): kedro, version 0.19.6python -V): Python 3.11.9