search

kedro-org / kedro-viz

Visualise your Kedro data and machine-learning pipelines and track your experiments.
https://demo.kedro.org
Apache License 2.0
648 stars 106 forks source link

Add `conf-source` to `kedro viz` CLI #1479

Open m-gris opened 11 months ago

m-gris commented 11 months ago

Hi,

It would be really handy and bring quite bit of flexibility if kedro viz could accept a conf-source argument.

As "rhetorical precaution", I must admit that this request is motivated by a bit of an un-usual way of organization our conf: We have a single pipeline / repo for all our clients and it seemed to us that the following organization of conf and data would be "safer" and provide a better "tenant isolation" (it also avoids issues such as duplicated keys in the clients' parameter files): 

conf
└── client_A
    ├── base
    └── local
└── client_B
    ├── base
    └── local
data
├── client_A
│   └── 01_raw
│   └── 02_intermediate
│   └── ...
└── client_B
│   └── 01_raw
│   └── 02_intermediate
│   └── ...

I hope that, in spite of the idiosyncratic nature of the above, you will consider this request as relevant.

Many thanks in advance, Regards, M.

yetudada commented 11 months ago

Ah, this is interesting. So kedro viz supports --env flag which would have allowed you to run kedro viz --env client_a. However, you've also included that you've structured your data in the same way. Is this that you expect to see the data related to client_a when you run kedro viz --env client_a? What happens when you do this? Do you see the right data?

On the Kedro side, we use conf-source to specify where to load configuration from, if the conf directory is not in a packaged Kedro project. env is used to choose your different directory folders.

m-gris commented 11 months ago

Thx @yetudada for your message.

Unfortunately, given the way I've organized the conf, the --env flag does work.

Bother kedro viz --env client_a and kedro viz --env client_a/base result in 

kedro viz --env client_a
Traceback (most recent call last):
  File "/home/vscode/.pyenv/versions/3.10.7/envs/kedro_dodo_ingestion/lib/python3.10/site-packages/kedro_viz/launchers/cli.py", line 134, in viz
    run_server(**run_server_kwargs)
  File "/home/vscode/.pyenv/versions/3.10.7/envs/kedro_dodo_ingestion/lib/python3.10/site-packages/kedro_viz/server.py", line 83, in run_server
    catalog, pipelines, session_store = kedro_data_loader.load_data(
  File "/home/vscode/.pyenv/versions/3.10.7/envs/kedro_dodo_ingestion/lib/python3.10/site-packages/kedro_viz/integrations/kedro/data_loader.py", line 81, in load_data
    with KedroSession.create(
  File "/home/vscode/.pyenv/versions/3.10.7/envs/kedro_dodo_ingestion/lib/python3.10/site-packages/kedro/framework/session/session.py", line 194, in create
    session._setup_logging()
  File "/home/vscode/.pyenv/versions/3.10.7/envs/kedro_dodo_ingestion/lib/python3.10/site-packages/kedro/framework/session/session.py", line 212, in _setup_logging
    logging_config = self._get_logging_config()
  File "/home/vscode/.pyenv/versions/3.10.7/envs/kedro_dodo_ingestion/lib/python3.10/site-packages/kedro/framework/session/session.py", line 201, in _get_logging_config
    logging_config = self._get_config_loader()["logging"]
  File "/home/vscode/.pyenv/versions/3.10.7/envs/kedro_dodo_ingestion/lib/python3.10/site-packages/kedro/framework/session/session.py", line 293, in _get_config_loader
    return config_loader_class(
  File "/home/vscode/.pyenv/versions/3.10.7/envs/kedro_dodo_ingestion/lib/python3.10/site-packages/kedro/config/templated_config.py", line 138, in __init__
    _get_config_from_patterns(
  File "/home/vscode/.pyenv/versions/3.10.7/envs/kedro_dodo_ingestion/lib/python3.10/site-packages/kedro/config/common.py", line 73, in _get_config_from_patterns
    raise ValueError(
ValueError: Given configuration path either does not exist or is not a valid directory: /workspaces/kedro-dodo-ingestion/conf/base
kedro.framework.cli.utils.KedroCliError: Given configuration path either does not exist or is not a valid directory: /workspaces/kedro-dodo-ingestion/conf/base
Run with --verbose to see the full exception
Error: Given configuration path either does not exist or is not a valid directory: /workspaces/kedro-dodo-ingestion/conf/base
astrojuanlu commented 1 month ago

xref https://github.com/kedro-org/kedro-plugins/issues/672 (different target, similar functionality)