Empty run command in Experiment Tracking if pipeline is executed from a Jupyter notebook

[ChristopherRabotin]

Description

In our workflow, we've designed basic Jupyter Notebooks as UIs to run Kedro pipelines. However, when we run the pipeline through the Python script that Jupyter calls, there is no run command in the Experiment Tracking. In a way that is understandable because we didn't run a command, we called the Kedro session and ran a specific pipeline with specific parameters. But we would like to be able to use the experiment tracking to follow the work done by the whole team.

Context

I was hoping to see the pipeline name, parameters, and potentially even the versions of the datasets used for that pipeline execution.

Steps to Reproduce

1. Enable experiment tracking
2. Set up a pipeline, run it with kedro run -p my_pipeline and check that this run shows up in the experiment tracking panel
3. Run the pipeline from a Python script, e.g.

        with KedroSession.create(
            project_path=self.prj_path, extra_params=params
        ) as session:
            session.run(pipeline_name="pipeline_name", load_versions=versions)

4. Restart kedro viz
5. Notice that the "run command" is empty in the latest run

Expected Result

I would expect to see the run command of the run.

Actual Result

Your Environment

• Windows 10
• Python 3.10.5
• Kedro-viz 9.1.0
• Experiment tracking set up via S3

[astrojuanlu]

@ChristopherRabotin we discussed about this today, this might be a limitation on KedroSession. We will do a short investigation and see where it leads us.

[astrojuanlu]

I was hoping to see the pipeline name, parameters, and potentially even the versions of the datasets used for that pipeline execution.

In the meantime, I know I sound like a broken record but... 😄

https://docs.kedro.org/en/latest/integrations/mlflow.html#easy-tracking-of-kedro-runs-in-mlflow-using-kedro-mlflow

[ChristopherRabotin]

We'll be looking into how we could use mlflow. We did a quick demo last week and it looks promising.

[rashidakanchwala]

@ChristopherRabotin, just wondering what did you end up doing and if u need anymore support on this issue?

[ChristopherRabotin]

Hi there,

We're considering moving to ML Flow for the visualization of the executed pipelines. We're busy finishing up some work so we likely won't be trying out mlflow until February or March.

[astrojuanlu]

Thanks @ChristopherRabotin. That being the case, and given the status of Experiment Tracking in Viz, we're closing this as "won't fix".