kedro-org / kedro-viz

Visualise your Kedro data and machine-learning pipelines and track your experiments.
https://demo.kedro.org
Apache License 2.0
672 stars 110 forks source link

Relax dependabot strategy #1967

Closed astrojuanlu closed 2 months ago

astrojuanlu commented 3 months ago

Description

In https://github.com/kedro-org/kedro-viz/issues/1582#issuecomment-1764451560 I wrote

Comment from the peanut gallery: since Kedro-Viz dependabot is using the default configuration, versioning-strategy is set to auto, meaning

Try to differentiate between apps and libraries. Use increase for apps and widen for libraries.

https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#versioning-strategy

and since Kedro-Viz uses requirements.txt for its Python requirements, it's probably applying the increase strategy, which is arguably too aggressive for a component like Kedro-Viz. In fact, since Kedro switched to pyproject.toml, dependabot PRs have been using the widen strategy.

Viz can either

* Do nothing and keep aggressively upgrading dependencies, but this can lead to problems with people that have environments with lots of dependencies.

* Configure dependabot to use `widen`.

* Migrate dependencies to `pyproject.toml` [Move to pyproject.toml #1527](https://github.com/kedro-org/kedro-viz/issues/1527) and keep the `auto` config.

Since the migration to pyproject.toml hasn't happened, Kedro-Viz continued to use the increase dependabot strategy, hence making too aggressive updates to requirements.txt files.

Context

This has caused trouble to me. Today I was trying to install Kedro-Viz alongside Prefect 3.0.0rc10, and got this error:

$ uv pip compile --universal requirements.in -o requirements.txt
  × No solution found when resolving dependencies:
  ╰─▶ Because kedro-viz==9.1.0 depends on uvicorn[standard]>=0.29.0,<0.30.dev0 and only the following versions
      of uvicorn[standard] are available:
          uvicorn[standard]<=0.29.0
          uvicorn[standard]>=0.30.dev0
      we can conclude that kedro-viz==9.1.0 depends on uvicorn[standard]==0.29.0.
      And because prefect==3.0.0rc10 depends on one of:
          uvicorn>=0.14.0,<0.29.0
          uvicorn>0.29.0
      and only the following versions of prefect are available:
          prefect<=3.0.0rc10
          prefect>=3.1.dev0
      we can conclude that kedro-viz==9.1.0 and prefect>=3.0.0rc10,<3.1.dev0 are incompatible.
      And because you require kedro-viz==9.1.0 and prefect>=3.0.0rc10, we can conclude that the requirements
      are unsatisfiable.

Kedro-Viz has such strict dependency pinnings because of Dependabot https://github.com/kedro-org/kedro-viz/pull/1839

The dependency was bumped in the main branch already https://github.com/kedro-org/kedro-viz/pull/1935 but since development installs are tricky https://github.com/kedro-org/kedro-viz/issues/1611 I cannot just run uv pip install "kedro-viz @ git+https://github.com/kedro-org/kedro-viz.git@main#subpackage=package", because that yields a broken installation:

$ uv pip install "kedro-viz @ git+https://github.com/kedro-org/kedro-viz.git@main#subdirectory=package"
 Updated https://github.com/kedro-org/kedro-viz.git (35d351f)
Resolved 103 packages in 4.48s
   Built kedro-viz @ git+https://github.com/kedro-org/kedro-viz.git@35d351f03172032876a9f6571f9b9c31145d7417#subdirec
Prepared 1 package in 1.49s
Uninstalled 2 packages in 29ms
Installed 2 packages in 16ms
 - kedro-viz==9.1.0
 + kedro-viz==9.1.0 (from git+https://github.com/kedro-org/kedro-viz.git@35d351f03172032876a9f6571f9b9c31145d7417#subdirectory=package)
 - uvicorn==0.29.0
 + uvicorn==0.30.1
$ kedro viz run
...
    FileNotFoundError: [Errno 2] No such file or directory:                             
                             '/Users/juan_cano/Projects/QuantumBlackLabs/workshop-from-zero                      
                             -to-mlops/.venv/lib/python3.11/site-packages/kedro_viz/html/in                      
                             dex.html' 

There are two solutions:

astrojuanlu commented 3 months ago

For the record, despite the broken dependency requirements, Kedro-Viz 9.1.0 works with uvicorn 0.30.1 just fine:

$ uv pip freeze | grep -E 'viz|uvi'
kedro-viz==9.1.0
uvicorn==0.30.1
$ uv pip check
Checked 215 packages in 2ms
Found 1 incompatibility
The package `kedro-viz` requires `uvicorn[standard]~=0.29.0`, but `0.30.1` is installed.
$ kedro viz run
[07/04/24 09:41:40] INFO     Using `conf/logging.yml` as logging configuration. You can change    __init__.py:249
                             this by setting the KEDRO_LOGGING_CONFIG environment variable                       
                             accordingly.                                                                        
Starting Kedro Viz ...
[07/04/24 09:41:42] INFO     Using `conf/logging.yml` as logging configuration. You can change    __init__.py:249
                             this by setting the KEDRO_LOGGING_CONFIG environment variable                       
                             accordingly.                                                                        
Kedro Viz started successfully. 

✨ Kedro Viz is running at 
 http://127.0.0.1:4141/

Probably after the Dependabot config is changed, some effort could be spent relaxing some of the dependency pinnings.