Pixi lock file is always out-of-date after introducing `pypi-dependencies`

Hofer-Julian commented 5 months ago

Checks

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

git clone -b pixi-install --single-branch https://github.com/Deltares/Ribasim.git
cd Ribasim
pixi install
pixi install --locked

The last command will fail with × lock-file not up-to-date with the project I can reproduce this behavior on Linux and Windows machines.

Issue description

Something seems to be fishy with our pixi setup after @Jingru923 moved to using pypi-dependencies: https://github.com/Deltares/Ribasim/pull/1400 Pixi always seem to consider the lock file out-of-date. I would love to make a smaller reproducer, but many other projects use pypi-dependencies just fine, and I don't see what makes the Ribasim setup special.

Expected behavior

Pixi should be able to update the lockfile in a way, that it afterwards is able to recognice that the lockfile is up-to-date.

tdejager commented 5 months ago

So I found out why this is. Note if you run with -v, this gives you the reason why the dependencies are out of date, in this case because it did not expect ribasim to be editable, which at first glance seems strange, because it is defined as editable in the toml.

I think because this project is using a nested structure to include multiple projects this problem occurs.

The pixi.toml specifies ribasim as editable.
The pyproject.toml of ribasim_testmodels specifies ribasim as a regular pypi dependency.

Editables and requirements are kind of disjunct in pypi world. So this may cause issues. I'll need to investigate as bit further.

niemiaszek commented 5 months ago

I'm getting similar issue when trying to use my env:

[dependencies]
python = "~=3.11.0"
pandas = "*"
pyyaml = "*"
tqdm = "*"
ffmpeg = "*"
ruff = "*"
pre-commit = "*"
srt = "*"

[pypi-dependencies]
ffmpeg-python = {git= "https://github.com/charlienewey/ffmpeg-python.git", branch = "bugfix/782", rev="37dd98ca210fb8132575bedd719c6f8f9bfd1021"}
tensorflow ="~=2.15.0"
tensorflow-addons = "*"
tensorflow-io = "*"
tensorflow-datasets = "*"
keras-cv = "*"
pytest = "*"
pytest-cov = "*"
clearml = "*"

[feature.train.system-requirements]
cuda = "12.2"

[feature.train.tasks]
setup_precommit = "pre-commit install"
setup_jupyter = "python -m ipykernel install --user --name 'speaker-recognition-pixi'"
test_gpu = {cmd = ["python3", "-c", "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'));"], depends_on=["setup_precommit", "setup_jupyter"]}

[feature.train.dependencies]
ipykernel = "*"
matplotlib = "*"
pandas = "*"
umap-learn = "*"
bokeh = "*"
pre-commit = "*"
dask = ">=2024.4.1,<2024.5"
ffmpeg = "*"

[feature.train.pypi-dependencies]
tensorflow = {version="~=2.15.0", extras=["and-cuda"]}
ray = {version = "*", extras=["data","train","tune","serve"]}
audiomentations = "*"
joblib = "~=1.4.0"

This basically ends changing lockfile every time when I use it. It can also be a bit inconvenient when sharing .lock in git repository. I think that using --frozen option would be a way to go, but still this is a bit concerning.

INFO pixi::lock_file::outdated: the pypi dependencies of environment 'train' for platform linux-64 are out of date because there are more pypi packages in the lock-file than are used by the environment: cachetools, flatbuffers, audioread, nvidia-cuda-cupti-cu12, nvidia-cuda-nvrtc-cu12, coverage, markdown, yarl, googleapis-common-protos, requests-oauthlib, astunparse, namex, clearml, opencensus-context, google-api-core

I guess tensorflow is not doing easy work for this env

ruben-arts commented 5 months ago

@niemiaszek Thanks for you example. We've been looking for a smaller example this helps! If you find any smaller reproducible examples please let us know!

niemiaszek commented 5 months ago

Sure, I will try to come with some examples. As I started to use Pixi daily, I will have more chance to experiment with "minimal not working examples". I will give it a chance over the weekend

baszalmstra commented 5 months ago

@Hofer-Julian I can confirm that your issue is fixed by https://github.com/prefix-dev/pixi/pull/1251

@niemiaszek I am pretty sure your issue is solved by https://github.com/prefix-dev/pixi/pull/1253 (not entirely sure because your example doesnt include a lock).

akhmerov commented 2 months ago

I confirm that the lockfile is declared as up to date, but an issue still remains. Specifically, it seems to always get overwritten with no changes despite pixi declaring that it is up to date. This, for example, makes direnv trigger on every action.

tdejager commented 2 months ago

Can you provide the pixi.toml?

akhmerov commented 2 months ago

My OS is Ubuntu 24.04 if that's important. I have the latest pixi (0.24.2). The toml is what I got after

pixi init
pixi add python
pixi add "black @ https://github.com/psf/black.git@24.4.2

[project]
name = "pixi_test"
version = "0.1.0"
description = "Add a short description here"
authors = ["Anton Akhmerov <anton.akhmerov@gmail.com>"]
channels = ["conda-forge"]
platforms = ["linux-64"]

[tasks]

[dependencies]
python = ">=3.12.4,<3.13"

[pypi-dependencies]
black = { git = "https://github.com/psf/black.git", rev = "24.4.2" }

I added .envrc as recommended by the docs and enabled direnv in fish shell.

watch_file pixi.lock 
eval "$(pixi shell-hook)"

I then observe that direnv runs pixi initialization on every action in the terminal.

pavelzw commented 2 months ago

You are using sdists instead of wheels. I experienced something similar in #1046

tdejager commented 2 months ago

My OS is Ubuntu 24.04 if that's important. I have the latest pixi (0.24.2). The toml is what I got after
pixi init
pixi add python
pixi add "black @ https://github.com/psf/black.git@24.4.2
[project]
name = "pixi_test"
version = "0.1.0"
description = "Add a short description here"
authors = ["Anton Akhmerov <anton.akhmerov@gmail.com>"]
channels = ["conda-forge"]
platforms = ["linux-64"]

[tasks]

[dependencies]
python = ">=3.12.4,<3.13"

[pypi-dependencies]
black = { git = "https://github.com/psf/black.git", rev = "24.4.2" }
I added .envrc as recommended by the docs and enabled direnv in fish shell.
watch_file pixi.lock 
eval "$(pixi shell-hook)"
I then observe that direnv runs pixi initialization on every action in the terminal.

Actually, that does look like a 🐛!

 INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /private/tmp/black/.pixi/envs/default/conda-meta/pixi_env_prefix
 INFO pixi::lock_file::outdated: the pypi dependencies of environment 'default' for platform osx-arm64 are out of date because the requirement 'black @ git+https://github.com/psf/black.git@24.4.2' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::resolve::uv_resolution_context: uv keyring provider is disabled
 INFO pixi::environment: Creating prefix file at: /private/tmp/black/.pixi/envs/default/conda-meta/pixi_env_prefix
 INFO pixi::environment: No update needed for the prefix file.
 INFO pixi::environment: Checking if history file exists: /private/tmp/black/.pixi/envs/default/conda-meta/history
 INFO pixi::lock_file::update: updated conda packages in the 'default' prefix in 30ms 992us 792ns
 INFO resolve_pypi{group=default platform=osx-arm64}: pixi::lock_file::resolve::pypi: there are no python packages installed by conda
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: root @ 0a0.dev0    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: black @ 24.4.2    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: click @ 8.1.7    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: mypy-extensions @ 1.0.0    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: packaging @ 24.1    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: pathspec @ 0.12.1    
 INFO solve:solve_tracked: pubgrub::internal::partial_solution: add_decision: platformdirs @ 4.2.2    
 INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'osx-arm64' in 741ms 828us 709ns
 INFO pixi::install_pypi: Nothing to do - Audited 6 distributions in 34ms
✔ The default environment has been installed.

This environment is considered unsatified while it shouldnt be.

tdejager commented 2 months ago

Ok @akhmerov I've just noticed, I think you are trying to use a tag as a revision, we don't support tags (yet). I'm thinking that's the problem.

akhmerov commented 2 months ago

Thank you for the explanation. I confirm that switching to a revision instead of a tag works.

niemiaszek commented 2 months ago

It seems there is WIP PR #1245 . I was also confused when I used branch option and in fact it didn't use desired branch and caused similar issues. My issue #1206

luizirber commented 1 month ago

I'm also seeing this one in https://github.com/sourmash-bio/sourmash

To reproduce on a specific commit with the issue (current latest):

$ git clone https://github.com/sourmash-bio/sourmash && cd sourmash && git checkout d3ae5daffb66a794d859e59f774a54383bceb0ac

Edit .envrc to add -v in the last line:

eval "$(pixi -v shell-hook)"

And allow direnv to run:

direnv run

``` $ direnv allow direnv: loading ~/work/sourmash/latest/.envrc INFO pixi::project::manifest::pyproject: Overriding the requires-python with the one defined in pixi dependencies WARN pixi::project::manifest::validation: The feature 'test' is defined but not used in any environment WARN pixi::project::manifest::validation: The feature 'demo' is defined but not used in any environment WARN pixi::project::manifest::validation: The feature 'doc' is defined but not used in any environment WARN pixi::project::manifest::validation: The feature 'storage' is defined but not used in any environment WARN pixi::project::manifest::validation: The feature 'all' is defined but not used in any environment INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /home/luizirber/work/sourmash/latest/.pixi/envs/default/conda-meta/pixi_env_prefix INFO pixi::lock_file::outdated: environment 'default' is out of date because the indexes used to previously solve to lock file are missing INFO pixi::lock_file::resolve::uv_resolution_context: uv keyring provider is disabled INFO resolve_conda{group=default platform=osx-arm64}: pixi::lock_file::update: fetched 7146 records in 2.665596967s INFO resolve_conda{group=default platform=linux-aarch64}: pixi::lock_file::update: fetched 14316 records in 3.826917714s ⠓ updating lock-file [00:00:04] [────────────────────────────────────────] 0/16 ⠄ default:linux-64 [00:00:04] loading repodata INFO resolve_conda{group=default platform=win-64}: pixi::lock_file::update: fetched 14870 records in 5.283371607s INFO resolve_conda{group=default platform=osx-64}: pixi::lock_file::update: fetched 15487 records in 6.65827434s INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'osx-arm64' in 7s 347ms 197us 493ns INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'linux-aarch64' in 7s 354ms 901us 180ns INFO resolve_conda{group=default platform=linux-64}: pixi::lock_file::update: fetched 20834 records in 8.427731937s INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'win-64' in 8s 435ms 247us 579ns INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'osx-64' in 8s 937ms 141us 626ns INFO pixi::lock_file::update: resolved conda environment for environment 'default' 'linux-64' in 9s 317ms 445us 952ns INFO pixi::environment: Creating prefix file at: /home/luizirber/work/sourmash/latest/.pixi/envs/default/conda-meta/pixi_env_prefix INFO pixi::environment: No update needed for the prefix file. INFO pixi::environment: Checking if history file exists: /home/luizirber/work/sourmash/latest/.pixi/envs/default/conda-meta/history INFO pixi::lock_file::update: updated conda packages in the 'default' prefix in 169ms 859us 452ns INFO resolve_pypi{group=default platform=osx-64}: pixi::lock_file::resolve::pypi: the following python packages are assumed to be installed by conda: pyparsing 3.1.2, cffi 1.16.0, tornado 6.4.1, bitstring 4.2.3, pillow 10.4.0, deprecation 2.1.0, python-dateutil 2.9.0, numpy 2.0.1, munkres 1.1.4, filelock 3.15.4, fonttools 4.53.1, cycler 0.12.1, unicodedata2 15.1.0, pycparser 2.22, chardet 5.2.0, tox 4.16.0, colorama 0.4.6, tomli 2.0.1, kiwisolver 1.4.5, contourpy 1.2.1, pyproject-api 1.7.1, distlib 0.3.8, certifi 2024.7.4, packaging 24.1, screed 1.1.3, virtualenv 20.26.3, setuptools 71.0.4, cachetools 5.4.0, pluggy 1.5.0, six 1.16.0, scipy 1.14.0, matplotlib 3.9.1, platformdirs 4.2.2, bitarray 2.9.2 INFO resolve_pypi{group=default platform=osx-arm64}: pixi::lock_file::resolve::pypi: the following python packages are assumed to be installed by conda: packaging 24.1, colorama 0.4.6, pycparser 2.22, deprecation 2.1.0, tornado 6.4.1, cffi 1.16.0, kiwisolver 1.4.5, python-dateutil 2.9.0, setuptools 71.0.4, contourpy 1.2.1, platformdirs 4.2.2, virtualenv 20.26.3, pyparsing 3.1.2, munkres 1.1.4, scipy 1.14.0, six 1.16.0, unicodedata2 15.1.0, matplotlib 3.9.1, chardet 5.2.0, tox 4.16.0, pillow 10.4.0, bitstring 4.2.3, cycler 0.12.1, numpy 2.0.1, fonttools 4.53.1, bitarray 2.9.2, pluggy 1.5.0, certifi 2024.7.4, filelock 3.15.4, tomli 2.0.1, distlib 0.3.8, pyproject-api 1.7.1, screed 1.1.3, cachetools 5.4.0 INFO resolve_pypi{group=default platform=linux-aarch64}: pixi::lock_file::resolve::pypi: the following python packages are assumed to be installed by conda: deprecation 2.1.0, pillow 10.4.0, packaging 24.1, unicodedata2 15.1.0, tornado 6.4.1, screed 1.1.3, bitstring 4.2.3, certifi 2024.7.4, setuptools 71.0.4, filelock 3.15.4, munkres 1.1.4, chardet 5.2.0, bitarray 2.9.2, contourpy 1.2.1, tomli 2.0.1, colorama 0.4.6, pycparser 2.22, pyparsing 3.1.2, cffi 1.16.0, scipy 1.14.0, matplotlib 3.9.1, virtualenv 20.26.3, distlib 0.3.8, platformdirs 4.2.2, pyproject-api 1.7.1, cycler 0.12.1, six 1.16.0, pluggy 1.5.0, fonttools 4.53.1, numpy 2.0.1, python-dateutil 2.9.0, cachetools 5.4.0, kiwisolver 1.4.5, tox 4.16.0 INFO resolve_pypi{group=default platform=win-64}: pixi::lock_file::resolve::pypi: the following python packages are assumed to be installed by conda: bitstring 4.2.3, pyqt5-sip 12.12.2, scipy 1.14.0, deprecation 2.1.0, cachetools 5.4.0, bitarray 2.9.2, pluggy 1.5.0, toml 0.10.2, contourpy 1.2.1, fonttools 4.53.1, pycparser 2.22, colorama 0.4.6, pyqt5 5.15.9, filelock 3.15.4, screed 1.1.3, matplotlib 3.9.1, virtualenv 20.26.3, six 1.16.0, munkres 1.1.4, sip 6.7.12, chardet 5.2.0, platformdirs 4.2.2, packaging 24.1, cffi 1.16.0, cycler 0.12.1, ply 3.11, pillow 10.4.0, tox 4.16.0, tornado 6.4.1, kiwisolver 1.4.5, pyproject-api 1.7.1, distlib 0.3.8, tomli 2.0.1, setuptools 71.0.4, numpy 2.0.1, certifi 2024.7.4, pyparsing 3.1.2, unicodedata2 15.1.0, python-dateutil 2.9.0 INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'osx-64' in 91ms 967us 872ns INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'osx-arm64' in 60ms 402us 400ns INFO resolve_pypi{group=default platform=linux-64}: pixi::lock_file::resolve::pypi: the following python packages are assumed to be installed by conda: cachetools 5.4.0, contourpy 1.2.1, pyparsing 3.1.2, deprecation 2.1.0, distlib 0.3.8, filelock 3.15.4, sip 6.7.12, pyqt5 5.15.9, bitarray 2.9.2, six 1.16.0, cffi 1.16.0, python-dateutil 2.9.0, setuptools 71.0.4, virtualenv 20.26.3, packaging 24.1, tornado 6.4.1, platformdirs 4.2.2, colorama 0.4.6, ply 3.11, pycparser 2.22, toml 0.10.2, pyqt5-sip 12.12.2, scipy 1.14.0, fonttools 4.53.1, munkres 1.1.4, kiwisolver 1.4.5, certifi 2024.7.4, matplotlib 3.9.1, screed 1.1.3, pluggy 1.5.0, tomli 2.0.1, tox 4.16.0, pillow 10.0.1, chardet 5.2.0, numpy 2.0.1, unicodedata2 15.1.0, bitstring 4.2.3, pyproject-api 1.7.1, cycler 0.12.1 INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'linux-aarch64' in 71ms 846us 103ns INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'linux-64' in 7ms 536us 725ns INFO pixi::lock_file::update: resolved pypi packages for environment 'default' 'win-64' in 45ms 16us 535ns direnv: export +ADDR2LINE +AR +AS +BUILD +CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER +CC +CC_FOR_BUILD +CFLAGS +CMAKE_ARGS +CMAKE_PREFIX_PATH +CONDA_BUILD_SYSROOT +CONDA_DEFAULT_ENV +CONDA_PREFIX +CONDA_TOOLCHAIN_BUILD +CONDA_TOOLCHAIN_HOST +CPP +CPPFLAGS +CXX +CXXFILT +CXXFLAGS +CXX_FOR_BUILD +DEBUG_CFLAGS +DEBUG_CPPFLAGS +DEBUG_CXXFLAGS +DWP +ELFEDIT +GCC +GCC_AR +GCC_NM +GCC_RANLIB +GPROF +GSETTINGS_SCHEMA_DIR +GSETTINGS_SCHEMA_DIR_CONDA_BACKUP +GXX +HOST +LD +LDFLAGS +LD_GOLD +MESON_ARGS +NM +OBJCOPY +OBJDUMP +PIXI_ENVIRONMENT_NAME +PIXI_ENVIRONMENT_PLATFORMS +PIXI_EXE +PIXI_IN_SHELL +PIXI_PROJECT_MANIFEST +PIXI_PROJECT_NAME +PIXI_PROJECT_ROOT +PIXI_PROJECT_VERSION +PIXI_PROMPT +RANLIB +READELF +SIZE +STRINGS +STRIP +XML_CATALOG_FILES +_CONDA_PYTHON_SYSCONFIGDATA_NAME +build_alias +host_alias ~PATH ```

What caught my attention was the line:

 INFO pixi::lock_file::outdated: environment 'default' is out of date because the indexes used to previously solve to lock file are missing

what is the index missing?

pixi info:

$ pixi info
      Pixi version: 0.26.1
          Platform: linux-64
  Virtual packages: __unix=0=0
                  : __linux=6.8.12=0
                  : __glibc=2.39=0
                  : __archspec=1=skylake
         Cache dir: /home/luizirber/.cache/rattler/cache
      Auth storage: /home/luizirber/.rattler/credentials.json

ruben-arts commented 1 month ago

@luizirber You've got an interesting situation. You perfectly mimicked the [project.dependencies] with the [tool.pixi.dependencies] so our logic is testing the pypi logic without having to install anything. This seems like a bug, will look into it some more.

ruben-arts commented 1 month ago

@luizirber That will be fixed in: https://github.com/prefix-dev/pixi/pull/1862

baszalmstra commented 4 weeks ago

Fixed with https://github.com/prefix-dev/pixi/pull/1862

prefix-dev / pixi