conda-lock fails to respect non-default git branch/ref/rev from poetry `pyproject.toml`

cameronraysmith commented 6 months ago

Checklist

[X] I added a descriptive title
[X] I searched open reports and couldn't find a duplicate

What happened?

On

❯ conda-lock --version  
conda-lock, version 2.5.2.dev12+g0481cc6

if I specify any

ONE of branch or rev

```toml [tool.poetry.dependencies] # Other dependencies... flytekit = { git = "https://github.com/cameronraysmith/flytekit.git", rev = "b1e64ba20f1ddd83723e455d2bf893d8f3fc84b7" } flytekit = { git = "https://github.com/cameronraysmith/flytekit.git", branch = "3928-pandas-2" } ```

other than the default branch HEAD in the dependencies and run conda-lock

❯ poe -d conda-lock
Poe => conda-lock --conda mamba --kind lock --kind env --no-dev-dependencies --filter-categories --category workflows --virtual-package-spec environments/conda/virtual-packages.yml --filename-template 'environments/conda/conda-{platform}.lock' --lockfile environments/conda/conda-lock.yml -f pyproject.toml

with virtual-packages.yml,

```yaml subdirs: linux-64: packages: __cuda: 12.3.0 __glibc: 2.35 ```

then

for the case of specifying a commit sha as rev, I get an error stating that conda-lock cannot checkout the default branch even though the commit sha is not part of that branch

    raise CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['git', '--git-dir', '/var/folders/zx/66kfpxbj16n3blcbnn1kys9h0000gp/T/pypoetry-git-flytekitj9ytruxm/.git', '--work-tree', '/var/folders/zx/66kfpxbj16n3blcbnn1kys9h0000gp/T/pypoetry-git-flytekitj9ytruxm', 'checkout', 'master']' returned non-zero exit status 1.

or, if I specify a non-default branch, the commit sha that is placed in the conda-lock output is the HEAD of the default branch having ignored the constraint pointing to a non-default branch or sha that is not part of the default branch history.

This also holds true if I place all the dependencies

inside the conda-lock dependencies table.

```toml [tool.conda-lock.dependencies] flytekit = { git = "https://github.com/cameronraysmith/flytekit.git", branch = "3928-pandas-2" } ```

In all cases, the

correct sha appears in the poetry.lock file.

```toml [package.source] type = "git" url = "https://github.com/cameronraysmith/flytekit.git" reference = "3928-pandas-2" resolved_reference = "b1e64ba20f1ddd83723e455d2bf893d8f3fc84b7" ```

Conda Info

conda info

```shell mamba version : 1.5.3 active environment : base active env location : /usr/local/Caskroom/mambaforge/base shell level : 1 user config file : /Users/user/.condarc populated config files : /usr/local/Caskroom/mambaforge/base/.condarc /Users/user/.condarc conda version : 23.10.0 conda-build version : not installed python version : 3.10.8.final.0 virtual packages : __archspec=1=m1 __osx=13.6=0 __unix=0=0 base environment : /usr/local/Caskroom/mambaforge/base (writable) conda av data dir : /usr/local/Caskroom/mambaforge/base/etc/conda conda av metadata url : None channel URLs : https://conda.anaconda.org/bioconda/osx-64 https://conda.anaconda.org/bioconda/noarch https://conda.anaconda.org/conda-forge/osx-64 https://conda.anaconda.org/conda-forge/noarch https://conda.anaconda.org/nodefaults/osx-64 https://conda.anaconda.org/nodefaults/noarch package cache : /usr/local/Caskroom/mambaforge/base/pkgs /Users/user/.conda/pkgs envs directories : /usr/local/Caskroom/mambaforge/base/envs /Users/user/.conda/envs platform : osx-64 user-agent : conda/23.10.0 requests/2.28.1 CPython/3.10.8 Darwin/22.6.0 OSX/13.6 solver/libmamba conda-libmamba-solver/23.11.0 libmambapy/1.5.3 UID:GID : 502:20 netrc file : None offline mode : False ```

Conda Config

conda channels from pyproject.toml `tool.conda-lock` table

```shell these are the channels specified in pyproject.toml [tool.conda-lock] channels = [ 'nodefaults', 'pytorch', 'nvidia', 'conda-forge', 'bioconda', ] ```

Conda list

I'm not working in an environment. I'm just using conda-lock from a pipx installation to generate conda-lock and environment files.

maresb commented 6 months ago

This may be due to our use of an older vendored version of Poetry. I hope to get a chance soon to upgrade it, but there is no timeline for that.

cameronraysmith commented 6 months ago

Thanks Ben!

cameronraysmith commented 6 months ago

It's certainly straightforward to patch this manually, at least up to the content hash of the lock file itself. Is the latter straightforward as well? I will update this comment if I find this in the source or docs.

maresb commented 6 months ago

It's certainly straightforward to patch this manually, at least up to the content hash of the lock file itself.

I'm not following, what do you mean specifically?

Is the latter straightforward as well?

I'm guessing that revendoring Poetry would be a day-long project for me, if that's what you're asking. It's really tough for me to find a full day to dedicate to this.

I will update this comment if I find this in the source or docs.

Any additional info would be much appreciated. Thanks so much for taking the time to report this!

cameronraysmith commented 6 months ago

Oh I see. I should have been more explicit. I was speaking from the perspective of user workaround. I did not mean to comment on the complexity of fixing the root cause of the issue and I certainly do appreciate the time investment that would be required to do so.

The workaround I was implicitly proposing was that a user can manually update the commit sha for the target dependency in the conda-lock file that incorrectly resolves to the default branch HEAD. After this, I believe the only issue remaining would be that the value of the metadata.content_hash field of the lock file would then be incorrect.

I believe this is not used, so it could potentially be left incorrect, but I might be wrong about that. In case it is required, I meant to ask what exactly the key for the metadata.content_hash is comprised of and if it could be manually recomputed without significant complication for a complete end-user workaround for this issue.

maresb commented 6 months ago

The metadata.content_hash will hopefully be removed sometime soon, as per #432.

I'm not sure offhand how well your proposed fix of patching the lockfile would work. I'd recommend trying it out first by hand. I'm curious about the result.

cameronraysmith commented 6 months ago

The metadata.content_hash will hopefully be removed sometime soon, as per #432.

Many thanks for this pointer.

I'm not sure offhand how well your proposed fix of patching the lockfile would work. I'd recommend trying it out first by hand. I'm curious about the result.

I will comment on the results if I test this.

conda / conda-lock