connection error while installing environment

tmirko commented 3 weeks ago

Checks

[x] I have checked that this issue has not already been reported.
[x] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

pyproject.toml.txt

~/git-projects/sample (feat/conda2pixi)$ pixi --version
pixi 0.34.0
~/git-projects/sample (feat/conda2pixi)$ pixi install -e prod
  ✔ validate cache       177 packages in 53s 805ms
  ⠉ download & extract   [00:00:55] [━╾──────────────────] 48.42 MiB @ 1.03 MiB/s pyspark (+281)
  ⠉ installing packages  [00:00:55] [━━━━━━━━────────────]   177/459                                                            × failed to fetch annotated-types-0.7.0-pyhd8ed1ab_0.conda
  ├─▶ error sending request for url (https://conda.anaconda.org/conda-forge/noarch/annotated-types-0.7.0-pyhd8ed1ab_0.conda)
  ├─▶ client error (SendRequest)
  ├─▶ connection error
  ╰─▶ peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/
      index.html#unexpected-eof

Issue description

connection error after executing pixi install command, can't be replicated exactly, sometimes fails on a different conda-forge dependency.

Expected behavior

Successful environment build

ruben-arts commented 3 weeks ago

Hey @tmirko, how did you install pixi? And are there any special rules on your network?

You can always try with --tls-no-verify to see if that works.

tmirko commented 3 weeks ago

Hi @ruben-arts , I've tested it as an installation in WSL ubuntu image as well as starting from https://ghcr.io/prefix-dev/pixi:0.33.0-bookworm docker image. I'm using company's proxy

--tls-no-verify doesn't help, same error.

ruben-arts commented 3 weeks ago

Could you try https://prefix.dev/conda-forge as a channel?

benmoss commented 3 weeks ago

I also get this periodically. Does retrying cause it to succeed, or at least get to another package before failing?

milicevica23 commented 3 weeks ago

Hi @ruben-arts answer on your question is it works with the conda-forge default chanell

@benmoss we also see it just periodically on my machine but on @tmirko it was the whole time

We spent today a bit smoke testing and get to some potential scenario where we think that the problem is:

private artifactory (with some network security) + slow internet is creating the problem that either some network packages get lost or artifactory stops the connectivity to the client which has open multiple connections to it

I don't know if this makes sense at all but by @tmirko connecting to mobile phone (faster internet) solved the problem for now and he was able to install the environment

Our questions are:

Do you see another way to debug/test it with some extra data which we can get out of pixi or underalying rust process?
Do we have a posabillity to slow down or setup the max connections to the artifactory in order to back the pressure?

benmoss commented 2 weeks ago

Could this be related to https://github.com/denoland/deno/issues/6197 ? Or at least the underlying issue there with rustls?

wolfv commented 2 weeks ago

I think @pavelzw and friends also had a number of issues with bad internet / private artifactory / lots of connections at the same time.

I think what we can do is:

add config parameters to control concurrency when downloading better (https://github.com/wolfv/rattler/blob/cf0e2f0ef36eb34be9703ebbb95fe57560243232/crates/rattler/src/install/installer/mod.rs#L96)
try to decouple downloading from extracting more (that might fix another issue that we've seen where donwloading becomes slow because extraction takes very long)

We'll prioritize these items soon.

pavelzw commented 1 week ago

i've also had problems with the exact same setup (artifactory + slowish internet)

i debugged some stuff as well with pixi install -vv and got some indication why the issue occurs.

2024-10-17T09:41:27.9270669Z WARN rattler_cache::package_cache: failed to download and extract https://conda.anaconda.org/conda-forge/linux-64/pynng-0.8.0-py312h2ec8cdc_4.conda to /home/runner/.cache/rattler/cache/pkgs/pynng-0.8.0-py312h2ec8cdc_4: an io error occurred: error decoding response body. Retry #1, Sleeping 543.56173ms until the next attempt...
2024-10-17T09:41:27.9350727Z WARN rattler_cache::package_cache: failed to download and extract https://conda.anaconda.org/conda-forge/linux-64/ocl-icd-2.3.2-hd590300_1.conda to /home/runner/.cache/rattler/cache/pkgs/ocl-icd-2.3.2-hd590300_1: an io error occurred: failed to iterate over archive. Retry #1, Sleeping 102.437491ms until the next attempt...
2024-10-17T09:41:27.9357826Z WARN rattler_cache::package_cache: failed to download and extract https://conda.anaconda.org/conda-forge/linux-64/scikit-learn-1.5.2-py312h7a48858_1.conda to /home/runner/.cache/rattler/cache/pkgs/scikit-learn-1.5.2-py312h7a48858_1: an io error occurred: failed to unpack `/home/runner/.cache/rattler/cache/pkgs/scikit-learn-1.5.2-py312h7a48858_1/lib/python3.12/site-packages/sklearn/ensemble/_hist_gradient_boosting/_predictor.cpython-312-x86_64-linux-gnu.so`. Retry #1, Sleeping 340.11399ms until the next attempt...
2024-10-17T09:41:27.9365965Z WARN rattler_cache::package_cache: failed to download and extract https://conda.anaconda.org/conda-forge/linux-64/numba-0.60.0-py312h83e6fd3_0.conda to /home/runner/.cache/rattler/cache/pkgs/numba-0.60.0-py312h83e6fd3_0: an io error occurred: failed to unpack `/home/runner/.cache/rattler/cache/pkgs/numba-0.60.0-py312h83e6fd3_0/lib/python3.12/site-packages/numba/tests/__pycache__/test_unicode_array.cpython-312.pyc`. Retry #1, Sleeping 983.529055ms until the next attempt...

see https://discord.com/channels/1082332781146800168/1296522386601939055/1296522470559580242 or setup-pixi-error.txt for the full logs

my assumption would be that the slow internet leads to corrupt packages that pixi can't extract, thus an IO error

xref #1994

pavelzw commented 1 week ago

what i found interesting in my findings was that the logs only say Retry #1 in ... so i assume after the first retry it works again. the package where the error then occurs (in my example libssh2) was not retried before, why is this the case?

prefix-dev / pixi