Closed jaimergp closed 3 weeks ago
Package fetching can happen in parallel (as conda does) but it's not being done that way right now:
conda
INFO:constructor.fcp:fetching: python-3.9.19-h4de0772_0_cpython.conda INFO:constructor.fcp:fetching: ca-certificates-2024.8.30-h56e8100_0.conda INFO:constructor.fcp:fetching: pybind11-abi-4-hd8ed1ab_3.tar.bz2 INFO:constructor.fcp:fetching: python_abi-3.9-5_cp39.conda INFO:constructor.fcp:fetching: tzdata-2024a-h8827d51_1.conda INFO:constructor.fcp:fetching: ucrt-10.0.22621.0-h57928b3_0.tar.bz2 INFO:constructor.fcp:fetching: vc14_runtime-14.40.33810-ha82c5b3_21.conda INFO:constructor.fcp:fetching: vc-14.3-h8a93ad2_21.conda INFO:constructor.fcp:fetching: vs2015_runtime-14.40.33810-h3bf8584_21.conda INFO:constructor.fcp:fetching: bzip2-1.0.8-h2466b09_7.conda INFO:constructor.fcp:fetching: fmt-10.2.1-h181d51b_0.conda INFO:constructor.fcp:fetching: libffi-3.4.2-h8ffe710_5.tar.bz2 INFO:constructor.fcp:fetching: libiconv-1.17-hcfcfb64_2.conda INFO:constructor.fcp:fetching: libsqlite-3.46.1-h2466b09_0.conda INFO:constructor.fcp:fetching: libzlib-1.3.1-h2466b09_1.conda INFO:constructor.fcp:fetching: lz4-c-1.9.4-hcfcfb64_0.conda INFO:constructor.fcp:fetching: lzo-2.10-hcfcfb64_1001.conda INFO:constructor.fcp:fetching: openssl-3.3.2-h2466b09_0.conda INFO:constructor.fcp:fetching: reproc-14.2.4.post0-hcfcfb64_1.conda INFO:constructor.fcp:fetching: tk-8.6.13-h5226925_1.conda INFO:constructor.fcp:fetching: xz-5.2.6-h8d14728_0.tar.bz2 INFO:constructor.fcp:fetching: yaml-cpp-0.8.0-h63175ca_0.conda INFO:constructor.fcp:fetching: krb5-1.21.3-hdf4eb48_0.conda INFO:constructor.fcp:fetching: libsolv-0.7.30-hbb528cf_0.conda INFO:constructor.fcp:fetching: libssh2-1.11.0-h7dfc565_0.conda INFO:constructor.fcp:fetching: libxml2-2.12.7-h0f24e4e_4.conda INFO:constructor.fcp:fetching: reproc-cpp-14.2.4.post0-h63175ca_1.conda INFO:constructor.fcp:fetching: zstd-1.5.6-h0ea2cb4_0.conda INFO:constructor.fcp:fetching: libarchive-3.7.4-haf234dc_0.conda INFO:constructor.fcp:fetching: libcurl-8.9.1-h18fefc2_0.conda INFO:constructor.fcp:fetching: menuinst-2.1.2-py39ha51f57c_1.conda INFO:constructor.fcp:fetching: archspec-0.2.3-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: boltons-24.0.0-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: brotli-python-1.1.0-py39ha51f57c_2.conda INFO:constructor.fcp:fetching: charset-normalizer-3.3.2-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: distro-1.9.0-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: frozendict-2.4.4-py39ha55e580_0.conda INFO:constructor.fcp:fetching: jsonpointer-3.0.0-py39hcbf5309_1.conda INFO:constructor.fcp:fetching: libmamba-1.5.9-hdee400e_0.conda INFO:constructor.fcp:fetching: packaging-24.1-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: platformdirs-4.3.2-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: pluggy-1.5.0-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: pycosat-0.6.6-py39ha55989b_0.conda INFO:constructor.fcp:fetching: ruamel.yaml.clib-0.2.8-py39ha55989b_0.conda INFO:constructor.fcp:fetching: setuptools-73.0.1-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: jsonpatch-1.33-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: libmambapy-1.5.9-py39h3a59221_0.conda INFO:constructor.fcp:fetching: ruamel.yaml-0.18.6-py39ha55989b_0.conda INFO:constructor.fcp:fetching: certifi-2024.8.30-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: colorama-0.4.6-pyhd8ed1ab_0.tar.bz2 INFO:constructor.fcp:fetching: tqdm-4.66.5-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: conda-libmamba-solver-24.7.0-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: hpack-4.0.0-pyh9f0ad1d_0.tar.bz2 INFO:constructor.fcp:fetching: hyperframe-6.0.1-pyhd8ed1ab_0.tar.bz2 INFO:constructor.fcp:fetching: h2-4.1.0-pyhd8ed1ab_0.tar.bz2 INFO:constructor.fcp:fetching: idna-3.8-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: pycparser-2.22-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: cffi-1.17.1-py39ha55e580_0.conda INFO:constructor.fcp:fetching: zstandard-0.23.0-py39h9bf74da_1.conda INFO:constructor.fcp:fetching: conda-package-handling-2.3.0-pyh7900ff3_0.conda INFO:constructor.fcp:fetching: conda-package-streaming-0.10.0-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: requests-2.32.3-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: conda-24.7.1-py39hcbf5309_0.conda INFO:constructor.fcp:fetching: mamba-1.5.9-py39hca8391b_0.conda INFO:constructor.fcp:fetching: wheel-0.44.0-pyhd8ed1ab_0.conda INFO:constructor.fcp:fetching: win_inet_pton-1.1.0-pyhd8ed1ab_6.tar.bz2 INFO:constructor.fcp:fetching: pip-24.2-pyh8b19718_1.conda INFO:constructor.fcp:fetching: pysocks-1.7.1-pyh070[1188](https://github.com/napari/packaging/actions/runs/10799611276/job/29955994779?pr=182#step:16:1188)_6.tar.bz2 INFO:constructor.fcp:fetching: urllib3-2.2.2-pyhd8ed1ab_1.conda INFO:constructor.fcp:fetching: _libavif_api-1.1.1-h57928b3_1.conda INFO:constructor.fcp:fetching: intel-openmp-2024.2.1-h57928b3_1083.conda INFO:constructor.fcp:fetching: msys2-conda-epoch-20160418-1.tar.bz2 INFO:constructor.fcp:fetching: m2w64-gmp-6.1.0-2.tar.bz2 INFO:constructor.fcp:fetching: m2w64-libwinpthread-git-5.0.0.4634.697f757-2.tar.bz2 INFO:constructor.fcp:fetching: m2w64-gcc-libs-core-5.3.0-7.tar.bz2 INFO:constructor.fcp:fetching: aom-3.9.1-he0c23c2_0.conda INFO:constructor.fcp:fetching: charls-2.4.2-h1537add_0.conda INFO:constructor.fcp:fetching: dav1d-1.2.1-hcfcfb64_0.conda INFO:constructor.fcp:fetching: giflib-5.2.2-h64bf75a_0.conda INFO:constructor.fcp:fetching: icu-73.2-h63175ca_0.conda INFO:constructor.fcp:fetching: jxrlib-1.1-hcfcfb64_3.conda INFO:constructor.fcp:fetching: lerc-4.0.0-h63175ca_0.tar.bz2 INFO:constructor.fcp:fetching: libaec-1.1.3-h63175ca_0.conda INFO:constructor.fcp:fetching: libbrotlicommon-1.1.0-h2466b09_2.conda INFO:constructor.fcp:fetching: libdeflate-1.21-h2466b09_0.conda INFO:constructor.fcp:fetching: libjpeg-turbo-3.0.0-hcfcfb64_1.conda INFO:constructor.fcp:fetching: libogg-1.3.5-h2466b09_0.conda INFO:constructor.fcp:fetching: libsodium-1.0.20-hc70643c_0.conda INFO:constructor.fcp:fetching: libwebp-base-1.4.0-hcfcfb64_0.conda INFO:constructor.fcp:fetching: libzopfli-1.0.3-h0e60522_0.tar.bz2 INFO:constructor.fcp:fetching: m2w64-gcc-libgfortran-5.3.0-6.tar.bz2 INFO:constructor.fcp:fetching: pthreads-win32-2.9.1-hfa6e2cd_3.tar.bz2 INFO:constructor.fcp:fetching: rav1e-0.6.6-h975169c_2.conda INFO:constructor.fcp:fetching: snappy-1.2.1-h23299a8_0.conda INFO:constructor.fcp:fetching: svt-av1-2.2.1-he0c23c2_0.conda INFO:constructor.fcp:fetching: yaml-0.2.5-h8ffe710_2.tar.bz2 INFO:constructor.fcp:fetching: zfp-1.0.1-he0c23c2_1.conda INFO:constructor.fcp:fetching: zlib-ng-2.2.1-he0c23c2_0.conda INFO:constructor.fcp:fetching: libavif16-1.1.1-h4e96d62_1.conda INFO:constructor.fcp:fetching: libbrotlidec-1.1.0-h2466b09_2.conda INFO:constructor.fcp:fetching: libbrotlienc-1.1.0-h2466b09_2.conda INFO:constructor.fcp:fetching: libintl-0.22.5-h5728263_3.conda INFO:constructor.fcp:fetching: libpng-1.6.43-h19919ed_0.conda INFO:constructor.fcp:fetching: libvorbis-1.3.7-h0e60522_0.tar.bz2 INFO:constructor.fcp:fetching: m2w64-gcc-libs-5.3.0-7.tar.bz2 INFO:constructor.fcp:fetching: pcre2-10.44-h3d7b363_2.conda INFO:constructor.fcp:fetching: blosc-1.21.6-h85f69ea_0.conda INFO:constructor.fcp:fetching: c-blosc2-2.15.1-hb461149_0.conda INFO:constructor.fcp:fetching: freetype-2.12.1-hdaf720e_2.conda INFO:constructor.fcp:fetching: libclang13-18.1.8-default_ha5278ca_4.conda INFO:constructor.fcp:fetching: libglib-2.80.3-h7025463_2.conda INFO:constructor.fcp:fetching: libhwloc-2.11.1-default_h8125262_1000.conda INFO:constructor.fcp:fetching: libintl-devel-0.22.5-h5728263_3.conda INFO:constructor.fcp:fetching: libtiff-4.6.0-hb151862_4.conda INFO:constructor.fcp:fetching: libxslt-1.1.39-h3df6e99_0.conda INFO:constructor.fcp:fetching: pthread-stubs-0.4-hcd874cb_1001.tar.bz2 INFO:constructor.fcp:fetching: xorg-libxau-1.0.11-hcd874cb_0.conda INFO:constructor.fcp:fetching: xorg-libxdmcp-1.1.3-hcd874cb_0.tar.bz2 INFO:constructor.fcp:fetching: zeromq-4.3.5-he1f189c_5.conda INFO:constructor.fcp:fetching: debugpy-1.8.5-py39ha51f57c_1.conda INFO:constructor.fcp:fetching: glib-tools-2.80.3-h4394cf3_2.conda INFO:constructor.fcp:fetching: kiwisolver-1.4.7-py39h2b77a98_0.conda INFO:constructor.fcp:fetching: lcms2-2.16-h67d730c_0.conda INFO:constructor.fcp:fetching: libxcb-1.16-h013a479_1.conda INFO:constructor.fcp:fetching: llvmlite-0.43.0-py39he94c479_1.conda INFO:constructor.fcp:fetching: markupsafe-2.1.5-py39ha55e580_1.conda INFO:constructor.fcp:fetching: msgpack-python-1.1.0-py39h2b77a98_0.conda INFO:constructor.fcp:fetching: openjpeg-2.5.2-h3d672ee_0.conda INFO:constructor.fcp:fetching: psutil-6.0.0-py39ha55e580_1.conda INFO:constructor.fcp:fetching: pywin32-306-py39h99910a6_2.conda INFO:constructor.fcp:fetching: pyyaml-6.0.2-py39ha55e580_1.conda INFO:constructor.fcp:fetching: pyzmq-26.2.0-py39h03e5c00_2.conda INFO:constructor.fcp:fetching: rpds-py-0.20.0-py39h92a245a_1.conda INFO:constructor.fcp:fetching: tbb-2021.13.0-hc790b64_0.conda INFO:constructor.fcp:fetching: tornado-6.4.1-py39ha55e580_1.conda INFO:constructor.fcp:fetching: wrapt-1.16.0-py39ha55e580_1.conda INFO:constructor.fcp:fetching: glib-2.80.3-h7025463_2.conda INFO:constructor.fcp:fetching: mkl-2024.1.0-h66d3029_694.conda INFO:constructor.fcp:fetching: pillow-10.4.0-py39hfa8c767_0.conda INFO:constructor.fcp:fetching: gstreamer-1.24.7-h5006eae_0.conda INFO:constructor.fcp:fetching: libblas-3.9.0-23_win64_mkl.conda INFO:constructor.fcp:fetching: gst-plugins-base-1.24.7-hb0a98b8_0.conda INFO:constructor.fcp:fetching: libcblas-3.9.0-23_win64_mkl.conda INFO:constructor.fcp:fetching: liblapack-3.9.0-23_win64_mkl.conda INFO:constructor.fcp:fetching: numpy-2.0.2-py39h60232e0_0.conda INFO:constructor.fcp:fetching: qt-main-5.15.8-h06adc49_23.conda
It will be faster!
We can borrow conda's approach and use several threads to fetch in parallel.
No response
Checklist
What is the idea?
Package fetching can happen in parallel (as
conda
does) but it's not being done that way right now:Why is this needed?
It will be faster!
What should happen?
We can borrow conda's approach and use several threads to fetch in parallel.
Additional Context
No response