conda / constructor

tool for creating installers from conda packages
https://conda.github.io/constructor/
Other
457 stars 168 forks source link

Parallelize fetching #855

Closed jaimergp closed 3 weeks ago

jaimergp commented 3 weeks ago

Checklist

What is the idea?

Package fetching can happen in parallel (as conda does) but it's not being done that way right now:

INFO:constructor.fcp:fetching: python-3.9.19-h4de0772_0_cpython.conda
INFO:constructor.fcp:fetching: ca-certificates-2024.8.30-h56e8100_0.conda
INFO:constructor.fcp:fetching: pybind11-abi-4-hd8ed1ab_3.tar.bz2
INFO:constructor.fcp:fetching: python_abi-3.9-5_cp39.conda
INFO:constructor.fcp:fetching: tzdata-2024a-h8827d51_1.conda
INFO:constructor.fcp:fetching: ucrt-10.0.22621.0-h57928b3_0.tar.bz2
INFO:constructor.fcp:fetching: vc14_runtime-14.40.33810-ha82c5b3_21.conda
INFO:constructor.fcp:fetching: vc-14.3-h8a93ad2_21.conda
INFO:constructor.fcp:fetching: vs2015_runtime-14.40.33810-h3bf8584_21.conda
INFO:constructor.fcp:fetching: bzip2-1.0.8-h2466b09_7.conda
INFO:constructor.fcp:fetching: fmt-10.2.1-h181d51b_0.conda
INFO:constructor.fcp:fetching: libffi-3.4.2-h8ffe710_5.tar.bz2
INFO:constructor.fcp:fetching: libiconv-1.17-hcfcfb64_2.conda
INFO:constructor.fcp:fetching: libsqlite-3.46.1-h2466b09_0.conda
INFO:constructor.fcp:fetching: libzlib-1.3.1-h2466b09_1.conda
INFO:constructor.fcp:fetching: lz4-c-1.9.4-hcfcfb64_0.conda
INFO:constructor.fcp:fetching: lzo-2.10-hcfcfb64_1001.conda
INFO:constructor.fcp:fetching: openssl-3.3.2-h2466b09_0.conda
INFO:constructor.fcp:fetching: reproc-14.2.4.post0-hcfcfb64_1.conda
INFO:constructor.fcp:fetching: tk-8.6.13-h5226925_1.conda
INFO:constructor.fcp:fetching: xz-5.2.6-h8d14728_0.tar.bz2
INFO:constructor.fcp:fetching: yaml-cpp-0.8.0-h63175ca_0.conda
INFO:constructor.fcp:fetching: krb5-1.21.3-hdf4eb48_0.conda
INFO:constructor.fcp:fetching: libsolv-0.7.30-hbb528cf_0.conda
INFO:constructor.fcp:fetching: libssh2-1.11.0-h7dfc565_0.conda
INFO:constructor.fcp:fetching: libxml2-2.12.7-h0f24e4e_4.conda
INFO:constructor.fcp:fetching: reproc-cpp-14.2.4.post0-h63175ca_1.conda
INFO:constructor.fcp:fetching: zstd-1.5.6-h0ea2cb4_0.conda
INFO:constructor.fcp:fetching: libarchive-3.7.4-haf234dc_0.conda
INFO:constructor.fcp:fetching: libcurl-8.9.1-h18fefc2_0.conda
INFO:constructor.fcp:fetching: menuinst-2.1.2-py39ha51f57c_1.conda
INFO:constructor.fcp:fetching: archspec-0.2.3-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: boltons-24.0.0-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: brotli-python-1.1.0-py39ha51f57c_2.conda
INFO:constructor.fcp:fetching: charset-normalizer-3.3.2-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: distro-1.9.0-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: frozendict-2.4.4-py39ha55e580_0.conda
INFO:constructor.fcp:fetching: jsonpointer-3.0.0-py39hcbf5309_1.conda
INFO:constructor.fcp:fetching: libmamba-1.5.9-hdee400e_0.conda
INFO:constructor.fcp:fetching: packaging-24.1-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: platformdirs-4.3.2-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: pluggy-1.5.0-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: pycosat-0.6.6-py39ha55989b_0.conda
INFO:constructor.fcp:fetching: ruamel.yaml.clib-0.2.8-py39ha55989b_0.conda
INFO:constructor.fcp:fetching: setuptools-73.0.1-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: jsonpatch-1.33-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: libmambapy-1.5.9-py39h3a59221_0.conda
INFO:constructor.fcp:fetching: ruamel.yaml-0.18.6-py39ha55989b_0.conda
INFO:constructor.fcp:fetching: certifi-2024.8.30-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: colorama-0.4.6-pyhd8ed1ab_0.tar.bz2
INFO:constructor.fcp:fetching: tqdm-4.66.5-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: conda-libmamba-solver-24.7.0-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: hpack-4.0.0-pyh9f0ad1d_0.tar.bz2
INFO:constructor.fcp:fetching: hyperframe-6.0.1-pyhd8ed1ab_0.tar.bz2
INFO:constructor.fcp:fetching: h2-4.1.0-pyhd8ed1ab_0.tar.bz2
INFO:constructor.fcp:fetching: idna-3.8-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: pycparser-2.22-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: cffi-1.17.1-py39ha55e580_0.conda
INFO:constructor.fcp:fetching: zstandard-0.23.0-py39h9bf74da_1.conda
INFO:constructor.fcp:fetching: conda-package-handling-2.3.0-pyh7900ff3_0.conda
INFO:constructor.fcp:fetching: conda-package-streaming-0.10.0-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: requests-2.32.3-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: conda-24.7.1-py39hcbf5309_0.conda
INFO:constructor.fcp:fetching: mamba-1.5.9-py39hca8391b_0.conda
INFO:constructor.fcp:fetching: wheel-0.44.0-pyhd8ed1ab_0.conda
INFO:constructor.fcp:fetching: win_inet_pton-1.1.0-pyhd8ed1ab_6.tar.bz2
INFO:constructor.fcp:fetching: pip-24.2-pyh8b19718_1.conda
INFO:constructor.fcp:fetching: pysocks-1.7.1-pyh070[1188](https://github.com/napari/packaging/actions/runs/10799611276/job/29955994779?pr=182#step:16:1188)_6.tar.bz2
INFO:constructor.fcp:fetching: urllib3-2.2.2-pyhd8ed1ab_1.conda
INFO:constructor.fcp:fetching: _libavif_api-1.1.1-h57928b3_1.conda
INFO:constructor.fcp:fetching: intel-openmp-2024.2.1-h57928b3_1083.conda
INFO:constructor.fcp:fetching: msys2-conda-epoch-20160418-1.tar.bz2
INFO:constructor.fcp:fetching: m2w64-gmp-6.1.0-2.tar.bz2
INFO:constructor.fcp:fetching: m2w64-libwinpthread-git-5.0.0.4634.697f757-2.tar.bz2
INFO:constructor.fcp:fetching: m2w64-gcc-libs-core-5.3.0-7.tar.bz2
INFO:constructor.fcp:fetching: aom-3.9.1-he0c23c2_0.conda
INFO:constructor.fcp:fetching: charls-2.4.2-h1537add_0.conda
INFO:constructor.fcp:fetching: dav1d-1.2.1-hcfcfb64_0.conda
INFO:constructor.fcp:fetching: giflib-5.2.2-h64bf75a_0.conda
INFO:constructor.fcp:fetching: icu-73.2-h63175ca_0.conda
INFO:constructor.fcp:fetching: jxrlib-1.1-hcfcfb64_3.conda
INFO:constructor.fcp:fetching: lerc-4.0.0-h63175ca_0.tar.bz2
INFO:constructor.fcp:fetching: libaec-1.1.3-h63175ca_0.conda
INFO:constructor.fcp:fetching: libbrotlicommon-1.1.0-h2466b09_2.conda
INFO:constructor.fcp:fetching: libdeflate-1.21-h2466b09_0.conda
INFO:constructor.fcp:fetching: libjpeg-turbo-3.0.0-hcfcfb64_1.conda
INFO:constructor.fcp:fetching: libogg-1.3.5-h2466b09_0.conda
INFO:constructor.fcp:fetching: libsodium-1.0.20-hc70643c_0.conda
INFO:constructor.fcp:fetching: libwebp-base-1.4.0-hcfcfb64_0.conda
INFO:constructor.fcp:fetching: libzopfli-1.0.3-h0e60522_0.tar.bz2
INFO:constructor.fcp:fetching: m2w64-gcc-libgfortran-5.3.0-6.tar.bz2
INFO:constructor.fcp:fetching: pthreads-win32-2.9.1-hfa6e2cd_3.tar.bz2
INFO:constructor.fcp:fetching: rav1e-0.6.6-h975169c_2.conda
INFO:constructor.fcp:fetching: snappy-1.2.1-h23299a8_0.conda
INFO:constructor.fcp:fetching: svt-av1-2.2.1-he0c23c2_0.conda
INFO:constructor.fcp:fetching: yaml-0.2.5-h8ffe710_2.tar.bz2
INFO:constructor.fcp:fetching: zfp-1.0.1-he0c23c2_1.conda
INFO:constructor.fcp:fetching: zlib-ng-2.2.1-he0c23c2_0.conda
INFO:constructor.fcp:fetching: libavif16-1.1.1-h4e96d62_1.conda
INFO:constructor.fcp:fetching: libbrotlidec-1.1.0-h2466b09_2.conda
INFO:constructor.fcp:fetching: libbrotlienc-1.1.0-h2466b09_2.conda
INFO:constructor.fcp:fetching: libintl-0.22.5-h5728263_3.conda
INFO:constructor.fcp:fetching: libpng-1.6.43-h19919ed_0.conda
INFO:constructor.fcp:fetching: libvorbis-1.3.7-h0e60522_0.tar.bz2
INFO:constructor.fcp:fetching: m2w64-gcc-libs-5.3.0-7.tar.bz2
INFO:constructor.fcp:fetching: pcre2-10.44-h3d7b363_2.conda
INFO:constructor.fcp:fetching: blosc-1.21.6-h85f69ea_0.conda
INFO:constructor.fcp:fetching: c-blosc2-2.15.1-hb461149_0.conda
INFO:constructor.fcp:fetching: freetype-2.12.1-hdaf720e_2.conda
INFO:constructor.fcp:fetching: libclang13-18.1.8-default_ha5278ca_4.conda
INFO:constructor.fcp:fetching: libglib-2.80.3-h7025463_2.conda
INFO:constructor.fcp:fetching: libhwloc-2.11.1-default_h8125262_1000.conda
INFO:constructor.fcp:fetching: libintl-devel-0.22.5-h5728263_3.conda
INFO:constructor.fcp:fetching: libtiff-4.6.0-hb151862_4.conda
INFO:constructor.fcp:fetching: libxslt-1.1.39-h3df6e99_0.conda
INFO:constructor.fcp:fetching: pthread-stubs-0.4-hcd874cb_1001.tar.bz2
INFO:constructor.fcp:fetching: xorg-libxau-1.0.11-hcd874cb_0.conda
INFO:constructor.fcp:fetching: xorg-libxdmcp-1.1.3-hcd874cb_0.tar.bz2
INFO:constructor.fcp:fetching: zeromq-4.3.5-he1f189c_5.conda
INFO:constructor.fcp:fetching: debugpy-1.8.5-py39ha51f57c_1.conda
INFO:constructor.fcp:fetching: glib-tools-2.80.3-h4394cf3_2.conda
INFO:constructor.fcp:fetching: kiwisolver-1.4.7-py39h2b77a98_0.conda
INFO:constructor.fcp:fetching: lcms2-2.16-h67d730c_0.conda
INFO:constructor.fcp:fetching: libxcb-1.16-h013a479_1.conda
INFO:constructor.fcp:fetching: llvmlite-0.43.0-py39he94c479_1.conda
INFO:constructor.fcp:fetching: markupsafe-2.1.5-py39ha55e580_1.conda
INFO:constructor.fcp:fetching: msgpack-python-1.1.0-py39h2b77a98_0.conda
INFO:constructor.fcp:fetching: openjpeg-2.5.2-h3d672ee_0.conda
INFO:constructor.fcp:fetching: psutil-6.0.0-py39ha55e580_1.conda
INFO:constructor.fcp:fetching: pywin32-306-py39h99910a6_2.conda
INFO:constructor.fcp:fetching: pyyaml-6.0.2-py39ha55e580_1.conda
INFO:constructor.fcp:fetching: pyzmq-26.2.0-py39h03e5c00_2.conda
INFO:constructor.fcp:fetching: rpds-py-0.20.0-py39h92a245a_1.conda
INFO:constructor.fcp:fetching: tbb-2021.13.0-hc790b64_0.conda
INFO:constructor.fcp:fetching: tornado-6.4.1-py39ha55e580_1.conda
INFO:constructor.fcp:fetching: wrapt-1.16.0-py39ha55e580_1.conda
INFO:constructor.fcp:fetching: glib-2.80.3-h7025463_2.conda
INFO:constructor.fcp:fetching: mkl-2024.1.0-h66d3029_694.conda
INFO:constructor.fcp:fetching: pillow-10.4.0-py39hfa8c767_0.conda
INFO:constructor.fcp:fetching: gstreamer-1.24.7-h5006eae_0.conda
INFO:constructor.fcp:fetching: libblas-3.9.0-23_win64_mkl.conda
INFO:constructor.fcp:fetching: gst-plugins-base-1.24.7-hb0a98b8_0.conda
INFO:constructor.fcp:fetching: libcblas-3.9.0-23_win64_mkl.conda
INFO:constructor.fcp:fetching: liblapack-3.9.0-23_win64_mkl.conda
INFO:constructor.fcp:fetching: numpy-2.0.2-py39h60232e0_0.conda
INFO:constructor.fcp:fetching: qt-main-5.15.8-h06adc49_23.conda

Why is this needed?

It will be faster!

What should happen?

We can borrow conda's approach and use several threads to fetch in parallel.

Additional Context

No response