MNT: Use micromamba to provision base environment on macOS and Windows

[jaimergp]

Investigating parts of https://github.com/conda-forge/conda-forge.github.io/issues/2323#issuecomment-2389529941

[github-actions[bot]]

Hi! This is the staged-recipes linter and I found some lint.

It looks like some changes were made outside the recipes/ directory. To ensure everything runs smoothly, please make sure that recipes are only added to the recipes/ directory and no other files are changed.

If these changes are intentional (and you aren't submitting a recipe), please add a maintenance label to the PR.

File-specific lints and/or hints:

• .azure-pipelines/azure-pipelines-win.yml:

  • lints:
  • Do not edit files outside of the recipes/ directory.

• .scripts/run_osx_build.sh:

  • lints:
  • Do not edit files outside of the recipes/ directory.

• .scripts/run_win_build.bat:

  • lints:
  • Do not edit files outside of the recipes/ directory.

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR () and found it was in an excellent condition.

[github-actions[bot]]

Hi! This is the staged-recipes linter and your PR looks excellent! :rocket:

[jaimergp]

With micromamba, provisioning base in macOS takes 1 minute. I assume a lot of it has to do with fetching and loading repodata for the first time, plus downloading the actual packages later. With a lockfile that might be down to 30s.

2024-10-02T20:43:31.2124820Z + curl -L -o /var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/tmp.lsbTZXh7/micromamba https://github.com/mamba-org/micromamba-releases/releases/download/1.5.10-0/micromamba-osx-64
2024-10-02T20:43:32.1788540Z + /var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/tmp.lsbTZXh7/micromamba create --yes --root-prefix /Users/runner/.conda --prefix /Users/runner/Miniforge3 --channel conda-forge --file .ci_support/requirements.txt
2024-10-02T20:44:39.7226680Z ##[group]Configuring conda

[jaimergp]

On Windows, micromamba takes 53s:

2024-10-02T20:43:23.1586063Z Downloading micromamba 1.5.10-0
2024-10-02T20:43:32.1787310Z Creating environment
2024-10-02T20:44:16.2673525Z ##[group]Configuring conda

[wolfv]

Yeah from what I am seeing with mamba / micromamba these days is that there is a non-negligible overhead in parsing the whole repodata. Which is also why lockfiles would shave off a lot. However, later we still need to parse the whole repodata (during the build process), so it moves the cost a little bit (and ideally both processes would use the same cache, and even more ideally a .solv file prepared by the first run).

Soon we can hopefully also use the sharded repodata, but that might be a few more months off. Sharded repodata will actually not greatly reduce the cost of downloading (because from my measurements that's pretty much instantly on CI) but the parsing time will be a lot reduced.

Pixi / rattler based tools actually do not load the whole repodata but only the subset that's actually needed (which is a good bit faster).

Btw. thank you @jaimergp for running these benchmarks! Exciting!

[jaimergp]

With micromamba 2.0.2:

macOS: 50s

2024-10-03T08:44:52.4121690Z Downloading micromamba
2024-10-03T08:44:54.5073750Z Creating environment
2024-10-03T08:45:42.2350200Z ##[group]Configuring conda

Windows: 1m52s

2024-10-03T08:44:54.3682262Z Downloading micromamba 2.0.2-0
2024-10-03T08:44:55.5335120Z Creating environment
2024-10-03T08:46:46.2232879Z ##[group]Configuring conda

[jaimergp]

Without simdjson:

macOS, 58s:

2024-10-03T12:19:57.4598330Z Downloading micromamba 2.0.2-0
2024-10-03T12:19:58.2463090Z Creating environment
2024-10-03T12:20:55.5235780Z ##[group]Configuring conda

Windows, 1m53s:

2024-10-03T12:20:00.2867473Z Downloading micromamba 2.0.2-0
2024-10-03T12:20:01.7028321Z Creating environment
2024-10-03T12:21:53.3463461Z ##[group]Configuring conda

[jaimergp]

Most recent timings for Micromamba v1:

macOS, 55s:

2024-10-15T23:49:59.6670350Z ##[group]Provisioning base env with micromamba
2024-10-15T23:50:54.6825880Z ##[group]Configuring conda

Windows (D:), 1m9s:

2024-10-15T23:50:09.2341101Z ##[group]Provisioning base env with micromamba
2024-10-15T23:51:18.9994415Z ##[group]Configuring conda