SSL timeout with mamba - Githubissues

wkjung commented 2 years ago

When I used mamba to download 82 packages (315MB) concurrently from each of 8 self-hosted runners (all the runners are running in a single machine), it randomly, but with high probability, fails with the error.

Job:

     - name: Setup conda (mamba)
        uses: conda-incubator/setup-miniconda@v2
        with:
            environment-file: conda/some-env.yml
            activate-environment: some-env
            use-mamba: true
            miniforge-variant: Mambaforge
            miniforge-version: latest
            use-only-tar-bz2: true

Error report.

  Warning:
  # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

      Traceback (most recent call last):
        File "/runner/_work/***/***/3/lib/python3.10/site-packages/conda/exceptions.py", line 1129, in __call__
          return func(*args, **kwargs)
        File "/runner/_work/***/***/3/lib/python3.10/site-packages/conda_env/cli/main.py", line 80, in do_call
          exit_code = getattr(module, func_name)(args, parser)
        File "/runner/_work/***/***/3/lib/python3.10/site-packages/conda/notices/core.py", line 72, in wrapper
          return_value = func(*args, **kwargs)
        File "/runner/_work/***/***/3/lib/python3.10/site-packages/conda_env/cli/main_update.py", line 130, in execute
          result[installer_type] = installer.install(prefix, specs, args, env)
        File "/runner/_work/***/***/3/lib/python3.10/site-packages/mamba/mamba_env.py", line 153, in mamba_install
          transaction.fetch_extract_packages()
      RuntimeError: Download error (28) Timeout was reached [https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-devel_linux-64-10.4.0-hd38fd1e_19.tar.bz2]
      SSL connection timeout

  `$ /runner/_work/***/***/3/condabin/mamba update --name repo-dev --file conda/repo-dev.yml`

    environment variables:
                   CIO_TEST=<not set>
                      CONDA=3
    CONDA_AUTO_UPDATE_CONDA=false
             CONDA_PKGS_DIR=/home/runner/conda_pkgs_dir
                 CONDA_ROOT=/runner/_work/***/***/3
             CURL_CA_BUNDLE=<not set>
           DOCKER_CERT_PATH=/certs/client
          GITHUB_EVENT_PATH=/runner/_work/_temp/_github_workflow/event.json
                GITHUB_PATH=/runner/_work/_temp/_runner_file_commands/add_path_f198bf2d-c1fd-403b-
                            9437-33a82e612850
                       PATH=3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bi
                            n:/home/runner/.local/bin
         REQUESTS_CA_BUNDLE=<not set>
              SSL_CERT_FILE=<not set>

       active environment : None
         user config file : /home/runner/.condarc
   populated config files : /runner/_work/***/***/3/.condarc
                            /home/runner/.condarc
            conda version : 22.9.0
      conda-build version : not installed
           python version : 3.10.6.final.0
         virtual packages : __linux=5.4.0=0
                            __glibc=2.31=0
                            __unix=0=0
                            __archspec=1=x86_64
         base environment : /runner/_work/***/***/3  (writable)
        conda av data dir : /runner/_work/***/***/3/etc/conda
    conda av metadata url : None
             channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                            https://conda.anaconda.org/conda-forge/noarch
                            https://repo.anaconda.com/pkgs/main/linux-64
                            https://repo.anaconda.com/pkgs/main/noarch
                            https://repo.anaconda.com/pkgs/r/linux-64
                            https://repo.anaconda.com/pkgs/r/noarch
            package cache : /home/runner/conda_pkgs_dir
         envs directories : /runner/_work/***/***/3/envs
                            /home/runner/.conda/envs
                 platform : linux-64
               user-agent : conda/22.9.0 requests/2.28.1 CPython/3.10.6 Linux/5.4.0-90-generic ubuntu/20.04.5 glibc/2.31
                  UID:GID : 1000:1000
               netrc file : None
             offline mode : False

  An unexpected error has occurred. Conda has prepared the above report.

# I don't know why, but the error report is duplicated.
# ...
# the-same-error-report
# ...

But when I stop using mamba (by removing use-mamba and mambaforge lines), the error does not happen. Should I release the 'timeout' limit or restrict the parallelism so that SSL connection timeout is not reached? How can I do that then?

jaimergp commented 2 years ago

Hi, thanks for the report!

I'd be inclined to think setup-miniconda itself is not responsible for this. Given the context of the issue (timeouts with a lot of concurrent calls) and the error coming from libmamba, I'd say the problem might have to do with these two things:

anaconda.org trying to keep up with that many requests
libmamba not having the same retry logic as conda

Maybe there's a way to increase the timeout threshold or disable SSL verification on the libmamba side? @wolfv, have you seen an issue like this with provision-with-micromamba?