search

mamba-org / mamba

The Fast Cross-Platform Package Manager
https://mamba.readthedocs.io
BSD 3-Clause "New" or "Revised" License
6.87k stars 353 forks source link

Windows : Failure linking wheel package with PyPy Python 3.7 #2808

Closed dalcinl closed 1 year ago

dalcinl commented 1 year ago

Minimal reproducer

name: windows
on:
  push: {branches: windows}
  workflow_dispatch:

jobs:
  job:
    runs-on: 'windows-latest'
    defaults:
      run:
        shell: bash -el {0}
    steps:
    - name: Setup Micromamba
      uses: mamba-org/setup-micromamba@v1
      with:
        init-shell: bash
        environment-name: test
        create-args: python=3.7 pypy pip

Error output

C:\Users\runneradmin\micromamba-bin\micromamba.exe create -y -r C:\Users\runneradmin\micromamba -n test python=3.7 pypy pip --log-level warning --rc-file C:\Users\runneradmin\micromamba-bin\.condarc

  Transaction

    Prefix: C:\Users\runneradmin\micromamba\envs\test

    Updating specs:

     - python=3.7
     - pypy
     - pip

    Package                 Version  Build          Channel          Size
  -------------------------------------------------------------------------
    Install:
  -------------------------------------------------------------------------

    + libexpat                2.5.0  h63175ca_1     conda-forge     139kB
    + ucrt             10.0.2[26](https://github.com/dalcinl/test-gha/actions/runs/5940007240/job/16107660807#step:2:29)21.0  h57928b3_0     conda-forge       1MB
    + ca-certificates     2023.7.22  h56e8100_0     conda-forge     150kB
    + expat                   2.5.0  h63175ca_1     conda-forge     2[27](https://github.com/dalcinl/test-gha/actions/runs/5940007240/job/16107660807#step:2:30)kB
    + vc14_runtime      14.36.32532  hfdfe4a8_17    conda-forge     741kB
    + vs2015_runtime    14.36.32532  h05e6639_17    conda-forge      17kB
    + vc                       14.3  h64f974e_17    conda-forge      17kB
    + xz                      5.2.6  h8d147[28](https://github.com/dalcinl/test-gha/actions/runs/5940007240/job/16107660807#step:2:31)_0     conda-forge     218kB
    + tk                     8.6.12  h8ffe710_0     conda-forge       4MB
    + libsqlite              3.42.0  hcfcfb64_0     conda-forge     840kB
    + openssl                 3.1.2  hcfcfb64_0     conda-forge       7MB
    + libzlib                1.2.13  hcfcfb64_5     conda-forge      56kB
    + libffi                  3.4.2  h8ffe710_5     conda-forge      42kB
    + bzip2                   1.0.8  h8ffe710_4     conda-forge     152kB
    + sqlite                 3.42.0  hcfcfb64_0     conda-forge     841kB
    + zlib                   1.2.13  hcfcfb64_5     conda-forge     108kB
    + pypy3.7                 7.3.7  h4094faa_3     conda-forge      27MB
    + python_abi                3.7  2_pypy37_pp73  conda-forge       5kB
    + python                 3.7.12  0_73_pypy      conda-forge       5kB
    + wheel                  0.41.2  pyhd8ed1ab_0   conda-forge      57kB
    + setuptools             68.1.2  pyhd8ed1ab_0   conda-forge     462kB
    + pip                    23.2.1  pyhd8ed1ab_0   conda-forge       1MB
    + pypy                    7.3.7  0_pypy37       conda-forge       5kB

    Summary:

    Install: 23 packages

    Total download: 45MB

  -------------------------------------------------------------------------

  Transaction starting
  Linking libexpat-2.5.0-h63175ca_1
  Linking ucrt-10.0.22621.0-h57928b3_0
  Linking ca-certificates-2023.7.22-h56e8100_0
  Linking expat-2.5.0-h63175ca_1
  Linking vc14_runtime-14.36.32532-hfdfe4a8_17
  Linking vs2015_runtime-14.36.32532-h05e6639_17
  Linking vc-14.3-h64f974e_17
  Linking xz-5.2.6-h8d14728_0
  Linking tk-8.6.12-h8ffe710_0
  Linking libsqlite-3.42.0-hcfcfb64_0
  Linking openssl-3.1.2-hcfcfb64_0
  Linking libzlib-1.2.13-hcfcfb64_5
  Linking libffi-3.4.2-h8ffe710_5
  Linking bzip2-1.0.8-h8ffe710_4
  Linking sqlite-3.42.0-hcfcfb64_0
  Linking zlib-1.2.13-hcfcfb64_5
  Linking pypy3.7-7.3.7-h4094faa_3
  Linking python_abi-3.7-2_pypy37_pp73
  Linking python-3.7.12-0_73_pypy
  Linking wheel-0.41.2-pyhd8ed1ab_0
  error    libmamba Error opening for writing "C:\\Users\\runneradmin\\micromamba\\envs\\test\\Scripts\\wheel-script.py": No such file or directory
  error    libmamba Error opening for writing "C:\\Users\\runneradmin\\micromamba\\envs\\test\\Scripts\\wheel.exe": No such file or directory
  critical libmamba permissions: The system cannot find the path specified.: "C:\Users\runneradmin\micromamba\envs\test\Scripts\wheel.exe"
  D:\a\_actions\mamba-org\setup-micromamba\v1\dist\index.js:4447
              error = new Error(`The process '${this.toolPath}' failed with exit code ${this.processExitCode}`);
                      ^

  Error: The process 'C:\Users\runneradmin\micromamba-bin\micromamba.exe' failed with exit code 1
      at ExecState._setResult (D:\a\_actions\mamba-org\setup-micromamba\v1\dist\index.js:4447:21)
      at ExecState.CheckComplete (D:\a\_actions\mamba-org\setup-micromamba\v1\dist\index.js:4433:16)
      at ChildProcess.<anonymous> (D:\a\_actions\mamba-org\setup-micromamba\v1\dist\index.js:4336:21)
      at ChildProcess.emit (node:events:513:28)
      at maybeClose (node:internal/child_process:1100:16)
      at Process.ChildProcess._handle.onexit (node:internal/child_process:[30](https://github.com/dalcinl/test-gha/actions/runs/5940007240/job/16107660807#step:2:33)4:5)

Additional information

Using python=3.8/3.9 pypy works just fine. I'm not a Windows expert, and I do not have ready access to a physical machine or VM to debug things further manually. However, I can help trying any suggestions you may have on GHA.

pavelzw commented 1 year ago

This seems like a https://github.com/mamba-org/mamba bug.

Have you tried running the action in debug mode to see if you get more informative error messages? image (Or set log-level: trace for even more debug messages)

Another alternative is to use https://github.com/mxschmitt/action-tmate to debug your failing CI.

      - name: Setup Micromamba
        uses: mamba-org/setup-micromamba@v1
        with:
          init-shell: bash
      - uses: mxschmitt/action-tmate@v3

With this, you can try to execute micromamba create -n test python=3.7 pypy pip manually in CI.

dalcinl commented 1 year ago

This is the raw log after setting log-level: trace.

The log ends with the following not much informative (for me) output:

2023-08-24T16:36:20.8074446Z trace    libmamba linking 'site-packages/wheel/wheelfile.py'
2023-08-24T16:36:20.8075119Z trace    libmamba hard-linked 'C:\Users\runneradmin\micromamba\pkgs\wheel-0.41.2-pyhd8ed1ab_0\site-packages\wheel\wheelfile.py'
2023-08-24T16:36:20.8075706Z      --> 'C:\Users\runneradmin\micromamba\envs\test\Lib\site-packages\wheel\wheelfile.py'
2023-08-24T16:36:20.8075966Z debug    libmamba 33 files linked
2023-08-24T16:36:20.8076692Z info     libmamba Running wrapped python compilation command C:\Users\runneradmin\micromamba\envs\test\python.exe -Wi -u C:\Users\RUNNER~1\AppData\Local\Temp\mambaf4zCMDaaHrp
2023-08-24T16:36:20.8124392Z info     libmamba Compiling 27 files to pyc
2023-08-24T16:36:20.8125201Z trace    libmamba entry point path: "Scripts\\wheel"
2023-08-24T16:36:20.8125447Z     
2023-08-24T16:36:20.8126326Z error    libmamba Error opening for writing "C:\\Users\\runneradmin\\micromamba\\envs\\test\\Scripts\\wheel-script.py": No such file or directory
2023-08-24T16:36:20.8126815Z error    libmamba Error opening for writing "C:\\Users\\runneradmin\\micromamba\\envs\\test\\Scripts\\wheel.exe": No such file or directory
2023-08-24T16:36:20.8127276Z debug    libmamba Unlocking 'C:\Users\runneradmin\micromamba\envs\test\conda-meta'
2023-08-24T16:36:20.8127810Z trace    libmamba Removing lock on 'C:\Users\runneradmin\micromamba\envs\test\conda-meta\conda-meta.lock'
2023-08-24T16:36:20.8128733Z trace    libmamba Removing file 'C:\Users\runneradmin\micromamba\envs\test\conda-meta\conda-meta.lock'
2023-08-24T16:36:22.0308310Z critical libmamba permissions: The system cannot find the path specified.: "C:\Users\runneradmin\micromamba\envs\test\Scripts\wheel.exe"
2023-08-24T16:36:22.0534584Z D:\a\_actions\mamba-org\setup-micromamba\v1\dist\index.js:4447
2023-08-24T16:36:22.0535729Z             error = new Error(`The process '${this.toolPath}' failed with exit code ${this.processExitCode}`);
2023-08-24T16:36:22.0551624Z                     ^
2023-08-24T16:36:22.0555242Z 
2023-08-24T16:36:22.0561802Z Error: The process 'C:\Users\runneradmin\micromamba-bin\micromamba.exe' failed with exit code 1
2023-08-24T16:36:22.0564263Z     at ExecState._setResult (D:\a\_actions\mamba-org\setup-micromamba\v1\dist\index.js:4447:21)
2023-08-24T16:36:22.0565545Z     at ExecState.CheckComplete (D:\a\_actions\mamba-org\setup-micromamba\v1\dist\index.js:4433:16)
2023-08-24T16:36:22.0566355Z     at ChildProcess.<anonymous> (D:\a\_actions\mamba-org\setup-micromamba\v1\dist\index.js:4336:21)
2023-08-24T16:36:22.0566874Z     at ChildProcess.emit (node:events:513:28)
2023-08-24T16:36:22.0568454Z     at maybeClose (node:internal/child_process:1100:16)
2023-08-24T16:36:22.0568932Z     at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)

I can try the tmate route, too. Any extra tips at what should I look at?

pavelzw commented 1 year ago

Unfortunately not, I'm also not that experienced in Windows with conda environments :( Maybe you could also try to open an issue at https://github.com/mamba-org/mamba with your reproducer?

dalcinl commented 1 year ago

@jonashaag I believe I've found the issue. libmamba is not ensuring that the output folder <prefix>\envs\{envname}\Scripts exists before creating *-script.py and *.exe entry points.

A possible and easy fix is to add a few lines of code somewere before this line:

        if (!fs::exists(script_path.parent_path()))
        {
            fs::create_directories(script_path.parent_path());
        }
jonashaag commented 1 year ago

Cool, would you be interested in sending a PR?

Is there any way to reproduce this issue reliable? Would be great if we were able to write a test for it

dalcinl commented 1 year ago

Submitting PR to fix the issue is trivial. However, testing is another story, I have very little Windows experience, not even as a user. I believe the issue can be reliably reproduced just by trying to create a new environment with spec python=3.7 pypy and the conda-forge channel. If you can point me to the right place to add such a test, I could try to draft a PR someday next week.

jonashaag commented 1 year ago

I think you should be able to add some code to https://github.com/mamba-org/mamba/blob/main/micromamba/tests/test_linking.py or https://github.com/mamba-org/mamba/blob/main/micromamba/tests/test_install.py.