PyO3 / maturin-action

GitHub Action to install and run a custom maturin command with built-in support for cross compilation
MIT License
127 stars 35 forks source link

Fails to install successfully built package for Windows x86 #235

Closed mRcSchwering closed 9 months ago

mRcSchwering commented 9 months ago

Maybe this is not maturin-action's fault, I just don't know where to ask.

I created a github workflow using maturin generate-ci --pytest github > .github/workflows/CI.yml. It builds a python3-rust hybrid package using maturin 1.4.0, pyo3 0.20.0 (github.com/mRcSchwering/magic-soup). All OSs and architectures are built, and for Linux and MacOS the pytest step runs green as well (CI.yml below).

Just for Windows x86 the pytest step doesn't run. Looking into the logs I can see that during the package installation in the test environment it seems to ignore one of the package dependencies.

My package has 2 dependencies: numpy and torch. During pip install magicsoup --find-links dist --force-reinstall it correctly starts to collect numpy. But then it doesn't collect torch and instead continues to install my package from the cached wheel:

Processing d:\a\magic-soup\magic-soup\dist\magicsoup-0.10.2-cp310-none-win32.whl
Collecting numpy<1.27.0,>=1.24.0 (from magicsoup)
  Downloading numpy-1.26.2-cp310-cp310-win32.whl.metadata (61 kB)
     ---------------------------------------- 61.2/61.2 kB 1.6 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of magicsoup to determine which version is compatible with other requirements. This could take a while.
Collecting magicsoup
  Downloading magicsoup-0.10.1.tar.gz (8.0 MB)
     ---------------------------------------- 8.0/8.0 MB 11.1 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting numpy<1.25.0,>=1.24.0 (from magicsoup)
  Downloading numpy-1.24.4-cp310-cp310-win32.whl.metadata (5.6 kB)
Collecting magicsoup
  Downloading magicsoup-0.10.0-py3-none-any.whl.metadata (5.5 kB)
  Downloading magicsoup-0.9.1-py3-none-any.whl.metadata (5.5 kB)
Downloading magicsoup-0.9.1-py3-none-any.whl ([55](https://github.com/mRcSchwering/magic-soup/actions/runs/7143208086/job/19454171432#step:6:56) kB)
   ---------------------------------------- 55.1/55.1 kB 2.8 MB/s eta 0:00:00
Installing collected packages: magicsoup
Successfully installed magicsoup-0.9.1

Afterwards, when tests are started, it fails at import torch because it didn't install it. This is strange because this behaviour doesn't exist in other OSs or architectures. E.g. here is the same step but from MacOS x86_64:

Collecting magicsoup
  Downloading magicsoup-0.10.2-cp310-cp310-macosx_10_7_x86_64.whl.metadata (5.6 kB)
Collecting numpy<1.27.0,>=1.24.0 (from magicsoup)
  Downloading numpy-1.26.2-cp310-cp310-macosx_10_9_x86_64.whl.metadata (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 2.9 MB/s eta 0:00:00
Collecting torch<2.2.0,>=2.0.0 (from magicsoup)
  Downloading torch-2.1.1-cp310-none-macosx_10_9_x86_64.whl.metadata (25 kB)
Collecting filelock (from torch<2.2.0,>=2.0.0->magicsoup)
  Downloading filelock-3.13.1-py3-none-any.whl.metadata (2.8 kB)
Collecting typing-extensions (from torch<2.2.0,>=2.0.0->magicsoup)
  Downloading typing_extensions-4.8.0-py3-none-any.whl.metadata (3.0 kB)
Collecting sympy (from torch<2.2.0,>=2.0.0->magicsoup)
  Downloading sympy-1.12-py3-none-any.whl (5.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB [22](https://github.com/mRcSchwering/magic-soup/actions/runs/7143208086/job/19454169531#step:6:23).7 MB/s eta 0:00:00
Collecting networkx (from torch<2.2.0,>=2.0.0->magicsoup)
  Downloading networkx-3.2.1-py3-none-any.whl.metadata (5.2 kB)
Collecting jinja2 (from torch<2.2.0,>=2.0.0->magicsoup)
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 2.3 MB/s eta 0:00:00
Collecting fsspec (from torch<2.2.0,>=2.0.0->magicsoup)
  Downloading fsspec-20[23](https://github.com/mRcSchwering/magic-soup/actions/runs/7143208086/job/19454169531#step:6:24).12.1-py3-none-any.whl.metadata (6.8 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch<2.2.0,>=2.0.0->magicsoup)
  Downloading MarkupSafe-2.1.3-cp310-cp310-macosx_10_9_x86_64.whl.metadata (3.0 kB)
Collecting mpmath>=0.19 (from sympy->torch<2.2.0,>=2.0.0->magicsoup)
  Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 29.4 MB/s eta 0:00:00
Downloading magicsoup-0.10.2-cp310-cp310-macosx_10_7_x86_64.whl (433 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 433.3/433.3 kB 23.8 MB/s eta 0:00:00
Downloading numpy-1.[26](https://github.com/mRcSchwering/magic-soup/actions/runs/7143208086/job/19454169531#step:6:27).2-cp[31](https://github.com/mRcSchwering/magic-soup/actions/runs/7143208086/job/19454169531#step:6:32)0-cp310-macosx_10_9_x86_64.whl (20.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.6/20.6 MB 10.6 MB/s eta 0:00:00
Downloading torch-2.1.1-cp310-none-macosx_10_9_x86_64.whl (1[47](https://github.com/mRcSchwering/magic-soup/actions/runs/7143208086/job/19454169531#step:6:48).0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 147.0/147.0 MB 4.9 MB/s eta 0:00:00
Downloading filelock-3.13.1-py3-none-any.whl (11 kB)
Downloading fsspec-2023.12.1-py3-none-any.whl (168 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.9/168.9 kB 11.6 MB/s eta 0:00:00
Downloading networkx-3.2.1-py3-none-any.whl (1.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 22.4 MB/s eta 0:00:00
Downloading typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Downloading MarkupSafe-2.1.3-cp310-cp310-macosx_10_9_x86_64.whl (13 kB)
Installing collected packages: mpmath, typing-extensions, sympy, numpy, networkx, MarkupSafe, fsspec, filelock, jinja2, torch, magicsoup
Successfully installed MarkupSafe-2.1.3 filelock-3.13.1 fsspec-2023.12.1 jinja2-3.1.2 magicsoup-0.10.2 mpmath-1.3.0 networkx-3.2.1 numpy-1.26.2 sympy-1.12 torch-2.1.1 typing-extensions-4.8.0

As you can see it correctly installs numpy and torch. I only have a Linux so I can't really reproduce this problem locally. I only see it in the github pipeline.

CI.yml

# This file is autogenerated by maturin v1.4.0
# To update, run
#
#    maturin generate-ci --pytest github
#
name: CI

on:
  workflow_dispatch:
  push:
    tags:
      - 'v*'

permissions:
  contents: read

jobs:
  linux:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        target: [x86_64, x86, aarch64]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      - name: Build wheels
        uses: PyO3/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          args: --release --out dist --find-interpreter
          sccache: 'true'
          manylinux: auto
      - name: Upload wheels
        uses: actions/upload-artifact@v3
        with:
          name: wheels
          path: dist
      - name: pytest
        if: ${{ startsWith(matrix.target, 'x86_64') }}
        shell: bash
        run: |
          set -e
          pip install magicsoup --find-links dist --force-reinstall
          pip install pytest
          pytest tests/fast

  windows:
    runs-on: windows-latest
    strategy:
      matrix:
        target: [x64, x86]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.10'
          architecture: ${{ matrix.target }}
      - name: Build wheels
        uses: PyO3/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          args: --release --out dist --find-interpreter
          sccache: 'true'
      - name: Upload wheels
        uses: actions/upload-artifact@v3
        with:
          name: wheels
          path: dist
      - name: pytest
        if: ${{ startsWith(matrix.target, 'x86') }}
        shell: bash
        run: |
          set -e
          pip install magicsoup --find-links dist --force-reinstall
          pip install pytest
          pytest tests/fast

  macos:
    runs-on: macos-latest
    strategy:
      matrix:
        target: [x86_64, aarch64]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      - name: Build wheels
        uses: PyO3/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          args: --release --out dist --find-interpreter
          sccache: 'true'
      - name: Upload wheels
        uses: actions/upload-artifact@v3
        with:
          name: wheels
          path: dist
      - name: pytest
        if: ${{ !startsWith(matrix.target, 'aarch64') }}
        shell: bash
        run: |
          set -e
          pip install magicsoup --find-links dist --force-reinstall
          pip install pytest
          pytest tests/fast

  sdist:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Build sdist
        uses: PyO3/maturin-action@v1
        with:
          command: sdist
          args: --out dist
      - name: Upload sdist
        uses: actions/upload-artifact@v3
        with:
          name: wheels
          path: dist

  release:
    name: Release
    runs-on: ubuntu-latest
    needs: [linux, windows, macos, sdist]
    steps:
      - uses: actions/download-artifact@v3
        with:
          name: wheels
      - name: Publish to PyPI
        uses: PyO3/maturin-action@v1
        env:
          MATURIN_PYPI_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
        with:
          command: upload
          args: --non-interactive --skip-existing *
messense commented 9 months ago

https://pytorch.org/docs/stable/notes/windows.html#installation

PyTorch doesn’t work on 32-bit system. Please use Windows and Python 64-bit version.