GitLab pipeline support in `maturin generate-ci` #1507

Open messense opened 1 year ago

I don't use GitLab so I'm not going to work on this anytime soon, help appreciated!

I'm trying to create PR for this issue and would love to get some thoughts about the implementation.

Gitlab CI essentially is different from github actions and thus I think not all logics built in the generate_github API here can be applied to a new API like generate_gitlab. So Does it make sense to start the CI pipeline for gitlab with image: ghcr.io/pyo3/maturin:latest ? Or we should start the pipeline with a base image like alpine and start everything from scratch ?

I'd imagine that the generated CI pipeline for gitlab would look sth like below:

linux_build:
  stage: build
  image: ghcr.io/pyo3/maturin:latest
  tags:
    - linux / window 
  script:
    parallel:
      matrix:
        - TARGET: ["x86_64", "x86", "aarch64", "armv7", "s390x", "ppc64le"]
    - >
      maturin build --release --out dist --find-interpreter --target $TARGET
      [rest of the script to follow]

The problem with starting with image: ghcr.io/pyo3/maturin:latest though, is that some logic built in for the Emscripten platfor may not be applicable. What do you think ?

We can start from scratch, but we should only use alpine for musl targets, for gnu targets you'd need to use some manylinux compatible docker images.

Hi, I'm quite interested in this too as I will try to set up a gitlab pipeline for my project. What's the best practice currently? Is there documentation somewhere already?

Sorry I don't know much about gitlab pipeline, we can come up with a basic one first and optimize later.

Hi there, @mpizenberg are you still working on this?

I've already setup a gitlab pipeline for my project using maturin and I'm interested to add a minimal version for review.

@shehab-as Feel free to open a PR.

@shehab-as could you share your gitlab yaml? Happy to collaborate on a PR. I have a rough untested draft of how I'd lay this out to achieve parallelization on different platforms and caching.

I think if we can pin down a valid gitlab yaml that covers all the edge cases that's the best starting point - then we can figure out how to build it dynamically in the generate-ci command.

default:
    before_script:
        - poetry config virtualenvs.in-project true
    cache:
        paths:
            - .venv
            - .cargo/bin
            - .cargo/registry/index
            - .cargo/registry/cache
            - target/debug/deps
            - target/debug/build

variables:
    CARGO_HOME: ${CI_PROJECT_DIR}/.cargo

stages:          
  - build
  - test
  - publish

build-general:
  needs: []
  stage: build
  image: ghcr.io/pyo3/maturin:latest
  environment:
    name: $TARGET
  tags:
    - $TARGET
  parallel:
    matrix:
      - TARGET: ["x86_64", "x86", "aarch64", "armv7", "s390x", "ppc64le"]
  script:
    - maturin build --release --out dist --find-interpreter --target $TARGET
  artifacts:
    when: on_success
    paths:
        - dist/

build-musl:
  needs: []
  stage: build
  image: alpine
  script:
    - maturin build --release --out dist --find-interpreter --target 'musl value goes here'
  artifacts:
    when: on_success
    paths:
        - dist/

test: 
  needs: [build-general, build-musl]
  stage: test
  script:
    - maturin develop
    - pip install pytest
    - pytest
  tags:
    - $IMAGE
  parallel:
    matrix:
      - IMAGE: ["linux", "windows", "macos"]
  image: $IMAGE

publish: 
  needs: [build-general, build-musl, test]
  stage: test
  image: ghcr.io/pyo3/maturin:latest
  dependencies: [build-general, build-musl]
  script:
    - maturin publish --non-interactive --skip-existing

@osintalex Feel free to take it over if there is no response/PR from @shehab-as for several weeks.

@osintalex hey, thanks for sharing. Here's a refined (and untested) version of my gitlab-ci yaml. It is similar to what you have, with addition to multi python version support and opinionated release rules.

Happy to collaborate on a PR. I've already made some progress in my fork, I can add you to the PR once I push latest changes in my branch, and from there we can discuss a "comprehensive" version of gitlab-ci per your proposal.

default:
  interruptible: true

variables:
    CARGO_HOME: ${CI_PROJECT_DIR}/.cargo

stages: 
  - build
  - test
  - release

build-linux:
  stage: build
  image: ghcr.io/pyo3/maturin:latest
  parallel:
    matrix:
      - TARGET:
        - x86_64-unknown-linux-gnu
        - x86_64-unknown-linux-musl
        - aarch64-unknown-linux-gnu
        - aarch64-unknown-linux-musl
  script:
    - maturin build -i python3.8 -i python3.9 -i python3.10 -i python3.11 -i python 3.12 --release --target $TARGET
  artifacts:
    paths:
      - target/wheels/*.whl

test-linux:
  stage: test
  image: ghcr.io/pyo3/maturin:latest
  needs: ["build-linux"]
  before_script:
    - maturin develop
    - pip install pytest
  script:
    - pytest

publish:
  stage: release
  image: ghcr.io/pyo3/maturin:latest
  needs: ["build-linux", "test-linux"]
  rules:
    - if: $CI_COMMIT_TAG
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_PIPELINE_SOURCE == "push"
      when: manual
      allow_failure: true
  script:
    - maturin publish --non-interactive --skip-existing

@messense apologies I've been occupied the last couple of weeks. Picking this up now.

Hey, sounds good! Feel free to add me to the fork and we can take it from there :-)

@osintalex I've sent you an invite and created a draft PR to start collaborating on this.

Thanks! Taking a look now :-)

Had a look and made some changes to get the yaml working in Gitlab with cross-compilation support for linux/macos/windows which are tier 1 rust targets. I think that's probably good enough for a first pass on this.

Haven't got cross compilation working for windows though even though I think that's possible from the docs - would appreciate help with this!

This is my CI yaml and this is an example pipeline.

For windows, I've been trying to get it working by adding the appropriate Pyo3 feature with cargo add pyo3 -F generate-import-lib and then based off these docs using the ZIG_COMMAND env var since I've already installed and cached zig from targeting linux/mac. But that doesn't seem to work :-(

Also open to other ways of doing this for windows, I couldn't really figure out how the github actions stuff does it since the github CI mostly just links to other actions and I found the repos for those actions fairly opaque.

This is my CI yaml and this is an example pipeline.

Seems like the repo is private?

For windows, I've been trying to get it working by adding the appropriate Pyo3 feature with cargo add pyo3 -F generate-import-lib and then based off these docs using the ZIG_COMMAND env var since I've already installed and cached zig from targeting linux/mac. But that doesn't seem to work :-(

Don't bother with zig windows cross compiling, it does not work for MSVC target yet and has issue with GNU target. #922

BTW maturin build --zig already sets up ZIG_COMMAND for you.

Sorry - just made it public. Thanks, will refactor to stop trying to cross compile to windows.

On Tue, Apr 23, 2024 at 02:41, messense @.***(mailto:On Tue, Apr 23, 2024 at 02:41, messense < wrote:

For windows, I've been trying to get it working by adding the appropriate Pyo3 feature with cargo add pyo3 -F generate-import-lib and then based off these docs using the ZIG_COMMAND env var since I've already installed and cached zig from targeting linux/mac. But that doesn't seem to work :-(

Don't bother with zig windows cross compiling, it does not work for MSVC target yet and has issue with GNU target. #922

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

Thanks @osintalex! I don't have enough permission to start a PR with refactored CI in your gitlab repo. However, I've updated your latest yml file on my PR supporting only target x86_64-pc-windows-msvc for windows cross compilation.

As you've mentioned earlier, I think we're in a good state now with an end-to-end version of the gitlab-ci for tier 1 rust targets. I've added the final version in test_generate_gitlab() test to start introducing changes. I've broken down the tasks needed in TODOs in the PR description, I'll start picking them up but do let me know if you want to parallelise development on this :)

Ah sorry, yeah I just made that repo since needed somewhere to actually test that yaml. As is evident from this thread I have not really set it up properly :-( .

Just gave you a review and v happy to collaborate, I just wanted to start from a working gitlab pipeline if that makes sense. Left you some comments and gonna try and sort out windows cross compilation without using zig.

No worries 🙂 Thanks! Here's MR !1 to your repo, I added a note around windows compilation bit.

Left you some comments

I don't think I can see them, did you add them to this PR?

Thanks! Sorry I forgot to submit my review lol. I just pushed your latest gitlab yaml to my repo after playing around some more, I think it's the closest we've got to this :-)

Erm @messense or anyone else who knows compilation to windows well, do you have any thoughts on how we could add support for more windows targets? Adding more targets here leads to pipeline failures.

Achieving this is complicated by the fact that gitlab doesn't support windows images out of the box like github does, so imo we need to find a way to cross compile to windows in a default template. That or use some kind of windows base docker image but afaik that doesn't exist.

I've tried getting this to work using a fedora base image but haven't had any luck yet.

For anyone reading this. If you're looking for a windows build on GitLab, your best option is to select a Windows-based runner on GitLab by using the available tag(s) from their docs here in your gitlab-ci file.

PyO3 / maturin

GitLab pipeline support in `maturin generate-ci` #1507