gitlab support

[Gankra]

All the places we support github-specific things, ideally we could support gitlab equivalents.

Doing this would be a nice proof-of-concept of the internal architecture and the premise of "migrate to a new platform with just one command".

[Gankra]

Currently we have:

• generate-ci github which generates CI scripts to:
  • wait for a git tag to be pushed
  • creates a Github Release
  • spins up tasks that run cargo-dist on different machines to build the targets
  • uploads the results to the Github Release
  • uploads a manifest
  • marks the Release as a non-draft on success
• --installer=github-shell which fetches from that github release
• --installer=github-powershell which fetches from that github release

[emirror-de]

I did a bit try and error using the GitLab pipeline editor as well as a bit of research.

One of the basic problems I found is that GitLab's shared runner for Windows and Mac are not as easy configurable as on Github using just runs-on parameter to select the host OS. So the only possibility I currently see is to cross compile from the rust:latest image to the desired target. This leads to having a separate job for each target that is being set up for cross compilation which makes it a bit complicated but also more clear in syntax in my opinion. At least it is not as easy as on Github by just using the desired OS as host. :)

I am not sure if cross compilation is the desired way for a solution? I am not sure what the downsides would be of this approach.

At least this script compiles the hello world for x86_64-pc-windows-gnu and x86_64-unknown-linux-gnu.

stages:
  - build
  - release

# Build the different targets
build-targets:
  stage: build
  parallel:
    matrix:
      - TARGET: x86_64-unknown-linux-gnu
      - TARGET: x86_64-apple-darwin
      - TARGET: x86_64-pc-windows-gnu
  image: "rust:latest"
  script:
    - apt-get update && apt-get -y install gcc-mingw-w64-x86-64 curl clang gcc g++ zlib1g-dev libmpc-dev libmpfr-dev libgmp-dev
    - rustc --version && cargo --version  # Print version info for debugging
    - rustup target add $TARGET
    - cargo build --release --target $TARGET
    - echo "Finished target $TARGET"

create-release:
  stage: release
  script: echo "Define your deployment script!"
  environment: production

Here are some resources that may be a starting point for future attempts, including my experiments: My experiment repo GitLab runner docs Article about GitLab CI setup for cross compilation gitlab-ci.yml file of previous mentioned article

[Gankra]

After #115 the installers are now totally factored out to just take "the download URL for the dir to fetch things from" and the github-ci code is increasingly factored out such that it might not be too bad to factor out further into an "abstract task graph" that we lower to different backends.

(Still have bigger fish to fry on the core implementation of cargo-dist before considering more backends)

[niklaswimmer]

Hello everyone, I would be interested in working on a Gitlab CI backend. Because this issue is already a bit old, I am unsure how relevant the previous discussion still is. I would therefore greatly appreciate an update on the how I would go about adding this feature with the current architecture :)

Also, of course, are you even interested in this right now? Asking because of the previous comment.

What follows are some notes about Gitlab's CI, which should hopefully highlight the most interesting differences between Gitlab and Github and help in drawing a decision (the list is for sure not exhaustive, just wrote down what I know from the top of my head).

• it would probably make most sense to concentrate on Gitlab's Runner SaaS platform for starters (i.e. hardcoding some values), support for using self-hosted runners could then be added later
• the Runner SaaS currently only supports Linux and Windows runners in the free tier, with Windows still being beta. MacOS runners are currently in beta too but can only be used with Gitlab Premium
  • cargo-dist should therefore probably only support Linux and Windows (or clarify that MacOS is untested)
• Gitlab's CI uses a tagging system to match CI runners to jobs, the jobs target platform can therefore be selected by using a tag that only selects runners that execute on those platforms
  • parallel:matrix can be used to run the same job with different tags and therefore for different platforms (see here)
• to publish a new release an additional job needs to be used, which can be configured to depend on the matrix jobs so that it does not run (and no release is published) if building one of the artifact fails
  • Gitlab however does not support draft releases like Github does, I am unsure to what extend cargo-dist currently makes use of those?
• there is currently nothing comparable to Github Actions on Gitlab, which just means that we will have to do Rust setup and similar too and can not rely on some well-known snippets
• Gitlab expects only a single file as the entry point for the CI configuration, and while it is possible to include additional files from the root, we will necessarily have to modify a file which may include user-written config for that purpose (idk how much of a problem that would be)

[Toasterson]

This is my pipeline for linux and macOS ARM. macOS x86 should be simple to add. Windows scripting is still broken and I would use the cross compile option as gitlab runners for windows are notoriously overbooked.

stages:
  - build
  - release

build:linux:
  stage: build
  tags:
    - linux
    - docker
  image: rust:latest
  script:
    - curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.4.3/cargo-dist-installer.sh | sh
    - cargo dist build --artifacts=local --target=x86_64-unknown-linux-gnu
  after_script:
    - echo "JOB_ID_LINUX=$CI_JOB_ID" >> job.env
  artifacts:
    paths:
      - target/distrib/*.tar.*
    expire_in: never
    reports:
     dotenv: job.env

build:macos:aarch64:
  stage: build
  tags:
    - saas-macos-medium-m1
  image: macos-13-xcode-14
  script:
    - curl --proto '=https' --tlsv1.2 -LsSf https://github.com/axodotdev/cargo-dist/releases/download/v0.4.3/cargo-dist-installer.sh | sh
    - source "$HOME/.cargo/env"
    - cargo dist build --artifacts=local --target=aarch64-apple-darwin
  after_script:
    - echo "JOB_ID_OSX=$CI_JOB_ID" >> job.env
  artifacts:
    paths:
      - target/distrib/*.tar.*
    expire_in: never
    reports:
     dotenv: job.env

.build:windows:
  tags:
    - shared-windows
    - windows-1809
  stage: build
  script:
    - irm  https://github.com/axodotdev/cargo-dist/releases/download/v0.4.3/cargo-dist-installer.ps1 | iex
    - winget install Microsoft.VisualStudio.2022.Community --silent --override "--wait --quiet --add ProductLang En-us --add Microsoft.VisualStudio.Workload.NativeDesktop --includeRecommended"
    - winget install Rustlang.Rustup
    - $env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
    - cargo dist build --artifacts=local --target=x86_64-pc-windows-msvc
  after_script:
    - echo "JOB_ID_WINDOWS=$CI_JOB_ID" >> job.env
  artifacts:
    paths:
      - target/distrib/*.tar.*
    expire_in: never
    reports:
     dotenv: job.env

create:release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  needs:
    - job: build:linux
      artifacts: true
#    - job: build:windows
#      artifacts: true
    - job: build:macos:aarch64
      artifacts: true
  only:
    - tags
  variables:
    TAG: '$CI_COMMIT_TAG'
  script:
    - echo "Create Release $TAG"
  release:
    name: 'Release $TAG'
    tag_name: '$TAG'
    ref: '$TAG'
    description: 'Release $TAG'
    assets:
      links:
        - name: "x86_64-unkown-linux-gnu.zip"
          url: "$GITLAB_URL/-/jobs/$JOB_ID_LINUX/artifacts/download"
        - name: "aarch64-apple-darwin.zip"
          url: "$GITLAB_URL/-/jobs/$JOB_ID_OSX/artifacts/download"
 #       - name: "x86_64-pc-windows-msvc.zip"
 #         url: "$GITLAB_URL/-/jobs/$JOB_ID_WINDOWS/artifacts/download"