Support for drone.io

[ShadowJonathan]

Could drone.io be supported?

https://docs.drone.io/

Drone.io is an open-source selfhosted/cloud CI, I expected an issue/request to be here regarding it, but I found none, so I'm opening one now.

Drone.io provides a versatile build environment on custom or cloud runners, not unlike selfhosted Github Actions runners. But my personal setup consists of linux-docker-based runners.

[joerick]

Could well be possible. The Linux build calls docker as a subprocess to perform the build. Can Drone.io Docker runners get access to a Docker daemon somehow?

[ShadowJonathan]

https://docs.drone.io/pipeline/docker/examples/services/docker_dind/

Yeah, that's possible.

[joerick]

Cool - it's possible we wouldn't have to modify cibuildwheel at all to make this work - it should detect a CI environment using the CI env var that's set by Drone. Start with that docker_dind example, and call cibuildwheel with something like cibuildwheel --output-dir wheelhouse .. It might just work! If so, I'd be happy to add it to the docs.

[ShadowJonathan]

Alrighty, i'll probably make work of this issue and make a pull request later on, but i'll keep this issue like this in case i forget or someone else wants to pick it up 👍

[ShadowJonathan]

Planning to work on this later today, have a bit of reference documentation that im gonna use: https://docs.drone.io/pipeline/environment/reference/

Also, I think I'll use snakepacker:modern/all as a base image reference for this.

[ShadowJonathan]

im afraid i cant support windows with this CI, there's too much messing around with container images and pre-flight-check installations to try to make it work, i've kinda given up after trying this config with these results:

Config

```yml steps: - name: Test #python:3.8-windowsservercore-1809 image: stefanscherer/chocolatey commands: - choco install visualstudio2017buildtools -y - choco install python3 --version 3.8.3 --no-progress -y # - choco install vcpython27 -f -y - py -m pip install -r requirements-dev.txt - py -u ./bin/run_tests.py environment: CIBW_BUILD: cp3?-* ```

Results

(through local `drone exec`, windows containers \w Docker Desktop) ``` [Test:169] Chocolatey installed 9/9 packages. [Test:170] See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log). [Test:171] [Test:172] Installed: [Test:173] - visualstudio2017buildtools v15.9.23.0 [Test:174] - kb2919355 v1.0.20160915 [Test:175] - kb2999226 v1.0.20181019 [Test:176] - dotnetfx v4.8.0.20190930 [Test:177] - chocolatey-visualstudio.extension v1.8.1 [Test:178] - kb2919442 v1.0.20160915 [Test:179] - visualstudio-installer v2.0.1 [Test:180] - chocolatey-dotnetfx.extension v1.0.1 [Test:181] - chocolatey-windowsupdate.extension v1.0.4 [Test:182] [Test:183] Packages requiring reboot: [Test:184] - dotnetfx (exit code 3010) [Test:185] [Test:186] The recent package changes indicate a reboot is necessary. [Test:187] Please reboot at your earliest convenience. 2020/05/26 12:53:02 Test : exit code 3010 ```

Which ultimately means there has to be a windows container image that has:

• Chocolatey
• Build Tools 2017
• dotnet3.5 installed/supported (I couldn't get this installed through chocolatey, and vcpython27 requires it)

Which I haven't found yet.

@joerick would it be alright to (not yet) support windows through drone.io if/when I make a PR to add support for Drone? I can probably make an open-standing issue for it if there's really that much interest for it.

[ShadowJonathan]

I'm running against issues in linux environments now, where the tests requires docker to run the manylinux images, but dind isnt supported in drone CI pipelines unless the pipeline (and by extension, repository on the drone server) is privileged, which would be a non-solution for a CI system.

That's why im kinda putting a halt to this right now till https://github.com/containers/libpod/issues/4056 and/or https://github.com/containers/libpod/issues/4131 is resolved, which would help with providing a rootless docker-compatible CLI which can be called and ran from within a container.

[YannickJadoul]

@ShadowJonathan No problem if only some platforms are supported for a certain CI provider. We've had that a lot more in the past (e.g., AppVeyor didn't support Linux/macOS for a long time, Travis CI says "Take note that our Windows environment is in early stages", ...) so don't worry about that! :-) It can always be added later, anyway.

I'm running against issues in linux environments now, where the tests requires docker to run the manylinux images, but dind isnt supported in drone CI pipelines unless the pipeline (and by extension, repository on the drone server) is privileged, which would be a non-solution for a CI system.

So it's not possible to run docker images on drone.io? That's quite annoying, indeed :-(

[ShadowJonathan]

So it's not possible to run docker images on drone.io? That's quite annoying, indeed :-(

it is possible if the repository on the drone server is privileged, basically being permitted to run administrative and root-powered commands (of which running dind is one of them).

Drone Cloud (drone.io's public free CI offering) won't make any repository privileged, which (to make manylinux wheels) is kinda needed in the linux pipeline as of right now.

That's why I mentioned podman and those issues, as there's interest in making it work from within a docker image, rootless, and unprivileged, to allow exactly this kind of workflow.

I kinda want to have drone CI covered across public offerings before i submit a PR, so that drone could automatically be tested without need for self-hosted pipelines, and tie it all to someone's personal offering or upkeep.

(for windows specifically (cuz Drone Cloud doesnt offer windows docker runners) i'm kinda looking if i could use Travis CI's windows environment, install docker, and run drone exec to run those containers locally, yet still be in the same "environment" it'd be in with an actual drone docker-runner-on-windows pipeline)

[YannickJadoul]

Alright, I see!

I kinda want to have drone CI covered across public offerings before i submit a PR, so that drone could automatically be tested without need for self-hosted pipelines, and tie it all to someone's personal offering or upkeep.

I very much agree, here, yes!

(for windows specifically (cuz Drone Cloud doesnt offer windows docker runners) i'm kinda looking if i could use Travis CI's windows environment, install docker, and run drone exec to run those containers locally, yet still be in the same "environment" it'd be in with an actual drone docker-runner-on-windows pipeline)

Not sure I fully understand, but we do only need docker support for Linux (because of the manylinux images).

Also, rereading your previous message:

Which ultimately means there has to be a windows container image that has:

* Chocolatey

* Build Tools 2017

* dotnet3.5 installed/supported (I couldn't get this installed through chocolatey, and `vcpython27` requires it)

We actually don't need Chocolatey for anything. I think that's just used in the AppVeyor examples to install Python. But as long as there's a new Python (>=3.6) to run cibuildwheel, that's OK. And if vcpython27 can't really be installed, that's a pity, but it seems possible to just add a note to the documentation that VC++ for Python 2.7 (and thus building Windows wheels with Python 2.7 without workarounds) is not supported on drone.io.

[ShadowJonathan]

Not sure I fully understand, but we do only need docker support for Linux (because of the manylinux images).

That proposed workaround is for the CI of this repository, to run cibuildwheel tests also in the environment of a Drone CI server, so that that could be tested as well. It could be achieved by running drone exec, which runs the Drone CI pipeline locally, on a windows CI offering (travis, azure, etc.) with docker on it. It's a bit of a hack.

We actually don't need Chocolatey for anything.

I thought that it was still needed for windows-related builds as i saw errors related to them pop up, but i may be misremembering it, I'll look again in a minute.

[Czaki]

Possible workaround for docker problem may be to use Singularity which is docker alternative which runs container in user space.

https://sylabs.io/guides/3.0/user-guide/singularity_and_docker.html

I heard that it works well and with docker containers. But I do not have experience with this tool.

[ShadowJonathan]

I've looked into that, to me it doesn't look like a drop-in replacement of docker, and support needs to be written for it for it to work, but maybe that'll pay off in the future.

I'll look into adding support for singularity, but in the meantime, I'll also look into making windows builds actually run again.

[Czaki]

Agree. But this is only option I know for system where you cannot execute docker. And this need changes in linux.py

[ShadowJonathan]

I'll make it so that it looks for the following: 1: Find docker, proceed with docker strategy if exists. 2: Find podman, proceed with docker strategy if exists (but with different binary name) 3: Find singularity, proceed with singularity strategy if exists

In that order.

[YannickJadoul]

Agree. But this is only option I know for system where you cannot execute docker. And this need changes in linux.py

I'd much prefer to stick to the officially supported workflow of using the manylinux docker images.

We do already have 4 free CI providers supporting all three Linux, macOS, and Windows, so I think I'd need a good argument on why drone.io offers something that's not supported by the others, before agreeing that we should throw around the current workflow, just to support drone.io.

[Czaki]

I'd much prefer to stick to the officially supported workflow of using the manylinux docker images.

But podman and singularity are only different runners for same container. They allow to use container without root.

[ShadowJonathan]

I'd much prefer to stick to the officially supported workflow of using the manylinux docker images.

singularity is able to pull and run docker images (according to the documentation), the only difference in strategy would be the actual and absolute commands involved, podman has CLI compatibility with docker, that's why its possible to just swap them out.

We do already have 4 free CI providers supporting all three Linux, macOS, and Windows, so I think I'd need a good argument on why drone.io offers something that's not supported by the others, before agreeing that we should throw around the current workflow, just to support drone.io.

drone.io works on multiple "runners", the runner i want to support first of all is the linux-based docker runner (which is most common), and for that i'd essentially need to run the cibuildwheel workflow in an unprivileged OCI container, which would provide much more opportunity beyond just drone CI, and allow cibuildwheel to be ran in custom workflows as long as the env variable CI is set to true

[Czaki]

quay.io has problem, so tests will fail now.

[modem7]

Hey guys,

Just came across this, was there any movement since https://github.com/pypa/cibuildwheel/issues/340#issuecomment-635291885?

[6543]

PS: it will also add WoodpeckerCI support if you do so... #1287 (just need to take care of some env var diffs)

[ShadowJonathan]

I was just about to add that, WoodpeckerCI is a fork from DroneCI after they decided to become more commercial / less open-source.

FWIW i consider this issue resolved if woodpeckerCI support is added.

[modem7]

I was just about to add that, WoodpeckerCI is a fork from DroneCI after they decided to become more commercial / less open-source.

FWIW i consider this issue resolved if woodpeckerCI support is added.

I'd say it'd make more sense to keep the requests separate. One for drone and one for woodpecker (#1287).

If both can be achieved at the same time, then major bonus.

[joerick]

Worth noting that since the last work on this by @ShadowJonathan we now support podman via the CIBW_CONTAINER_ENGINE option, which might make things easier.