Offering nodes to reduce PR build-time on Linux

conda-forge / conda-forge.github.io

The conda-forge website.

https://conda-forge.org

BSD 3-Clause "New" or "Revised" License

132 stars 277 forks source link

Offering nodes to reduce PR build-time on Linux #695

Open sodre opened 5 years ago

sodre commented 5 years ago

I approached @jakirkham mid-last-year about the possibility of using about 11 servers I personally own to help with the build times for PRs only, i.e. I think the master builds still need to happen in the trusted CI services.

I already invested some time getting Jenkins and Docker setup in my cluster and I am starting to make the adaptations for a sample Jenkinsfile(the .travis.yaml equivalent).

Is there any interest from @conda-forge/core in pursuing this offer?

msarahan commented 5 years ago

The big thing here is the automation of submission of jobs and also public exposure of build logs. It would be really, really nice to have use of your servers, but I feel like we also need a load balancing thing of some sort to augment our build capacity rather than replace it.

We at anaconda have wished that we do such a thing, but haven't had time for engineering.

msarahan commented 5 years ago

On second thought, distributing destination based on job type (pr vs merge) might be enough. We will need the load balancer at some point, though.

sodre commented 5 years ago

I agree with you wholeheartedly on the automation and public exposure of build logs. Since these are servers that I personally own, I have complete control of them.

My thought is to use Jenkins, with the GitHub authentication and authorization plugin tie the access control to the GitHub repo, ie. I think it is very similar to what we have today with CircleCI. This also gives full-visibility to build-logs as we would expect.

On the automation part an initial option is to simply use the GitHub Organization [plugin](). What it does, is essentially monitors the GitHub org for any new repos and adds the web hooks automatically if needed.

Finally, I would not want to replace our current build system, I think CircleCI, Travis, and AppVeyors are reliable enough and should be used for final builds.

sodre commented 5 years ago

@msarahan,

I didn't hear anything back here so I went ahead and started some of the work:

[X] [conda-forge/docker-images#80] Added Jenkins slave images based on linux-anvil and linux-anvil-comp7.
[ ] WIP: [conda-forge/conda-smithy#989] Add Jenkinsfile generation to conda-smithy.

The next steps on my end would be to test whether my infrastructure can even support building all the PRs in staged-recipes. To start those tests I need:

[ ] An appropriate Jenkinsfile added to staged-recipes
[ ] Github organizational web-hooks added to conda-forge (to build PRs without polling)
[ ] A github token or one of my bots added with permission to report build-status.

Alternatively, we can decide that we don't want to pursue this route at all...

jakirkham commented 5 years ago

Thanks for starting this @sodre. Sorry that I've been personally slow to respond of late.

Since we originally talked about this last year, there has been a fair bit of work done to get conda-forge to use Azure to build and upload packages. So this is one way to handle our build load generally (Linux specifically as a consequence).

That said, it doesn't mean we shouldn't have a wide array of options for building packages. Leveraging your Linux cluster would also be great.

Should add integrating support for Jenkins is a good idea for a number of reasons. IIRC NVIDIA did some work to leverage Jenkins for building and testing Conda packages, which might be interesting if we want to build and test packages with NVIDIA GPU support in the future.