Work In Progress: Finally fix CI

arrowcircle commented 4 years ago

This is work in progress of fixing CI in master branch. Do not merge yet.

I separated releases of NPM and pypy packages and made release pipeline trigger only on tags in master branch.

Also, it looks like not all configuration of buildkite is in the repo.

Is it possible to move all buildkite steps and configuration to the repo, so we have everything in one place?

Also, I think we should bundle js files in python package before having something stable. Releasing separate versions of JS and python packages just makes a lot of confusion. And npm package release does not make any sense right now, because Yukon does not use this package in any way.

UPD1.

Step one is to separate release pipeline to separate pipeline: .buildkite/pipeline.release.yml and trigger it from main pipeline.yml only on master and tags. Done. Needs review.

Next step is to divide JS and Python pipelines and run inside docker containers. Frontend and backend does not have interconnections and do not require to be run in one container with both python and nodejs. JS step separated in ':nodejs: Run JavaScript checks' label. Done. Needs review.

Then pipelines of buildkite should be configured in yml file. This does have downside, because trigger requires a slug, and slug is available only via web interface. This is huge problem, because requires to do something in web interface => requires to config something in another place. Travis CI does look better because these web-interface steps are not needed. Who can help with this? Or should we move to Travis CI?

thirtytwobits commented 4 years ago

Let me know if we have UAVCAN organization members that need access to our Buildkite organization.

A few of things to consider:

Be careful what you allow any CI system to do from a PR. I setup a separate pipeline with steps that were in the Buildkite UI instead of pulling from the source so that random third parties could not use our CI as a zombie actor with elevated permissions.
We don't have a dedicated AWS account for UAVCAN but if we want one I can set this up. I've been using my personal account and I was able to get AWS to sponsor our project so I'd have to move our account credits over if we do want to do this.
I started using Buildkite because it offers maximum flexibility for CI workers. It's the only service I'm aware of that lets you insert arbitrary (but authenticated) resources into their pipelines. We're planning on using this for libuavcan to provide on-target testing for multiple platforms (e.g. we can actually put a Pixhawk hawk into a Buildkite pipeline if we wanted to). For Yukon this type of automation would be interesting to allow test rigs involving a full IP stack and remoting between two systems. See Nanaimo for details.

This all said, if you find the complexity of Buildkite stifling (and, as CI services go, it is in the complex/powerful quadrant of the matrix) and want to switch to something else I'm not going to object. I am not the maintainer of Yukon. Only a hopeful customer and fan.

Thanks for contributing everyone!

arrowcircle commented 4 years ago

@thirtytwobits I will fix security concern and will remove it from PR. Is it possible to share access to buildkite account so I can configure pipeline for both PR and release?

If it's possible to separate secrets between commit and release pipelines, we can easily store commit pipeline in the repo and store release pipeline in buildkite (I dont like this, but it it's the best way for now).

BTW, is it possible to add some restrictions on PR commits, so it's not possible to run releases from PR?

arrowcircle commented 4 years ago

I made a quick look through the docs and interface of the buildkite and found, that interface differs from documentation. For now I will make pipeline templates files, that can be copied to the web interface of the buildkite.

I think main pipeline should be run on every event including master branch commits. Release pipeline should not run automatically, but will be triggered via main pipeline on tags and success of previous steps.

@thirtytwobits Does this make sense? Could you please look into pipeline.yml.template and pipeline.release.yml.template files?

UPD1:

Added yarn audit command that is allowed to fail
Rewritten release pipeline to use different steps for each part. Version of JS package is set by buildkite with npm version $BUILDKITE_TAG

arrowcircle commented 4 years ago

@thirtytwobits It looks like we have one question about docker image. But main question is are we ready to update pipelines in buildkite web interface?

I am ready to change pr pipeline to content of pipeline.yml.template. Is it ok?

pavel-kirienko commented 4 years ago

I am ready to change pr pipeline to content of pipeline.yml.template. Is it ok?

Please go ahead but mind the security as advised by Scott.

arrowcircle commented 4 years ago

I uploaded new yml to buildkite for pull requests. Will look into what will break.

UPD. As expected, js tests are failing. Need to understand why. UPD2. It looks like JS tests pass only because of tox's ignore_outcome = true. Did JS tests passed in the past? Or do they failed from the beginning?