Move to buildkite build system

[devmotion]

This PR is a draft of the buildkite integration. Most code is copied from or modified according to https://github.com/SciML/SciMLTutorials.jl. Probably something is broken currently :stuck_out_tongue:

The only problem is that https://github.com/staticfloat/coppermind-buildkite-plugin (used for memoization and avoiding unnecessary builds) seems to require a custom S3 bucket (e.g., SciMLTutorials uses s3://julialang-buildkit-artifacts/scimltutorials) and to not work with the default buildkite artifact system since one has to provide credentials explicitly. Is this correct, @staticfloat? And do we have to set up our own S3 bucket or is it possible to use something like s3://julialang-buildkit-artificats/turingtutorials?

[staticfloat]

Is this correct, @staticfloat?

Yes, that's correct. You can't use buildkite artifacts, because those artifacts are not easily available to other jobs.

is it possible to use something like s3://julialang-buildkit-artificats/turingtutorials?

Do you guys already have an S3 account? It's not too hard to set up, and will probably be quite cheap. We have our bucket configured to auto-delete things after 30 days, which helps us not pay ever-increasing amounts of money for a build cache.

[yebai]

@staticfloat Turing doesn’t have any S3 setup yet. It would be helpful to share a Julia’s S3 bucket if possible. I don’t expect Turing tutorials will consume a lot of spaces.

[staticfloat]

I discussed it with Viral, and we're okay with granting you access to the same bucket.

Are you happy with the workflow from the SciMLBenchmarks repo? If so, we can basically just copy everything over from there like you've done here, and it should all "just work".

[yebai]

Thanks @staticfloat - that sounds great!

[devmotion]

Any progress here?

[devmotion]

@staticfloat what's the path forward here? Should I just update this PR with the recent changes in SciMLTutorials? Or is there more we have to fix regarding the S3 bucket?

[staticfloat]

Yeah, just copy over all the SciMLTutorials stuff. We'll just have you use the same bucket for now. :)

[devmotion]

Yeah, just copy over all the SciMLTutorials stuff. We'll just have you use the same bucket for now. :)

OK, I copied most files and replaced SciML with Turing - should I copy the .key and .signature files as well? I guess also one of the BuildKite admins has to enable the pipeline?

[staticfloat]

Yes, copy the .key and .signature files as well. I've setup a buildkite repo, try pushing again.

[devmotion]

It seems the buildkite run passed, so it seems at least something is working 🙂 But I can't look at the details (seems I need a buildkite login?), is this expected?

[staticfloat]

Heh, I forgot to make the pipeline public. :P

The way the S3 upload credentials are managed is via the cryptic plugin, which uses cryptographic signatures to verify that pieces of the build pipeline have not been modified (e.g. so that someone can't come by and make a PR that says echo ${AWS_SECRET_KEY}.

As you will now be able to see, the build currently pauses when attempting to run the S3 upload step, because the signatures don't match. This is because some of the files that were being hashed (these ones) were modified when you brought them into this repository. The pipeline will automatically pause and allow a buildkite admin to override the hash check by clicking a button in the buildkite webui. I suggest creating a buildkite account, then sending me your account name/email so that I can add you as an admin for this repository, and you can unblock pipelines when you need to make a change and I'm not here to update the signatures. Updating the signatures shouldn't be needed often (only when we change how things get uploaded to AWS).

[staticfloat]

Signature update PR here: https://github.com/TuringLang/TuringTutorials/pull/283 (can't push directly to this branch)

[staticfloat]

Great; so we can now see that the signatures allow execution just fine, but there was a legitimate test failure.

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[devmotion]

Thanks for the detailed explanations!

I suggest creating a buildkite account, then sending me your account name/email so that I can add you as an admin for this repository, and you can unblock pipelines when you need to make a change and I'm not here to update the signatures. Updating the signatures shouldn't be needed often (only when we change how things get uploaded to AWS).

OK, I created an account with my university email address: http://www.it.uu.se/katalog/davwi492

[staticfloat]

I've invited you to join the buildkite group!

[devmotion]

Thanks, I've accepted the invite!

I copied some additional things and cleaned the repo a bit, tests seem to pass now and the test artifacts look reasonable.

[devmotion]

It looks quite promising, the tutorials seem to build after updating the manifest files (no breaking changes of direct dependencies such as Turing etc.): https://buildkite.com/julialang/turingtutorials/builds/12 Only the DiffEq tutorial is broken it seems (but probably better to fix this separately).

[devmotion]

Updating the signatures shouldn't be needed often (only when we change how things get uploaded to AWS).

I changed launch_tutorials.yml, so the signature does not match anymore and I had to bypass the check manually in the web interface :grimacing: The figures directory with the figures generated by Markdown execution was missing from the artifacts (it seems in SciMLTutorials Chris added e.g. https://github.com/SciML/SciMLTutorialsOutput/tree/master/markdown/advanced/figures manually :thinking:). It seems to work: https://buildkite.com/julialang/turingtutorials/builds/14#7e9f5599-3957-40d9-bb71-376062b7321a

[devmotion]

I think this PR is ready for a proper review. You can check the buildkite results and have a look at the artifacts.

[devmotion]

@staticfloat Do you know what the problem is on the master branch? It didn't run so I triggered buildkite manually but it fails with an error that points to the pre-command hook of coppermind: https://buildkite.com/julialang/turingtutorials/builds/18#5c8ac6f4-aad8-473c-a5fe-c369e5432a10

[staticfloat]

It was a worker misconfiguration; I fixed it and re-ran your job here: https://buildkite.com/julialang/turingtutorials/builds/22#19e0274a-5ad5-4426-8eaa-a792be86b491

Looks like the publishing step has some problems; we're trying to upload with an encrypted deploy key that is supposedly associated with https://github.com/TuringLang/TuringTutorialsOutput but I bet it's actually the encrypted deploy key for the SciML outputs directory. I'm going to need someone to generate a deploy key and send it to me securely so I can encrypt it and add it to this pipeline configuration.

[devmotion]

Thanks for fixing the issue!

Looks like the publishing step has some problems; we're trying to upload with an encrypted deploy key that is supposedly associated with https://github.com/TuringLang/TuringTutorialsOutput but I bet it's actually the encrypted deploy key for the SciML outputs directory. I'm going to need someone to generate a deploy key and send it to me securely so I can encrypt it and add it to this pipeline configuration.

I already assumed that it will fail and we have to add/modify the deploy key. What's the best way to send you the key? Or is it easier if I make you an admin of TuringTutorialsOutput temporarily?

[staticfloat]

You can send it to me on slack, or make me an admin; either way is fine