CI: Proposal: Use Azure piplines

[cugu]

I started to implement a Azure Pipeline for the whole kaitai CI: https://github.com/cugu/kaitai_struct/tree/azure

The complete pipeline (Build Compiler, Translator Test, Build JS, Build Packages, Target Language Tests: https://dev.azure.com/cugu/dfir/_build/results?buildId=106&view=results) takes about 5:25 minutes (with caching).

Currently only Python and Ruby tests are working, but implementing more languages should not take more time, as they are tested in parallel.

I wanted to get some feedback on this before I continue.

Proposed actions:

• [ ] Implement all tests
• [ ] Delete ci_targets repository
• [ ] kaitai_struct:
  • [ ] Delete .travis.yml
• [ ] kaitai_struct_compiler:
  • [ ] Delete .circleci/config.yml (is this used anyway?)
• [ ] Rewrite CI documentation

[GreyCat]

Sorry for coming so late. Can I ask you to re-run this pipeline, so I can check out the results?

As of now, it says "Build not found".

[cugu]

Here you go https://dev.azure.com/cugu/dfir/_build/results?buildId=229&view=results

Another option might be using GitHub Actions.

[okuryu]

I definitely recommend using GitHub Actions instead.

[cugu]

@okuryu is there any reasoning behind this?

A reason for Azure DevOps would be the large number of preinstalled SDKs (https://github.com/microsoft/azure-pipelines-image-generation/blob/master/images/linux/Ubuntu1804-README.md), while those need to be installed or activated in GitHub Actions first. This would make GitHub Action pipelines significantly slower.

[okuryu]

You can use similar (perhaps almost the same?) tools in the GitHub Actions pipeline. I never thought GitHub Actions build container is slow. https://help.github.com/en/github/automating-your-workflow-with-github-actions/software-in-virtual-environments-for-github-actions

[GreyCat]

Thanks for this update!

There are some things which I agree with and some things which I strongly disagree with in this direction.

• Changing Travis to anything faster / more modern would be definitely a good thing and is a way to go, no objections here.
• Having faster VMs, more parallel jobs executions, faster queue times, etc, is definitely good.
• What I don't like is going back to basically one-repo-that-integrates-everything model (i.e. having central umbrella repo like kaitai_struct at all). Ideally, I'd like to have it as modular as possible — i.e. we have compiler + runtimes + tests repos separately, and change (or even PR) to every single one of them triggers exactly what is necessary to validate the change.
• For example, if there is a change in test specs for language X only, I'd only like to run a small subset of steps:
  • No compiler build
  • No test formats build
  • Test run only for language X, using pre-build test formats based on the previous stable result
  • No test runs for languages other than X, as nothing has changed for them => no point in re-running all that
• Moving everything into one system definitely has benefits of better integration, but it is clearly a road to vendor lock-in. The way it is currently implemented — we don't really have a standard way to store and access the build results — one needs to use specifically Azure DevOps APIs to access them. Compare it to current way https://ci.kaitai.io just requests plain JSON file over HTTPS — it's very straightforward and totally doable in client-side JS code only.

[cugu]

Thanks for the feedback @GreyCat . I totally agree and the pipeline above was not meant to be a final solution but a foundation of discussion.

1. I agree that we should not have a single pipeline for everything. Even though when tests are fast enough there is not really a point for testing a bit more than required. The whole kaitai struct project is quite complex and we should not introduce further complexity for some seconds less CI runtime. The question here is how many pipelines we need:

   1. A CI pipeline to test and build the compiler and generate test code for all languages. This pipeline should also trigger all pipelines from ii.
   2. A CI pipeline for every language to test the runtime and the generated test code.

2. The runtime repositories currently do not tests against the generated ci_targets. Would it be and option to split the kaitai_struct_tests repo to the compiler and the runtime repos?

3. I'm okay with not storing artifacts in Azure, though I do not really like storing artifacts (and websites) in git repositories as well. Especially mixture of generated code and scripts like in https://github.com/kaitai-io/ci_targets feels weird.

[GreyCat]

Even though when tests are fast enough

I actually doubt that "fast enough" would be scalable in the first place. Travis ci_targets currently run ~20 jobs, launching ~4 in parallel. Azure DevOps claims that it runs 10 in parallel. So, given that we're growing in that direction, and every major language like C++ or C# yields runs in 3-6-10 environments, it is still a pain point.

there is not really a point for testing a bit more than required.

Definitely agree — given that majority of changes actually only change behavior for certain one language/target — it's a waste to rebuild/retest everything from scratch every time.

i. A CI pipeline to test and build the compiler and generate test code for all languages. This pipeline should also trigger all pipelines from ii. ii. A CI pipeline for every language to test the runtime and the generated test code.

Technically, this is (almost) how it is now. Ideally, the only difference I'd like to have is to have a step in between, to be able to trigger a pipeline on change in tests repo, while compiler remains the same.

2. The runtime repositories currently do not tests against the generated ci_targets. Would it be and option to split the kaitai_struct_tests repo to the compiler and the runtime repos?

Sorry, I don't quite follow. Fresh runtime API repos are currently brought into the picture on every test run in ci_targets. What exactly would you want to split with kaitai_struct_tests?

3. I'm okay with not storing artifacts in Azure, though I do not really like storing artifacts (and websites) in git repositories as well.

Storing code generated by the compiler in ci_targets actually makes a lot of sense to me. It is code, after all, and repos are to store code in the first place. It makes it very natural to do lots of operations, like tracing where exactly a certain change in code generation happened and linking that to changes in the compiler. Most other artifact storages are just dumb blob stores — i.e. you either don't have the access to previously built artifacts at all (after the retention has expired them), or you need to download them all separately and diff them manually.

Storing test run results in ci_artifacts is:

• on one hand, benefits from the same bissecting/history automation brought by version control as well,
• but, on the other hand, I can totally agree that ideally I'd rather have some sort of database, so we could do some in-depth analytic queries like "how commits by user X are affecting the whole scene", "what are the trends for total pass/fail ratios over the time for all the languages / particular language / particular environment", etc.

Unfortunately, I haven't yet seen a good solution for storing test results in a hosted manner for free. And as soon as "non-free" factor kicks in, it normally also means that we have to do some kind of authentication.

The way that ci_artifacts are architected today directly influences https://ci.kaitai.io/, so, at the very least I'd like to retain something similar if we're changing it.

Especially mixture of generated code and scripts like in https://github.com/kaitai-io/ci_targets feels weird.

I don't like this particular aspect either. One of the easy ways out would be probably moving these prepare-* scripts to tests, but the core problem is that most current CIs are incredibly repo-centric and require you to have something checked into the repo (like .travis.yml, or .pipelines/*, or .circleci/*) — and that thing would unfortunately also have some logic in it :( So, I don't see any good way around it...

[cugu]

Travis ci_targets currently run ~20 jobs, launching ~4 in parallel. Azure DevOps claims that it runs 10 in parallel. So, given that we're growing in that direction, and every major language like C++ or C# yields runs in 3-6-10 environments, it is still a pain point. Is the limitation of 10 parallel run per repository? Can find the info for Azure. For GitHub actions it is: https://help.github.com/en/github/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#usage-limits

If we would have the tests for e.g. ruby in the kaitai_struct_ruby_runtime instead of kaitai_struct_tests/spec/ruby we would have some benefits:

• We can trigger tests for single languages
• PRs on this repository can trigger the pipeline and are validated this way
• The compiler repo could than trigger those language CI pipelines on changes.
• We have up to 10 parallel runs per language (at least for GitHub action, unknown for Azure). So this would scale better.