Continuous Integration

[4ydan]

Goal: Improve and refactor the continuous integration process to speed up the feedback loop and not waste resources.

Speed

• [ ] Achieve a pipeline time of 5minutes (Currently takes around 30 20 10minutes)

  527

  472

  577

  Link

Space

• 585

• 591

• [ ] Reduce permaplant-rust image size (low-prio)

Node/Resource/Code managment

• 591

• 647

• 587

• [ ] Incorporate ci/build-scripts into make

New features

• 134

  596

• Static analysis

  • 736

  • 604

  • 597

  • [ ] Spell checker (low-prio, also questionable)

• 556

• 552

  549

  588

Can't be done easily

• Skip stages that don't need to be executed, when files were not modified. This can only be done if we store older build artifacts, so they can be reused for deploying.

Not important:

• Move from scripted to declarative pipeline syntax. Possible if the jenkins configuration is fixed.

• Use ENV variables at the top of the Jenkinsfile, for configuring various envs inside the containers (DATABASE_URL, USER, etc.

[markus2330]

Thanks for these many ideas! :heart:

Use ENV variables at the top of the Jenkinsfile, for configuring various envs inside the containers (DATABASE_URL, USER, etc.

What benefit would this have? Looks like a purely syntactic change? PermaplanT is basically designed that none of these variables ever need to be touched as long as the deployment stays the same. So not really something devs will touch.

Add a timeout to the whole job (like 2 hours).

Ok, somehow endless loops should be killed :+1:

Disable concurrent builds on one branch.

What benefit would this have? Doesn't this make the build slower?

Use/Maintain only one Dockerfile instead of three (rust, mdbook, node).

I like the idea. The Docker images also create significant space problems, at least in Elektra (where we have much more than 3 Docker files)

Move from scripted to declarative pipeline syntax.

Would be great.

Static analysis stage (pre-commit check).

For tools that run fast pre-commit might be okay but for most tools we probably need to have it in the CI pipeline. Which tools to do you suggest?

[4ydan]

Use ENV variables at the top of the Jenkinsfile, for configuring various envs inside the containers (DATABASE_URL, USER, etc.

What benefit would this have? Looks like a purely syntactic change? PermaplanT is basically designed that none of these variables ever need to be touched as long as the deployment stays the same. So not really something devs will touch.

Yes this would only be syntactic sugar, to have an overview of the pipeline configuration.

Disable concurrent builds on one branch.

What benefit would this have? Doesn't this make the build slower?

With concurrent builds I mean when you push again to your branch to stop older builds running with older commits.

Static analysis stage (pre-commit check).

For tools that run fast pre-commit might be okay but for most tools we probably need to have it in the CI pipeline. Which tools to do you suggest?

I think we should do all static analysis with pre-commit as first stage inside the pipeline. Maybe add a spelling-check and the link checker.

[markus2330]

With concurrent builds I mean when you push again to your branch to stop older builds running with older commits.

Yes, this is a very good idea. In Elektra this is already implemented.

I think we should do all static analysis with pre-commit as first stage inside the pipeline. Maybe add a spelling-check and the link checker.

Yes, spell check and link check would be amazing, too.

[markus2330]

I did minor updates in the top post (added spell checker, moved "declarative pipeline syntax" etc. down)

[markus2330]

Most of this issue is done, so probably best if we close it and create a new follow-up. Can you please use - [x] for what is (not) done in the next issue?