Requirements: Model-building CI

[jastram]

WP4 requested continuous integration for the Scade model, and indicated that WP3 is likely interested as well. Here are the assumptions and requirements for such a system:

• At DLR, there is already a build computer that allow the automated building of the Scade Model. The job of the CI system would be to trigger the build, "when required".
• As a build system would always require a Scade license, the build computer will reside at DLR.
• Making the build computer visible to the Internet would be cumbersome. Therefore, the build should be triggered via a "pull", rather than a "push".
• The build would be triggered by observing certain files in the git repository.

  Specification

• The Cloudbees server project is openETCS-WP4
• The project monitors the modeling repository
• If a change is the repository is detected, then the status file is updated. Its URL is: https://openetcs.ci.cloudbees.com/job/openETCS-WP4/ws/status.txt

  Status file

• The file grows continuously, appending new status at the end of this file.
• Each time the status is updated, two lines are appended: * A timestamp, in the form of an integer (seconds passed since 1970) * A number, indicating the number of files changes in the /model/Scade folder

  Reasoning

Just updating the status would be problematic: imagine two subsequent changes that are picked up by Jenkins: the first one with changes in the Scade folder, the second without. If we'd just update the status, the change could be missed. Using timestamps allows the requesting script to memorize how far the status file had been processed last time. Any non-zero value since that timestamp would count as a significant change.

[jastram]

Possible solution discussed by @MarcBehrens and @jastram :

We already have Jenkins (via CloudBees) in place. Jenkins is capable of observing a git repository. Therefore, the process could work as follows:

• We set up a dedicated Jenkins project for the model building
• This project is configured to observe the model files in the git repository
• If a relevant change is detected, Jenkins will push a new file (the trigger-file) to the git repository (e.g. /build-pricess/build-requested.txt)
• The build computer (at DLR) runs a cron job that checks for the existence of that file on a regular basis.
• If the file is found, it triggers the building locally
• Upon completion, the build artifacts are processed (pushed to git, uploaded somewhere, etc.)
• Upon completion, the trigger-file is deleted and the deletion pushed back to git.

[MarcBehrens]

The environment and automation for the build computer is already available. First results of a complete code generation and compilation is available at https://github.com/openETCS/modeling/pull/384 . Is there an upate on the CI?

[jastram]

Hello Marc,

Unfortunately, nothing new yet, it's in the pipeline.

[jastram]

Hello Marc,

We're making some progress here and started to set up the infrastructure. While we do this, a few questions:

1. What changes shall we monitor, exactly: Which repository (validation or model)? Which files?
2. We can put a marker file in the repository, as proposed above. But keep in mind that this will create a lot of commits that are not really meaningful. Is this really what you want? It's probably very convenient for you. If you want Jenkins to write a marker file, then Jenkins needs write permissions to the repository. We'll have to set up a corresponding account.
3. If you decide against a marker file in the repository, we can put a marker file on the cloudbees server. You could then simply check the content of that file in a regular interval by retrieving it with wget or similar.

So: Which route would you like to go, 2 or 3? My recommendation is 3.

• Michael

[MarcBehrens]

Hello Michael,

To 1) since we are working on SCADE code the entrance point is the folder

https://github.com/openETCS/modeling/tree/master/model/Scade

and below.

To 2) and 3) A marker file would be a good way for the second step.

I think the first step would be to get a nightly automatic code generation and compilation running with the aim to provide a nightly executable of the OBU.

This means that we would provide the nightly code which is then further processed.

Should we also provide the executable or is it possible to process this on cloudbee?

In a second step we could then talk about a marker or summary within the cloudbees server providing only an update of code/executable when changes occur.

I would like to keep the commits to the modeling repository as slim as possible.

Kind regards

Marc

[jastram]

Hello Marc,

Please check out the updated description of this issue, where I propose a new implementation. Would that work for you?

[jastram]

@MarcBehrens - CloudBees is now connected to the modeling repository and checks once an hour for changes. As far as I am concerned, you can start using this for your automation.

So essentially, we are done with this. I will therefore close this issue.

I filed #492 as a follow-up, as you requested at the Backlog grooming session.

[RobertoKretschmer]

I will be out of the office starting 05.06.2015 and will not return until 20.06.2015

Ich werde mich sobald wie m�glich bei Ihnen melden. Bei dringenden Angelegenheiten wenden Sie sich bitte an: Carsten.Kuebler@twt-gmbh.de

I will get back to you as soon as possible. In case of urgent matters please contact: Carsten.Kuebler@twt-gmbh.de

Hinweis: Dies ist eine automatische Antwort auf Ihre Nachricht  "Re: [toolchain] Requirements: Model-building CI (#487)" gesendet am 09.06.2015 23:00:27.

Diese ist die einzige Benachrichtigung, die Sie empfangen werden, w�hrend diese Person abwesend ist.