Add workflow for La Semaforica Cartesio

[emiltin]

The runner has been added. Just waitning for the test config from La Semaforica.

[emiltin]

GitHub reports that Ruby must be installed manually on self-hosted runners:

Run ruby/setup-ruby@v1 Using 3.3.2 as input from file .tool-versions Error: The current runner (ubuntu-22.10-x64) was detected as self-hosted because the platform does not match a GitHub-hosted runner image (or that image is deprecated and no longer supported). In such a case, you should install Ruby in the $RUNNER_TOOL_CACHE yourself, for example using https://github.com/rbenv/ruby-build You can take inspiration from this workflow for more details: https://github.com/ruby/ruby-builder/blob/master/.github/workflows/build.yml $ ruby-build 3.3.2 /<...>/_tool/Ruby/3.3.2/x64 Once that completes successfully, mark it as complete with: $ touch /<..>/_work/_tool/Ruby/3.3.2/x64.complete It is your responsibility to ensure installing Ruby like that is not done in parallel.

[mariusrazvanplugaru]

The runner is active with the controller reachable from my PC, must configure something else? It would be possible to change the name in La Semaforica - TECSEN - Cartesio? KR, Marius

[emiltin]

The runner is active with the controller reachable from my PC, must configure something else? It would be possible to change the name in La Semaforica - TECSEN - Cartesio? KR, Marius

Yes, you should install Ruby 3.3.2 in the $RUNNER_TOOL_CACHE yourself, for example using https://github.com/rbenv/ruby-build.

[mariusrazvanplugaru]

Thank you! Now i will install, please give me some min.

[emiltin]

It would be possible to change the name in La Semaforica - TECSEN - Cartesio?

Do you need both names, or TECSEN rather than La Semaforica?

[emiltin]

Since the product seems to be known as La Semaforica Cartesio, according to https://lasemaforica.com/en/products/cartesio-highest-processing-power-interactive-traffic-controllers/ I would prefer to use that as the name for the workflow.

[mariusrazvanplugaru]

Since the product seems to be known as La Semaforica Cartesio, according to https://lasemaforica.com/en/products/cartesio-highest-processing-power-interactive-traffic-controllers/ I would prefer to use that as the name for the workflow.

We speak also with my colleagues and we preffer to mantain both names.

[emiltin]

You can check the latest result of the Cartesio tests runs at https://github.com/rsmp-nordic/rsmp_validator/actions/workflows/semaforica_cartesio.yaml

That's were it currently reports that you need to install Ruby first.

[emiltin]

Currently the log include errors in Italian, eg: mv: impossibile eseguire stat di 'log/validation.log': File o directory non esistente

Would it be possible for you set your runner machine to use english as language? It might make it a bit easier for us to help you with debugging things related to the validator and github actions.

[mariusrazvanplugaru]

Currently the log include errors in Italian, eg: mv: impossibile eseguire stat di 'log/validation.log': File o directory non esistente

Would it be possible for you set your runner machine to use english as language? It might make it a bit easier for us to debug things related to the validator and github actions.

Yes, no probl. We are trying to install ruby manually because we are not using mac on this machine but ubuntu. As soon as i have installed i change the language and restart all.

[emiltin]

I recommend installing ruby using a version manager like rbevn, or mise. That makes it a lot easier to upgrade versions. Also there's a .tool-versions file in the validator repo, which will make the version manager automatically use the right ruby version - or notify you if it's not yet installed.

[mariusrazvanplugaru]

I reactivate the runner. I have installer ruby 3.2.2 and changed the language in English. Could you restart tests?

[mariusrazvanplugaru]

[emiltin]

Same error: In such a case, you should install Ruby in the $RUNNER_TOOL_CACHE yourself, for example using https://github.com/rbenv/ruby-build

I guess Ruby must be installed in a location where the runner expects it, although I'm not sure how $RUNNER_TOOL_CACHE works.

I'll have to leave now, but we can pick it up next week. David @otterdahl will be back from vacation, he has experience with setting this up with other controllers.

[mariusrazvanplugaru]

I'm also executing this:

$ ruby-build 3.3.2 /home/marius/Scrivania/Progetti/actions-runner/_work/_tool/Ruby/3.3.2/x64 Once that completes successfully, mark it as complete with: $ touch /home/marius/Scrivania/Progetti/actions-runner/_work/_tool/Ruby/3.3.2/x64.complete It is your responsibility to ensure installing Ruby like that is not done in parallel.

Let's hope that this solves the issue.

[emiltin]

OK let me rerun the test again.

[mariusrazvanplugaru]

Sorry i restart the runner.

[mariusrazvanplugaru]

Now it works ?

[emiltin]

Now it works ?

Yes, now it seems to actually be running the validator! Now the question is whether the TLC can contact the validator, on the port configured.

[mariusrazvanplugaru]

I see changes on functioning so this is good

[mariusrazvanplugaru]

I see that all checks have passed. Is the final result?

[emiltin]

All tests green, well done!

Connection to site CAR_TC_001 established, using core 3.1.5, tlc 1.0.15
...
Finished in 7 minutes 1 second (files took 0.29828 seconds to load)
89 examples, 0 failures

5 sentinel warnings:

  5 RSMP::RepeatedAlarmError

There are a few sentinel errors. These are not failed tests, but instances where the validator noticed that the TLC sends the same alarm again, which it should not.

[mariusrazvanplugaru]

Ok, thank you! This is enough to be pubblished on rsmp-nordic site? The next week i'm on summer vacantion, i'll be back on 19 August and i hope that we can continue to test the newest versions. KR

[emiltin]

I'll fix the naming next week, and also let @otterdahl review the PR before merging.

I will also add a badge at https://rsmp-nordic.org/compliance/.

After merging, the test will be run again whenever there are changes to the validator (e.g. when we add or fix a test case), and also nightly at 22PM UTC.

[mariusrazvanplugaru]

I'll fix the naming next week, and also let @otterdahl review the PR before merging.

I will also add a badge at https://rsmp-nordic.org/compliance/.

After merging, the test will be run again whenever there are changes to the validator (e.g. when we add or fix a test case), and also nightly at 22PM UTC.

'and also nightly at 22PM UTC.' what mean that? We must leave the system active at this hour?

[mariusrazvanplugaru]

I'll fix the naming next week, and also let @otterdahl review the PR before merging.

I will also add a badge at https://rsmp-nordic.org/compliance/.

After merging, the test will be run again whenever there are changes to the validator (e.g. when we add or fix a test case), and also nightly at 22PM UTC.

Thank you for all!

[mariusrazvanplugaru]

Another thing, if is possible we would like to insert also TECSEN in your partners list and add this second name also here on product - project. Best regards, have a nice weekend!

[emiltin]

Another thing, if is possible we would like to insert also TECSEN in your partners list and add this second name also here on product - project. Best regards, have a nice weekend!

No problem just send an email to Dia https://rsmp-nordic.org/contact/#dia-bækgaard-pedersen.

[otterdahl]

Look good. Great work. Feel free to merge

[emiltin]

'and also nightly at 22PM UTC.' what mean that? We must leave the system active at this hour?

The preferrred setup is to leave the TLC always connected to the runner. Test are run whenever the validator is updated, and also once every night. This way you can update the software on the TLC and get fresh results.

We plan to update the reporting of results to include things like, core/sxl versions, date of last test, percent of tests passed within the last months, etc.

[mariusrazvanplugaru]

'and also nightly at 22PM UTC.' what mean that? We must leave the system active at this hour?

The preferrred setup is to leave the TLC always connected to the runner. Test are run whenever the validator is updated, and also once every night. This way you can update the software on the TLC and get fresh results.

We plan to update the reporting of results to include things like, core/sxl versions, date of last test, percent of tests passed within the last months, etc.

Good morning, our intention is to validate multiple versions of the RSMP core version - SXL version combination. The list is below.

RSMP Core	SXL
3.1.5	1.0.15
3.1.5	1.1
3.1.5	1.2
3.1.5	1.2.1
3.2.1	1.0.15
3.2.1	1.1
3.2.1	1.2
3.2.1	1.2.1
3.2.2	1.0.15
3.2.2	1.1
3.2.2	1.2
3.2.2	1.2.1

Does this mean that for each version we must have an available controller and an associated PC with the runner active? Thank you!

[emiltin]

hi @mariusrazvanplugaru, great to hear you suport the various versions of core and sxl, and want to test them. I think it would impractical to have so many controllers connected. Does your controllers support different versions without changing firmware?

How to test a matrix of core/sxl versions is not clear yet. A challenge is that while the core version is negotiated when connecting, the sxl version is simply stated by the connecting site, see https://github.com/rsmp-nordic/rsmp_core/issues/65.

When running tests, the validator acts as a supervisor. So the validator cannot control what version of the sxl is being tested. To able to test different versions, we would need a way for the controller to connect using different sxl versions in turn.

An alternative would be to modify the core spec so that the sxl version is also negotiated during connection, so that the validator can influence the sxl version used. This would be my preference.

But for now, you can try to connect using different versions. Note that the test config must be updated to match the versions used by the controller. If you want to ty this, I suggest you open a PR where you modify the Cartesio workflow to specify another version combination. When the controller uses that version we can add the 'testhub' tag to the PR, and the tests will run.

[emiltin]

Regarding the core version, this is negotiated, so we should be able to extend the test hub to run the test suite multiple times, using different core versions.

[emiltin]

I've opened an issue about running tests on a matrix of core/sxl verisons: https://github.com/rsmp-nordic/rsmp_validator/issues/406

[emiltin]

Will you be able to keep the controller connected?

[mariusrazvanplugaru]

hi @mariusrazvanplugaru, great to hear you suport the various versions of core and sxl, and want to test them. I think it would impractical to have so many controllers connected. Does your controllers support different versions without changing firmware?

How to test a matrix of core/sxl versions is not clear yet. A challenge is that while the core version is negotiated when connecting, the sxl version is simply stated by the connecting site, see rsmp-nordic/rsmp_core#65.

When running tests, the validator acts as a supervisor. So the validator cannot control what version of the sxl is being tested. To able to test different versions, we would need a way for the controller to connect using different sxl versions in turn.

An alternative would be to modify the core spec so that the sxl version is also negotiated during connection, so that the validator can influence the sxl version used. This would be my preference.

But for now, you can try to connect using different versions. Note that the test config must be updated to match the versions used by the controller. If you want to ty this, I suggest you open a PR where you modify the Cartesio workflow to specify another version combination. When the controller uses that version we can add the 'testhub' tag to the PR, and the tests will run.

The same firmware of Cartesio can support different combinations of RSMP core version - SXL version but can be set in configuration only one combination. Any automatic choice by the supervisor cannot be made as SXL cannot be negotiated in the version to be tested - how you said. So during our firmware update we should manually carry out a test for combination 1, a test for combination 2 etc... We are available to carry out all tests for all versions of the combined RSMP - SXL at each firmware update. However, it seems more reasonable to us to run a single test for each RSMPSXL version(1), Cartesio version(2) and Validator version (3) and not every day. If one of conditions (1), (2) or (3) changes then a new test will be agreed. What do you think about? How can we continue?

KR

[emiltin]

I think a good next step would be to test with the latest core and sxl version.

We should also try to modify the validator so that it can automatically test on multiple core versions, but this will take a bit of time/consideration to implement.

The reasons we recommend having the TLC online all the time is that:

1) The validator is being develop actively and there are changes often. It's practical for us to be able to rerun tests whenever needed. 2) Some bugs occur only occasionally and having many runs greatly improves the accuracy of testing and can help provide a lot of insights into when bugs occur. Keep in mind that this is not a software unit tests where we can guarantee the starting conditions, and each run will be deterministic. Although we do what we can to reset the TLC when testing starts, there is often many things that vary in the TLC between runs; things like the clock, signal group stages, sensor states, etc.

[mariusrazvanplugaru]

Ok, we understand. We can leave the TLC online for testing. However, we don't understand how we can test for multiple versions of RSMP/SXL and have the green flag at https://rsmp-nordic.org/compliance/. At the moment, if we understand correctly, the green flag is only possible for one version of RSMP/SXL. Instead, we would like to have the green flag for multiple RSMP/SXL versions.

Thank you! KR,

[emiltin]

You're right, our test hub infrastructure currently only handles a single version. We need to extend this to handle several core/sxl versions. In general we also want to provide more information about test result on our public website.

[emiltin]

Since this PR is closed, let's continue the disussion about how to test different core/sxl version at https://github.com/rsmp-nordic/rsmp_validator/issues/406.