Add test: Add functional TLC tests

[otterdahl]

It would be useful to add tests for functional aspects of the TLC.

For instance:

• Variable red. Defined in signal group status. Also known as the R in LHOVRA
• Green-yellow-green sequence, known as A in LHOVRA, section 3.8

[emiltin]

What would these tests do? Would they look at the state of signal groups over time?

[otterdahl]

What would these tests do? Would they look at the state of signal groups over time? Yes. In combination with commands, e.g. activating detector logics.

I don't know if these kinds of tests are within scope of rsmp_validator, but they would be very useful to verify the functionality of the TLC.

An example could be that by activating a certain detector logic a signal group is expected to turn green within a given time. It can also check the signal group sequence in order to verify that the functions (e.g. 'varible red') actually works.

[emiltin]

I think these kinds of tests are within scope. But we need to consider how you would configure the tests to tests various signal programs?

[otterdahl]

Yes. The config file needs to define the mapping between detector logics and signal group.

Each detector logic is expected to turn a single signal group to green within a certain time. When the detector logic no longer is active the signal group either stays in green ("green passive" or similar) or turns red. This list is needed to tell which status a signal group has (red/yellow/green)

Perhaps something like this?

      components:
        main:
          KK+AG9998=001TC000:
        signal_group:
          KK+AG9998=001SG001:
        detector_logic:
          KK+AG9998=001DL001: 1
          KK+AG9998=001DL002: 1
          KK+AG9998=001DL003: 1
          KK+AG9998=001DL004: 1
          KK+AG9998=001DL005: 1
          KK+AG9998=001DL006: 2
          KK+AG9998=001DL007: 2
          KK+AG9998=001DL008: 2
          KK+AG9998=001DL009: 2

In the example above detector logics 1-5 controls signal group 1 and detector logics 6-9 control signal group 2. This example is taken from the current signal program that runs in the Swarco simulator. It it possible to export/convert his information from the TLC configuration file.

[emiltin]

I think we might be more explicit about what happens when a detected is activated, especially if requesting green for a group is not the only thing a detector might do? eg.

components:
  main:
    KK+AG9998=001TC000:
  cycle_time: 60
  signal_groups:
    KK+AG9998=001SG001:
    KK+AG9998=001SG002:
  detector_logics:
    KK+AG9998=001DL001:
        action: request
        phase: green
        group: KK+AG9998=001SG001

Can a detector request other things than going to green? Can a detector affect more than one group?

[emiltin]

The more of these things we define in the yaml, the more it sounds like a way to specify a signal program in rsmp. It would be neat if you could send the yaml above to the TLC to define the signal program.

[emiltin]

To do functional tests we might also need to know e.g the cycle time of the signal program?

[otterdahl]

I think we might be more explicit about what happens when a detected is activated, especially if requesting green for a group is not the only thing a detector might do? eg.

components:
  main:
    KK+AG9998=001TC000:
  cycle_time: 60
  signal_groups:
    KK+AG9998=001SG001:
    KK+AG9998=001SG002:
  detector_logics:
    KK+AG9998=001DL001:
      action: request
      phase: green
      group: KK+AG9998=001SG001

Can a detector request other things than going to green?

A detector logic either request a signal group (to become green) and/or extends a signal group (to stay in green) for a specific amount of time.

Perhaps:

detector_logics:
    KK+AG9998=001DL001:
        request: true
        extend: true
        group: KK+AG9998=001SG001

Can a detector affect more than one group?

A physical detector can, but not a detector logic. It always affects a single signal group. Another detector logic needs to be defined to affect another signal group (as far as I understand). The detector logic also contain other variables as well, such as various time outs, how long time it should extend the signal group, etc.

[otterdahl]

The more of these things we define in the yaml, the more it sounds like a way to specify a signal program in rsmp. It would be neat if you could send the yaml above to the TLC to define the signal program.

Yes. It contains the same type of logic. It's a way to validate the signal program.

[otterdahl]

Do you functional tests we might also need to know e.g the cycle time of the signal program?

Not necessarily. For some types of tests perhaps. In general, the cycle time is not used in "isolated" control mode or in local coordination

[emiltin]

the tag :functional was added to the small number of functional tests that are currently included in the validator test suite.

[sveitech]

Bear in mind that functional tests can be difficult to maintain. For the ITC-3, different versions might behave differently, and may need updated configuration files.

Consider splitting the validator into two separate parts, so the functional tests can be run in isolation from the core specification tests. A controller can be 100% RSMP core valid, but still have a functional behavior that deviates from your functional tests due to configuration.

You are slowly moving away from the core responsibility of this repo: Testing that the RSMP protocol is in compliance.

Instead you are testing that a traffic controller behaves as you think it will, at the time you write your functional tests. I can guarantee you that these tests will be brittle and unmaintainable.

[emiltin]

Thank you for the input. I agree that there's a distinction between testing the protocol aspects, versus testing functional behaviour. That's why we're now tagging functional tests, so you can easily exclude them, or run only those tests, if desired.

Regarding the core responsibility of the validator, I don't think there was ever a decision to limit the validator to just the protocol aspects. It was of course the natural starting point, but for road authorities, it's also useful to be able to test functional aspects of equipment.

However, at the moment, the functional tests still comprise only a small fracrtion of the test suite, and is really a topic we're just starting to look at.

As you say, functional tests rely on additional configuration. That's something to keep in mind when we move forward with these kinds of test.The official repo should focus on what's useful to most road authorities and suppliers. If someone have specialized needs, they can add their own tests (in their own repos).

[otterdahl]

I agree that functional tests may require a lot of extra configuration and the tests can become quite specific for a certain TLC configuration or for the requirements of a specific country/road authority.

But I do think it important to point out that the functional test cases verifies that there is actual logic behind the commands and statuses. It also tries to a certain degree to check that they were interpreted correctly. It's otherwise pretty straight forward to simply reply with a valid CommandResponse or StatusUpdate for each status and command of the SXL. But we want to check that the controller behaves as expected and that there is actual logic behind the curtains.

I think it is a reasonable solution to use tags (e.g. :functional) to filter which tests are performed in order to lower the requirements of using specific TLC configurations in those cases where it is not needed or relevant.

[emiltin]

a possiblity is to exclude functional tests by default using the .rspec file.