Better/more straightforward integration with greentea testing

I think this powerful test suite would be made even better with a more straightforward integration with Mbed's existing greentea testing system.

While this test suite was originally developed for integration tests of the BLE stack, if combined with the greentea testing suite it would provide an ideal way of performing integration tests on BLE services.

My vision for Mbed BLE is to eventually offer a number of standard BLE services (currently working on a more robust implementation of the old UARTService).

This is similar to what Mbed did in the past (and still does). Some of the previous standard Mbed BLE services were clearly lacking in terms of robustness, usability, and quality and were recently removed from Mbed master.

I think Mbed should provide common-use-case BLE services that are put through the same rigorous testing as the BLE stack.

I'm currently working on an integration test using this suite and the greentea suite for the above UARTService implementation. My integration test is set up so that a single DUT runs the greentea client (which hosts a GattServer with a UARTService). Using the greentea framework, I plan to call python host test scripts to interact with boards running the ble-cliapp. The host test scripts will manipulate the boards running ble-cliapp to interact with the greentea DUT over BLE.

I'm starting this issue to collect feedback on the proposed "standard services" idea and the integration testing strategy.

The change I really need is:

The official test_suite python package must be installable so that out-of-source python scripts can import things from the common directory. This would provide most of what the host tests need to interact with connected boards running the ble-cliapp.

Not only would this allow Mbed to provide standard BLE services that are thoroughly tested, it would provide an example of how customers can use these tools to do integration testing on their own end devices running the Mbed BLE stack.

@pan- @paul-szczepanek-arm Thoughts?

Many things here 👍 .

Standard collection of BLE services

We would love to have standard services and common services (such as a UART port, FW upgrade transfer, ...) accessible to our users, it has always been in our backlog but never a priority. That being said, the connectivity team would gladly offer it's active support in review/guidance/organization for any community effort in that direction.

I strongly believe the mbed OS repository is not the place where it should happen: there is no policy for experimental code which lead to code harder to maintain if the code committed was wrong or the API were incomplete as we want keep backward compatibility between releases.

If such effort was happening I think it would be reasonable to open a repository in Github just for these services. Once the services are mature enough, we can think of releases and development following semantic versioning. It should be easier to issue a major version if needed compared to Mbed OS. You really don't want to maintain for eternity a rotten piece of code.

Greentea and Bluetooth test suite

We've been thinking of the using greentea for the test suite but the model didn't seems to really fit with what we've been doing with BLE as we need to interact with multiple BLE devices. That's how the ancestor of icetea was born. It eventually died as the only thing you need is pytest and a way to communicate with your DUT: there's little point in reinventing a complete test framework and maintaining it if it just offer worst features than open source counter part.

What benefit do you see in using greentea instead of what is present in this repository ? It is possible, today, to create specialized test application using https://github.com/ARMmbed/mbed-os/tree/master/features/frameworks/mbed-client-cli and interact with them using the Device class.

In all case (greentea or not), the technical challenge seems to be around board allocations and retrieving the right board with the right binary when the test is run.

If we had to publish a package, I don't think it has to be the test suite, the common part of it should be sufficient.

@pan-

Standard collection of BLE services

Next on my list after UART is some sort of FW upgrade transfer. Though I'm thinking the progression will more likely go:

UARTService |---> GuaranteedUARTService (subclass using write w/ response and indications) |---> DFUService (subclass of GuaranteedUARTService w/ configurable privacy and security requirements) (side note: DFUService serial stream will just write raw bytes from the stream to a given BlockDevice so any update mechanism can be used :) )

Of course, there are more design considerations to think about...

I agree that mbed-os itself is not the right place for these common services. It would be nice to have an official ARM repository for them though.

Greentea and Bluetooth test suite

In the case of testing services, I think the ble test suite + greentea model works well. I started with the idea because I was already fairly familiar with how the whole greentea system worked (from my previous work on porting nRF52840 USB to Mbed).

The reason I think it works for testing services is that usually, you only have one DUT running the service whereas you want to test how the service implementation may perform under load or with multiple simultaneous connections. Using greentea lets you write an app as you would normally using the service in Mbed and not have to pull in all that extra client cli code each time you want to write a test for a service. I think this would make it easier to maintain tests themselves.

The power of the blecli-app and python library is that it lets you orchestrate multiple Mbed BLE devices from one script. I see it's usefulness far beyond just integration testing.

I have already used the test_suite outside the pytest framework to write data collection scripts for BLE sensors without having to worry about platform support. Cross-platform support for generic (vs. common BLE devices like mice) use of BLE GATT is quite poor, especially in python. There are a million packages that are nearly full-featured but fall short somehow. Even if they have all BLE features available, they only work on Windows or something.

With this, you just plug in an Mbed board with ble-cliapp loaded and use the python test_suite to interact with your BLE devices. The BLE API is then already familiar and you have more direct control over the BLE connection.

I was able to get greentea and the BLE test_suite to play nicely together today. I wrote a simple host test that calls into the test_suite functions to control other BLE-enabled boards that are not the target of the host test runner.

You can see how I did this here: https://github.com/EmbeddedPlanet/ep-oc-mcu/blob/ble-uart-service/features/FEATURE_BLE/experimental/TESTS/host_tests/ble_serial_test.py

I'll point out a couple really important lines to making them work together:

class UARTSerialAllocator(fixtures.BoardAllocator):

    def __init__(self, ht : BaseHostTest, platforms_supported: List[str], binaries: Mapping[str, str],
                 serial_inter_byte_delay: float, baudrate: int, command_delay: float):

        # Call the BoardAllocator init
        super(UARTSerialAllocator, self).__init__(platforms_supported, binaries, serial_inter_byte_delay, baudrate,
                                                  command_delay)

        self.port = ht.get_config_item("port")

        # Remove the serial port allocated to the host test
        self.allocation = [x for x in self.allocation if self.port not in x.description['serial_port']]

This subclass of fixtures.BoardAllocator removes the "allocated board" that is being controlled by greentea, preventing any conflicts.

I'm open to how you suggest doing the service tests but I want to know what you think about my proposal first.

Are there any future use cases where either of our solutions doesn't work?

Thanks for your detailed reply @AGlass0fMilk.

Standard collection of BLE services

I will create a repository in ARMmbed for standard/common services today.

Greentea and Bluetooth test suite

That's an interesting way of tackling the problem. I think I would have processed differently building an application using mbed-client-cli with the commands limited to the bare minimum. For example a command that starts advertising or another that enables the echo mode. The DUT would emit traces when events happens (connection, failure, peer connected to serial, ...) that are parsed by pytest which can assert if the device is acting correctly or not.

I would process the same way to test examples were the example under test just output traces (no need for command I suppose, the behavior is well defined).

But one size rarely fits all, it is always interesting to have different option to work with. I'd like to understand when using greentea shines compared to the approach I highlighted above.

In all cases, it has a lot of sense to put the common part of the test suite in its own package so it can be reused in new ways.

For unit testing internally we use gtest/gmock to mock classes and simulate the behavior when event are received.

@pan- ~I'm having a trouble with multiple client testing... I put it on the forum because I'm not sure it's a bug worthy of an issue:~

https://forums.mbed.com/t/ble-cordio-peripheral-multiple-central-connections/10719

Can you make the services repository? I have a bunch of ideas for potential services to support in the future that we can discuss there :)

@AGlass0fMilk I'm quite busy, but I created a repo: https://github.com/ARMmbed/mbed-os-experimental-ble-services . It is naked but I will add issues for standard services and common services. Please feel free to do the same.

ARMmbed / mbed-os-bluetooth-integration-testsuite