search

project-chip / rs-matter

Rust implementation of the Matter protocol. Status: Experimental
Apache License 2.0
303 stars 43 forks source link

[TESTS] End-to-end tests #187

Open ivmarkov opened 1 month ago

ivmarkov commented 1 month ago

I have the not so comfy feeling, that the tests covering rs-matter are just not enough, and so we are prone to regressions as we develop it over time. (Guilty as charged here, btw.)

Unit tests are one thing, but they can only go so far. Also, the higher up you climb the stack, the more difficult it becomes to create and maintain such ones.

Though - I must say - the way I've influenced the existing semi-unit-tests/semi-e2e-tests that originally tested the Interaction/Data Layer is that they are currently more of an e2e test suite - as they are exercising aspects of the transport layer too - than what they were originally - tests bound to the Data layer only. So we might evolve this "framework" too, but writing the test cases themselves is super-boring, ergo, I'm looking at reusing someone else's test cases.

Regardless, and without digressing any further, the question: @kedars, @andreilitvin: Are there / is there an e2e test framework in the C++ SDK that we can somehow extract and re-use in the context of rs-matter?

I'm imagining - roughly speaking - a bunch of automation tests - likely in the form of shell or Python scripts exercising the chip-tool command line and executed against an up-and-running version of the C++ SDK (which we should be able to replace with an up-and-running rs-matter stack).

So, does such an animal exist, in there? Can you point me to it?

ivmarkov commented 1 month ago

Hm, maybe this one?

andreilitvin commented 1 month ago

We have 2 sides:

In theory, a full python driven test (or any test for that matter) should be able to run against any app running on linux, including a rust application. See the app argument in https://github.com/project-chip/connectedhomeip/blob/master/.github/workflows/tests.yaml#L478

It may be useful to create example apps in rust and test them against the C++ test suite.

--Andrei

On Thu, Jun 13, 2024 at 3:54 AM ivmarkov @.***> wrote:

Hm, maybe this one https://github.com/project-chip/connectedhomeip/tree/master/src/python_testing ?

— Reply to this email directly, view it on GitHub https://github.com/project-chip/rs-matter/issues/187#issuecomment-2164900164, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB26UOD6TKXLHZC7O5NNT6DZHFF37AVCNFSM6AAAAABJHYNX7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRUHEYDAMJWGQ . You are receiving this because you were mentioned.Message ID: <project-chip/rs-matter/issues/187/2164900164 @.*** com>