2020/12/16/testing

[utterances-bot]

Testing robotics systems | Michael Jae-Yoon Chung

Starcraft II, Photo by Zach Tirrell on flickr

http://localhost:4000/2020/12/16/testing.html

[jimmybaraglia]

NIce post on the difficulties of testing automation systems Mike!

I am curious to hear your opinion on simulating tests for hardware issues. How can this be achieved efficiently?

Indeed, the search space is unbounded. Anything can go wrong at the hardware level and potentially affect the system, or break it. Is it therefore possible for a developer, or even an AI, to create a test for everything that could go wrong? Would using HITL tests be more approprioate?

[mjyc]

Thanks for reading and posting the first comment on my blog!

... simulating tests for hardware issues ...

Do you mean like auto-generating test cases? That's an interesting idea. What kind of hardware issues do you want to test? To test unreliable hardware issues (e.g., bringup failure), I could imagine first modeling the hardware behavior (e.g., in simulation) somehow (e.g., by hand or ... RL?) and building some mechanism to auto-generate test cases. However, I'd imagine doing so wouldn't be cost-effective (at least in near future) in practice. A lot of work to build/maintain/improve hardware behavior models but there are always new critical failure cases to cover, i.e., there's a long tail of possible failure cases--I assume this is what you meant by "unbounded", and even if such thing exists today, it would take sometime to gain developer trusts.

Instead, I've seen companies setting up stress test physical environments for testing individual hardware devices in isolation or end-to-end testing top-level applications--which is also costly but I see some form of this in every robotics company I've seen.

Regarding HIL testing, from what I've seen they are used to create interesting scenarios to test software components on the same compute hardware as in production and not much focused on testing (e.g., sensor/actuator) hardware issues.

[slivingston]

Thanks for writing this. First, a small comment: the link https://www.airbotics.io/blog/software-deployment-landscape is not valid. The most recent copy I found in the Internet Archive is https://web.archive.org/web/20230330175041/https://www.airbotics.io/blog/software-deployment-landscape

Second, it is surprising to read an article about testing robots in industry without any reference to verification and validation (V&V) as used for safety-critical systems. Has safety not been important for the kinds of robots you have built?

[mjyc]

Thank you for checking out this post and bringing up the missing safety literature.

While the safety was important to the robots I worked with in the past, my direct experience in handling safety-related responsibilities has been limited. I didn't have well-organized insights on the topic.

That being said, I would greatly appreciate safety references you might recommend. Safety is undoubtedly a vital topic. I would like to update the content once I gain more insights.