Mitigate potential testing failure cases for discovery

AMWA-TV / nmos-testing

Testing tool for the AMWA NMOS Specifications

https://specs.amwa.tv/nmos-testing/

Apache License 2.0

39 stars 48 forks source link

Mitigate potential testing failure cases for discovery #344

Open andrewbonney opened 5 years ago

andrewbonney commented 5 years ago

Discovery testing (currently only used in IS-04-01) isn't completely straightforward as it requires some form of isolation between the testing tool and the device under test. It may be possible to improve this (or at least indicate where errors may be occurring) as follows:

When testing unicast or multicast discovery, perform an mDNS browse for any advertised registries which may interfere with the process, and produce a warning.
Prevent unicast DNS queries to the mock DNS server from anything other than the IP of the device under test.

garethsb commented 1 year ago

These issues were noted again during JT-NM Tested August 2022.

Reports:

IS-04-01 test_05: there was a few times sign of another node interacting with the test tool, how can this happen/be avoided?
test suite indicated pass to IS-04-01 test_02 while the device doesn't support DNS-SD. spotted for 1 vendor P..C - false positive.

garethsb commented 1 year ago

Also after discovery, same mitigation could be beneficial on the mock Registration API, System API etc. See also https://github.com/AMWA-TV/nmos-testing/issues/508...

Previously concern was expressed that some devices might make DNS-SD or API requests from different IP addresses than the one on which e.g. the Node API is being tested. So we might need to add an additional/optional field to provide this info before starting a test run.