MicroOCA compliance

[dbaudio-sfe]

Hi!

Using the OCA compliance tool on the MicroOCA implementation revelas a few gaps in objects, managers and events. We used the compliance tool 2.0.1 and parameters -s tcp -v AES70-2015

Is there a compliance test that does pass the MicroOCA stack? Or any effort on the way to make it comliant? Or should that be something to be started?

Thanks. Sebastian

Testresult excerpt: ... [Minimum object compliancy test][Result] [Minimum object compliancy test][Result] Starting Test : Minimum object compliancy test [Minimum object compliancy test][Result] [Test Suite][Result] - Prepare test (Minimum object compliancy test) result: Passed [Minimum object compliancy test][Result] Testing mandatory manager(s) for test device [Minimum object compliancy test][Result] Testing 4 mandatory manager(s) for compliancy [Minimum object compliancy test][Result] ERROR: Device must implement method GetControlNetworks (index 3) for object OcaNetworkManager (ONo 6, result 8) [Minimum object compliancy test][Result] ERROR: Failed to test the methods of MANDATORY manager 6 for type OcaNetworkManager [Minimum object compliancy test][Result] ERROR: Failed to test managers [Minimum object compliancy test][Result] Testing mandatory object(s) for test device [Minimum object compliancy test][Result] Testing 8 mandatory object(s) for compliancy [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 100, result 8) [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 100, result 8) [Minimum object compliancy test][Result] ERROR: Failed to test methods of MANDATORY object 100 of type OcaBlock [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268435456, result 8) [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268435456, result 8) [Minimum object compliancy test][Result] ERROR: Failed to test methods of MANDATORY object 268435456 of type OcaBlock [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268435712, result 8) [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268435712, result 8) [Minimum object compliancy test][Result] ERROR: Failed to test methods of MANDATORY object 268435712 of type OcaBlock [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268436224, result 8) [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268436224, result 8) [Minimum object compliancy test][Result] ERROR: Failed to test methods of MANDATORY object 268436224 of type OcaBlock [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268436736, result 8) [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268436736, result 8) [Minimum object compliancy test][Result] ERROR: Failed to test methods of MANDATORY object 268436736 of type OcaBlock [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268436992, result 8) [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268436992, result 8) [Minimum object compliancy test][Result] ERROR: Failed to test methods of MANDATORY object 268436992 of type OcaBlock [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268437248, result 8) [Minimum object compliancy test][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268437248, result 8) [Minimum object compliancy test][Result] ERROR: Failed to test methods of MANDATORY object 268437248 of type OcaBlock [Minimum object compliancy test][Result] ERROR: Device must implement method GetStatistics (index 7) for object OcaNetwork (ONo 9000, result 8) [Minimum object compliancy test][Result] ERROR: Device must implement method ResetStatistics (index 8) for object OcaNetwork (ONo 9000, result 8) [Minimum object compliancy test][Result] ERROR: Device must implement method SetSystemInterfaces (index 10) for object OcaNetwork (ONo 9000, result 8) [Minimum object compliancy test][Result] ERROR: Device must implement method GetMediaPorts (index 11) for object OcaNetwork (ONo 9000, result 8) [Minimum object compliancy test][Result] ERROR: Failed to test methods of MANDATORY object 9000 of type OcaNetwork [Minimum object compliancy test][Result] ERROR: Failed to test objects [Minimum object compliancy test][Result] Test Minimum object compliancy test is now unsuccesfull [Test Suite][Result] - Execute test (Minimum object compliancy test) result: Failed [Minimum object compliancy test][Result] [Test Suite][Result] - Cleanup test Minimum object compliancy test done [Test Suite][Result] - Overall test Minimum object compliancy test result: Failed [Test Suite][Result] Executing test: OCC Object Compliancy Tests [OCC Object Compliancy Tests][Result] [OCC Object Compliancy Tests][Result] Starting Test : OCC Object Compliancy Tests [OCC Object Compliancy Tests][Result] [Test Suite][Result] - Prepare test (OCC Object Compliancy Tests) result: Passed [OCC Object Compliancy Tests][Result] Testing 4 manager(s) for test device [OCC Object Compliancy Tests][Result] ERROR: Device must implement event NotificationDisabled (index 1) for object 4 (result 10). It may return not implmented. [OCC Object Compliancy Tests][Result] ERROR: Device must implement event SynchronizeState (index 2) for object 4 (result 10). It may return not implmented. [OCC Object Compliancy Tests][Result] ERROR: Failed to test the events of manager 4 for type OcaSubscriptionManager [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetControlNetworks (index 3) for object OcaNetworkManager (ONo 6, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test the methods of manager 6 for type OcaNetworkManager [OCC Object Compliancy Tests][Result] ERROR: Failed to test managers [OCC Object Compliancy Tests][Result] Testing 191 object(s) for test device [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 100, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 100, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test methods of MANDATORY object 100 of type OcaBlock [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268435456, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268435456, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test methods of MANDATORY object 268435456 of type OcaBlock [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268435712, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268435712, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test methods of MANDATORY object 268435712 of type OcaBlock [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268436224, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268436224, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test methods of MANDATORY object 268436224 of type OcaBlock [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268436736, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268436736, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test methods of MANDATORY object 268436736 of type OcaBlock [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268436992, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268436992, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test methods of MANDATORY object 268436992 of type OcaBlock [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPaths (index 9) for object OcaBlock (ONo 268437248, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetSignalPathsRecursive (index 10) for object OcaBlock (ONo 268437248, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test methods of MANDATORY object 268437248 of type OcaBlock [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetStatistics (index 7) for object OcaNetwork (ONo 9000, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method ResetStatistics (index 8) for object OcaNetwork (ONo 9000, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method SetSystemInterfaces (index 10) for object OcaNetwork (ONo 9000, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetMediaPorts (index 11) for object OcaNetwork (ONo 9000, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test methods of MANDATORY object 9000 of type OcaNetwork [OCC Object Compliancy Tests][Result] ERROR: Failed to test objects [OCC Object Compliancy Tests][Result] Test OCC Object Compliancy Tests is now unsuccesfull [Test Suite][Result] - Execute test (OCC Object Compliancy Tests) result: Failed [OCC Object Compliancy Tests][Result] [Test Suite][Result] - Cleanup test OCC Object Compliancy Tests done [Test Suite][Result] - Overall test OCC Object Compliancy Tests result: Failed ...

[tljdebrouwer]

Hi Sebastien,

I think the device is mostly compliant with the 2018 standard (AES70-2018), but it has been a while ago that I worked on it.

Could you retest it with AES70-2018 as version?

Methods as GetMediaPorts, GetSignalPaths and GetSignalPathsRecursive are no longer mandatory in that version of the standard.

Note that the GetControlNetwork is actually not implemented in this version of the stack, so on that part it might actually be non-compliant. It still uses an older way of handling the control messages.

[dbaudio-sfe]

Good Morning Tom,

thank you for being on this so quickly.

Using the AES70-2018 standard shows

... [Minimum object compliancy test][Result] [Minimum object compliancy test][Result] Starting Test : Minimum object compliancy test [Minimum object compliancy test][Result] [Test Suite][Result] - Prepare test (Minimum object compliancy test) result: Passed [Minimum object compliancy test][Result] Testing mandatory manager(s) for test device [Minimum object compliancy test][Result] Testing 1 mandatory manager(s) for compliancy [Minimum object compliancy test][Result] ERROR: Device must implement method GetDeviceRevisionID (index 20) for object OcaDeviceManager (ONo 1, result 8) [Minimum object compliancy test][Result] ERROR: Failed to test the methods of MANDATORY manager 1 for type OcaDeviceManager [Minimum object compliancy test][Result] ERROR: Failed to test managers [Minimum object compliancy test][Result] Testing mandatory object(s) for test device [Minimum object compliancy test][Result] ERROR: Missing mandatory object OcaControlNetwork [Minimum object compliancy test][Result] Testing 8 mandatory object(s) for compliancy [Minimum object compliancy test][Result] ERROR: Failed to test objects [Minimum object compliancy test][Result] Test Minimum object compliancy test is now unsuccesfull [Test Suite][Result] - Execute test (Minimum object compliancy test) result: Failed [Minimum object compliancy test][Result] [Test Suite][Result] - Cleanup test Minimum object compliancy test done [Test Suite][Result] - Overall test Minimum object compliancy test result: Failed [Test Suite][Result] Executing test: OCC Object Compliancy Tests [OCC Object Compliancy Tests][Result] [OCC Object Compliancy Tests][Result] Starting Test : OCC Object Compliancy Tests [OCC Object Compliancy Tests][Result] [Test Suite][Result] - Prepare test (OCC Object Compliancy Tests) result: Passed [OCC Object Compliancy Tests][Result] Testing 4 manager(s) for test device [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetDeviceRevisionID (index 20) for object OcaDeviceManager (ONo 1, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test the methods of manager 1 for type OcaDeviceManager [OCC Object Compliancy Tests][Result] ERROR: Device must implement method AddPropertyChangeSubscription (index 5) for object OcaSubscriptionManager (ONo 4, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method RemovePropertyChangeSubscription (index 6) for object OcaSubscriptionManager (ONo 4, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test the methods of manager 4 for type OcaSubscriptionManager [OCC Object Compliancy Tests][Result] ERROR: Device must implement event NotificationDisabled (index 1) for object 4 (result 10). [OCC Object Compliancy Tests][Result] ERROR: Device must implement event SynchronizeState (index 2) for object 4 (result 10). [OCC Object Compliancy Tests][Result] ERROR: Failed to test the events of manager 4 for type OcaSubscriptionManager [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetControlNetworks (index 3) for object OcaNetworkManager (ONo 6, result 8) [OCC Object Compliancy Tests][Result] ERROR: Failed to test the methods of manager 6 for type OcaNetworkManager [OCC Object Compliancy Tests][Result] ERROR: Failed to test managers [OCC Object Compliancy Tests][Result] Testing 191 object(s) for test device [OCC Object Compliancy Tests][Result] WARNING: Object OcaNetwork (ONo=9000) is deprecated! (ClassID 1.2.1. Class version 2) [OCC Object Compliancy Tests][Result] Test OCC Object Compliancy Tests is now unsuccesfull [Test Suite][Result] - Execute test (OCC Object Compliancy Tests) result: Failed [OCC Object Compliancy Tests][Result] [Test Suite][Result] - Cleanup test OCC Object Compliancy Tests done [Test Suite][Result] - Overall test OCC Object Compliancy Tests result: Failed ...

So it's missing some different methods here. But the two events are the same (NotificationDisabled and SynchronizeState in teh OcaSubscriptionManager).

[tljdebrouwer]

Hi Sebastien,

There seem different reasons indeed for these faults:

1) [Minimum object compliancy test][Result] ERROR: Device must implement method GetDeviceRevisionID (index 20) for object OcaDeviceManager (ONo 1, result 8)

Can be solved by defining "_DEBUG", which actually seems wrong, we always need to enable this function, see https://github.com/OCAAlliance/OcaToolsAndDemos/blob/master/OCAMicro/OCAMicro/Src/common/OCALite/OCC/ControlClasses/Managers/OcaLiteDeviceManager.cpp#L625-L657

This seems an easy fix.

2) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method AddPropertyChangeSubscription (index 5) for object OcaSubscriptionManager (ONo 4, result 8) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method RemovePropertyChangeSubscription (index 6) for object OcaSubscriptionManager (ONo 4, result 8)

AddPropertyChangeSubscription / RemovePropertyChangeSubscription seems deliberable not implemented. @JeffBerryman do you have some context on this one?

3) [OCC Object Compliancy Tests][Result] ERROR: Device must implement event NotificationDisabled (index 1) for object 4 (result 10). [OCC Object Compliancy Tests][Result] ERROR: Device must implement event SynchronizeState (index 2) for object 4 (result 10).

We don't implement the functions, and that seems ok, so we could implement the events and never thrown them to be compliant. If we do that, that should be an easy fix.

4) [OCC Object Compliancy Tests][Result] ERROR: Device must implement method GetControlNetworks (index 3) for object OcaNetworkManager (ONo 6, result 8)

We could implement the GetControlNetworks and always return an empty list. As mentioned by "WARNING: Object OcaNetwork (ONo=9000) is deprecated! (ClassID 1.2.1. Class version 2)" we implement the legacy network and not the control networks.

Implementing the empty list is easy as well.

So I would like to propose to open issues to solve item 1, 3 and 4. And see what Jeff has to say on item 2.

Would that work for you?

[dbaudio-sfe]

Hi Tom,

I would not want to hide anything to make the stack look compliant. If there are things that just are not, because it's an old implementation and there is not really a need to update it, then that is ok. But if there are some fixes like always implementing GetDeviceRevisionID that's great.

My intention for the compliance tool is to point developers to it so they can quickly get an idea whether basic things are implemented properly. And not getting an error at the end is always nice. But to clearly say you'll get this and that error but that's ok would be fine, too.

Point 1. seems like it should just be done. Getting background to 2. would be good. If there is a good reason then we just have to fail there. I guess implementing events that never get thrown is ok for 3. For 4. I really don't know if that would just pretend to be something it's not. Maybe keeping the error is better here.

[JeffBerryman]

I'm guessing these faults arose from running OCTT on the OCA Micro implementation -- right?

Regarding fault #2: I would guess OCA Micro implements AES70-2015, not AES70-2018. The 2015 version of OcaSubscriptionManager does not include the [Add|Remove]PropertyChangeSubscription() methods. It only defines the more general [Add|Remove]Subscription() methods.

So, if there is an OCTT script for AES70-2015, it should not invoke the fault. Alternatively, the 2018 OCTT script could be changed to make these methods mandatory only for AES70-2018 devices. I don't know if OCTT scripts support that kind of conditional testing. If we did make such a modification for the two property-change subscription methods, we should make it for OcaSubscriptionManager.GetmaximumSubscriberContextLength(), which was also new in 2018.

/Jeff

[tljdebrouwer]

Opened issues #9 #10 #11 #12 to get above issues solved.