project-chip / connectedhomeip

Matter (formerly Project CHIP) creates more connections between more objects, simplifying development for manufacturers and increasing compatibility for consumers, guided by the Connectivity Standards Alliance.
https://buildwithmatter.com
Apache License 2.0
7.45k stars 1.99k forks source link

Darwin tests expect no endpoints to ever be added to all-clusters-app #34374

Closed tcarmelveilleux closed 3 months ago

tcarmelveilleux commented 3 months ago

Darwin tests have many assumptions that there are endpoints 0, 1, 2. Adding a 3rd endpoint breaks a lot of tests.

Example:

2024-07-17T03:16:51.5527330Z /Users/runner/work/connectedhomeip/connectedhomeip/src/darwin/Framework/CHIPTests/MTRPerControllerStorageTests.m:2548: error: -[MTRPerControllerStorageTests testDataStorageUpdatesWhenRemovingEndpoints] : (([[controller.controllerDataStore _fetchEndpointIndexForNodeID:deviceID] isEqualToArray:testEndpoints]) is true) failed

which relates to:

    __block NSMutableDictionary<NSNumber *, NSArray<NSNumber *> *> * initialClusterIndex = [[NSMutableDictionary alloc] init];
    __block NSMutableArray<NSNumber *> * testEndpoints;

    delegate.onReportEnd = ^{
        XCTAssertNotNil(dataVersionForPartsList);
        XCTAssertNotNil(testClusterDataValueForPartsList);

        testEndpoints = [self getEndpointArrayFromPartsList:testDataForPartsList forDevice:device];

        // Make sure that the cluster data in the data storage is populated with cluster index and cluster data for endpoints 0, 1 and 2.
        // We do not need to check _persistedClusterData here. _persistedClusterData will be paged in from storage when needed so
        // just checking data storage should suffice here.
        dispatch_sync(self->_storageQueue, ^{

The test above (among others) expects exactly 3 endpoints enabled, and all of them to use NV storage.

This breaks the ability of anyone to change the composition of the all-clusters-app without cryptic failures related to low-level tests of the Darwin regression suite.

tcarmelveilleux commented 3 months ago

FYI @nivi-apple @bzbarsky-apple