Mesh network export has data loss problem

KcLeung514 commented 3 years ago

Describe the bug The "App key binding data" of last provisioned node is missing in the exported JSON file. Import this mesh network data to the iOS nrf mesh library, and send mesh message to the model, the library will throw AccessError.modelNotBoundToAppKey error.

To Reproduce Steps to reproduce the behavior:

Provision one node to the mesh network.
Bind app key to its vendor model (06170000). ( Send "ConfigModelAppBind" ) ( Ref: Log-01 )
Export current mesh network to JSON file. ( Ref: MeshNetwork-FirstExport.json )

Review the exported JSON file, and check the "App key binding data" of first provisioned node. The "App key binding data" is empty now. ( Path=Root > nodes > elements > models > bind )

{
  "UUID": "17067101-0000-002A-3098-90296C020000",
  ...
  "elements": [
    {
      "name": "Element: 0x0002",
      "index": 0,
      "location": "0000",
      "models": [
        ...
        {
          "modelId": "06170000",
          "bind": [],
          "subscribe": []
        }
      ]
    }
  ],
  "excluded": false
}

Provision second node to the mesh network.
Bind app key to its vendor model (06170000). ( Send "ConfigModelAppBind" ) ( See Log-02 )
Export current mesh network to JSON file. ( Ref: MeshNetwork-SecondExport.json )

Review the exported JSON file, and check the "App key binding data" of first provisioned node. The "App key binding data" is filled. ( Path=Root > nodes > elements > models > bind )

{
  "UUID": "17067101-0000-002A-3098-90296C020000",
  ...
  "elements": [
    {
      "name": "Element: 0x0002",
      "index": 0,
      "location": "0000",
      "models": [
        ...
        {
          "modelId": "06170000",
          "bind": [
            0
          ],
          "subscribe": []
        }
      ]
    }
  ],
  "excluded": false
},

Review the exported JSON file, and check the "App key binding data" of second provisioned node. The "App key binding data" is empty now. ( Path=Root > nodes > elements > models > bind )

{
  "UUID": "17067101-0000-0028-7832-50BBA8020000",
  ...
  "elements": [
    {
      "name": "Element: 0x0003",
      "index": 0,
      "location": "0000",
      "models": [
        ...
        {
          "modelId": "06170000",
          "bind": [],
          "subscribe": []
        }
      ]
    }
  ],
  "excluded": false
}

Expected behavior The "App key binding data" of nodes should not loss in the exported JSON file.

Platform details:

Device: Pixel 4a
OS: Android 11
Library Version: v3.1.6

Logs / Screenshots Log-01 - App client send "ConfigModelAppBind" to the first provisioned node, and receive response from that node.

V/MeshTransport(31110): Src address: 0001
V/MeshTransport(31110): Dst address: 0002
V/MeshTransport(31110): Key: 5424903DD9397E6355B6ACE27172E95B
V/MeshTransport(31110): akf: 0
V/MeshTransport(31110): aid: 0
V/MeshTransport(31110): aszmic: 0
V/MeshTransport(31110): Sequence number: 8
V/MeshTransport(31110): Access message opcode: 803d
V/MeshTransport(31110): Access message parameters: 0200000017060000
V/AccessLayer(31110): Created Access PDU 803D0200000017060000
V/UpperTransportLayer(31110): Device nonce: 02000000080001000200000000
V/UpperTransportLayer(31110): Encrypted upper transport pdu: 7C8C8AAA7B4511CF59BD7C2B484F
V/LowerTransportLayer(31110): Segmented Lower transport access PDU: 800020017C8C8AAA7B4511CF59BD7C2B 0 of 2
V/LowerTransportLayer(31110): Segmented Lower transport access PDU: 80002021484F 1 of 2
V/NetworkLayer(31110): Encryption key: B9E576B17E0B769B18D386946D87759F
V/NetworkLayer(31110): Privacy key: E2F004A6655029F47AB71168847189D6
V/NetworkLayer(31110): Sequence Number: 000008
V/NetworkLayer(31110): Encrypted Network payload: 1666B7ECB7C4BB52B4AA151EA3A9557B363DB10CDA03
V/NetworkLayer(31110): Sequence Number: 000009
V/NetworkLayer(31110): Encrypted Network payload: 27CF52654BA454E2370DA0F2
V/MeshManagerApi(31110): MeshNetwork pdu sent: 0x0053067B8D6A58CC27CF52654BA454E2370DA0F2
V/MeshManagerApi(31110): Received network pdu: 0x0053C78774E5782518FB0F9516F4F9075E3F5A907DF18BF7D3
V/BaseMeshMessageHandler(31110): TTL for received message: 7
V/BaseMeshMessageHandler(31110): Sequence number of received access message: 7
V/NetworkLayer(31110): TTL for received message: 7
V/DefaultNoOperationMessageState(31110): Acknowledgement payload: 00000003
V/BlockAcknowledgementMessage(31110): Segment 0 of 1 received by peer
V/BlockAcknowledgementMessage(31110): Segment 1 of 1 received by peer
V/MeshManagerApi(31110): Received network pdu: 0x0053ABC7D9817B5A07D9EBC544C794056BF3B4B6E1498E3E3162AF55A637
V/BaseMeshMessageHandler(31110): TTL for received message: 7
V/BaseMeshMessageHandler(31110): Sequence number of received access message: 8
V/NetworkLayer(31110): TTL for received message: 7
V/NetworkLayer(31110): Dst: 0x0001
D/LowerTransportLayer(31110): IV Index of received message: 0
D/LowerTransportLayer(31110): SeqAuth: 8
V/AccessLayer(31110): Received Access PDU 803E000200000017060000
V/ConfigModelAppStatus(31110): Status code: 0
V/ConfigModelAppStatus(31110): Status message: Success
V/ConfigModelAppStatus(31110): Element address: 0002
V/ConfigModelAppStatus(31110): App key index: 0000
V/ConfigModelAppStatus(31110): Model identifier: 6170000

Log-02 - App client send "ConfigModelAppBind" to the second provisioned node, and receive response from that node.

V/MeshTransport(31110): Src address: 0001
V/MeshTransport(31110): Dst address: 0003
V/MeshTransport(31110): Key: A246D36B9041D3A20F56BD786C940D58
V/MeshTransport(31110): akf: 0
V/MeshTransport(31110): aid: 0
V/MeshTransport(31110): aszmic: 0
V/MeshTransport(31110): Sequence number: 17
V/MeshTransport(31110): Access message opcode: 803d
V/MeshTransport(31110): Access message parameters: 0300000017060000
V/AccessLayer(31110): Created Access PDU 803D0300000017060000
V/UpperTransportLayer(31110): Device nonce: 02000000110001000300000000
V/UpperTransportLayer(31110): Encrypted upper transport pdu: 0E5DDA03810AD510B1B4FD40223B
V/LowerTransportLayer(31110): Segmented Lower transport access PDU: 800044010E5DDA03810AD510B1B4FD40 0 of 2
V/LowerTransportLayer(31110): Segmented Lower transport access PDU: 80004421223B 1 of 2
V/NetworkLayer(31110): Encryption key: B9E576B17E0B769B18D386946D87759F
V/NetworkLayer(31110): Privacy key: E2F004A6655029F47AB71168847189D6
V/NetworkLayer(31110): Sequence Number: 000011
V/NetworkLayer(31110): Encrypted Network payload: 1472FCB6F1CA8F8130ED67EC273CCC405D2B6174439B
V/NetworkLayer(31110): Sequence Number: 000012
V/NetworkLayer(31110): Encrypted Network payload: 6DF12D8196C6197A91FC5DB3
V/MeshManagerApi(31110): MeshNetwork pdu sent: 0x00536DAA3A5C6FAA6DF12D8196C6197A91FC5DB3
V/MeshManagerApi(31110): Received network pdu: 0x0053152E6C51D170BA7952EAB516157A171775C87C095CBD16
V/BaseMeshMessageHandler(31110): TTL for received message: 7
V/BaseMeshMessageHandler(31110): Sequence number of received access message: 7
V/NetworkLayer(31110): TTL for received message: 7
V/DefaultNoOperationMessageState(31110): Acknowledgement payload: 00000003
V/BlockAcknowledgementMessage(31110): Segment 0 of 1 received by peer
V/BlockAcknowledgementMessage(31110): Segment 1 of 1 received by peer
V/MeshManagerApi(31110): Received network pdu: 0x0053FD6356B702E8B6A72BD5B4C798F718BF37A85B6BAC470B19BBD22114
V/BaseMeshMessageHandler(31110): TTL for received message: 7
V/BaseMeshMessageHandler(31110): Sequence number of received access message: 8
V/NetworkLayer(31110): TTL for received message: 7
V/NetworkLayer(31110): Dst: 0x0001
D/LowerTransportLayer(31110): IV Index of received message: 0
D/LowerTransportLayer(31110): SeqAuth: 8
V/AccessLayer(31110): Received Access PDU 803E000300000017060000
V/ConfigModelAppStatus(31110): Status code: 0
V/ConfigModelAppStatus(31110): Status message: Success
V/ConfigModelAppStatus(31110): Element address: 0003
V/ConfigModelAppStatus(31110): App key index: 0000
V/ConfigModelAppStatus(31110): Model identifier: 6170000

MeshNetwork-FirstExport.json

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://www.bluetooth.com/specifications/assigned-numbers/mesh-profile/cdb-schema.json#",
  "version": "1.0.0",
  "meshUUID": "E05AFE3B-E9C8-4ABA-B6B5-31EADB1B0006",
  "meshName": "nRF Mesh Network",
  "timestamp": "2021-08-12T13:52:53+08:00",
  "partial": false,
  "netKeys": [
    {
      "name": "Network Key 1",
      "index": 0,
      "key": "799F6C4B3801A652F485021174614CE1",
      "phase": 0,
      "minSecurity": "insecure",
      "timestamp": "2021-08-12T13:51:58+08:00"
    }
  ],
  "appKeys": [
    {
      "name": "Application Key 1",
      "index": 0,
      "boundNetKey": 0,
      "key": "3C75A88FAB4CC2FEED5A5F95BFEC3B5E"
    },
    {
      "name": "Application Key 2",
      "index": 1,
      "boundNetKey": 0,
      "key": "C05C632052B07C9D6CC748A1BC919EF7"
    },
    {
      "name": "Application Key 3",
      "index": 2,
      "boundNetKey": 0,
      "key": "90DDD253D08F63BC1660075A6494FC28"
    }
  ],
  "provisioners": [
    {
      "provisionerName": "nRF Mesh Provisioner",
      "UUID": "6DD1CF43-C9F4-4C38-AB9C-D8C7B269ED60",
      "allocatedUnicastRange": [
        {
          "lowAddress": "0001",
          "highAddress": "199A"
        }
      ],
      "allocatedGroupRange": [
        {
          "lowAddress": "C000",
          "highAddress": "CC9A"
        }
      ],
      "allocatedSceneRange": [
        {
          "firstScene": "0001",
          "lastScene": "3333"
        }
      ]
    }
  ],
  "nodes": [
    {
      "UUID": "6DD1CF43-C9F4-4C38-AB9C-D8C7B269ED60",
      "name": "nRF Mesh Provisioner",
      "deviceKey": "FCBA0DB25970FD97357C0AC5ABE24E5E",
      "unicastAddress": "0001",
      "security": "insecure",
      "configComplete": true,
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 2,
        "relay": 2
      },
      "defaultTTL": 5,
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        },
        {
          "index": 1,
          "updated": false
        },
        {
          "index": 2,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0001",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0001",
              "bind": [],
              "subscribe": []
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "17067101-0000-002A-3098-90296C020000",
      "name": "My Node",
      "deviceKey": "5424903DD9397E6355B6ACE27172E95B",
      "unicastAddress": "0002",
      "security": "insecure",
      "configComplete": false,
      "cid": "0617",
      "pid": "0000",
      "vid": "0000",
      "crpl": "000A",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 7,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0002",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "06170000",
              "bind": [],
              "subscribe": []
            }
          ]
        }
      ],
      "excluded": false
    }
  ],
  "groups": [],
  "scenes": [],
  "networkExclusions": []
}

MeshNetwork-SecondExport.json

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://www.bluetooth.com/specifications/assigned-numbers/mesh-profile/cdb-schema.json#",
  "version": "1.0.0",
  "meshUUID": "E05AFE3B-E9C8-4ABA-B6B5-31EADB1B0006",
  "meshName": "nRF Mesh Network",
  "timestamp": "2021-08-12T13:55:47+08:00",
  "partial": false,
  "netKeys": [
    {
      "name": "Network Key 1",
      "index": 0,
      "key": "799F6C4B3801A652F485021174614CE1",
      "phase": 0,
      "minSecurity": "insecure",
      "timestamp": "2021-08-12T13:51:58+08:00"
    }
  ],
  "appKeys": [
    {
      "name": "Application Key 1",
      "index": 0,
      "boundNetKey": 0,
      "key": "3C75A88FAB4CC2FEED5A5F95BFEC3B5E"
    },
    {
      "name": "Application Key 2",
      "index": 1,
      "boundNetKey": 0,
      "key": "C05C632052B07C9D6CC748A1BC919EF7"
    },
    {
      "name": "Application Key 3",
      "index": 2,
      "boundNetKey": 0,
      "key": "90DDD253D08F63BC1660075A6494FC28"
    }
  ],
  "provisioners": [
    {
      "provisionerName": "nRF Mesh Provisioner",
      "UUID": "6DD1CF43-C9F4-4C38-AB9C-D8C7B269ED60",
      "allocatedUnicastRange": [
        {
          "lowAddress": "0001",
          "highAddress": "199A"
        }
      ],
      "allocatedGroupRange": [
        {
          "lowAddress": "C000",
          "highAddress": "CC9A"
        }
      ],
      "allocatedSceneRange": [
        {
          "firstScene": "0001",
          "lastScene": "3333"
        }
      ]
    }
  ],
  "nodes": [
    {
      "UUID": "6DD1CF43-C9F4-4C38-AB9C-D8C7B269ED60",
      "name": "nRF Mesh Provisioner",
      "deviceKey": "FCBA0DB25970FD97357C0AC5ABE24E5E",
      "unicastAddress": "0001",
      "security": "insecure",
      "configComplete": true,
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 2,
        "relay": 2
      },
      "defaultTTL": 5,
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        },
        {
          "index": 1,
          "updated": false
        },
        {
          "index": 2,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0001",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0001",
              "bind": [],
              "subscribe": []
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "17067101-0000-002A-3098-90296C020000",
      "name": "My Node",
      "deviceKey": "5424903DD9397E6355B6ACE27172E95B",
      "unicastAddress": "0002",
      "security": "insecure",
      "configComplete": false,
      "cid": "0617",
      "pid": "0000",
      "vid": "0000",
      "crpl": "000A",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 7,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0002",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "06170000",
              "bind": [
                0
              ],
              "subscribe": []
            }
          ]
        }
      ],
      "excluded": false
    },
    {
      "UUID": "17067101-0000-0028-7832-50BBA8020000",
      "name": "My Node",
      "deviceKey": "A246D36B9041D3A20F56BD786C940D58",
      "unicastAddress": "0003",
      "security": "insecure",
      "configComplete": false,
      "cid": "0617",
      "pid": "0000",
      "vid": "0000",
      "crpl": "000A",
      "features": {
        "friend": 2,
        "lowPower": 2,
        "proxy": 1,
        "relay": 1
      },
      "defaultTTL": 7,
      "networkTransmit": {
        "count": 2,
        "interval": 20
      },
      "netKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "appKeys": [
        {
          "index": 0,
          "updated": false
        }
      ],
      "elements": [
        {
          "name": "Element: 0x0003",
          "index": 0,
          "location": "0000",
          "models": [
            {
              "modelId": "0000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "1000",
              "bind": [],
              "subscribe": []
            },
            {
              "modelId": "06170000",
              "bind": [],
              "subscribe": []
            }
          ]
        }
      ],
      "excluded": false
    }
  ],
  "groups": [],
  "scenes": [],
  "networkExclusions": []
}

roshanrajaratnam commented 3 years ago

Hi, I am unable to reproduce this issue? is this something you see happening intermittently?

roshanrajaratnam commented 2 years ago

Closing due to inactivity. Feel free to open if the problem persists.

KcLeung514 commented 2 years ago

Finally, I found out that the problem was caused by my usages. Sorry for that.

I created 2 BleManager instances. In order to handle the callback, they have their own MeshManagerApi instances. Let's say "meshManagerApi1" and "MeshManagerApi2"

Here is the scene.

meshManagerApi1.startProvisioning(meshNode)
// After provisioning done, align the `MeshNetwork` between "meshManagerApi1" and "MeshManagerApi2".
MeshManagerApi2.loadMeshNetwork()
// Export the mesh network data
MeshManagerApi2.exportMeshNetwork()

And the "App key binding data" is not included in "MeshManagerApi2" 's mesh network data.

========== @roshanrajaratnam

Is that possible to let 2 MeshManagerApi instances share the same MeshNetwork instance?

I tried something like below.

String networkJson = "..."
meshManagerApi1.importMeshNetworkJson(networkJson);
meshManagerApi2.loadMeshNetwork();

The meshManagerApi1 and meshManagerApi2 can have the same mesh network data, but mesh network data not aligned after provisioning.

roshanrajaratnam commented 2 years ago

Hi @KcLeung514 that's fine. It's an easy mistake to make.

To answer your question you have to export the network and import it from the other mesh manager instance. How do you manage the callbacks with multiple instances of the mesh network? I hope you are validating against the mesh network UUID?

KcLeung514 commented 2 years ago

To answer your question you have to export the network and import it from the other mesh manager instance.

I think this solution can let "meshManagerApi1" and "MeshManagerApi2" have the same mesh network data. But I think I need to do this again when one of the "meshManagerApi" made any changes in mesh network. MeshManagerApi#importMeshNetworkJson involves write DB operations, will it write the same data to the DB again?

How do you manage the callbacks with multiple instances of the mesh network?

I have 2 MeshManagerApi instances, so that I can handle the callback separately.
I used this 2 MeshManagerApi manage the same network data.
Any MeshManagerApi make changes in the mesh network, the data will be saved under the same mesh UUID.
I managed these MeshManagerApi, avoid make changes in the mesh network at the same time.
I trying to align the network data when any MeshManagerApi make changes in the mesh network.

@roshanrajaratnam

Please correct me if I doing something silly...

====== Assume that we have a method "MeshManagerApi#setMeshNetwork". If we align network data this way.

meshManagerApi2.setMeshNetwork(meshManagerApi1.meshNetwork)

Is that a workaround?

roshanrajaratnam commented 2 years ago

Well both instances will be using the same database in my opinion and mesh network UUID is used as the primary key in the database so theoretically this should work. Although I have not tested this thoroughly.

First why do you use 2 separate instances to manage the same network data?

Is that a workaround?

The correct way would be to import it as a json, this will validate the network before importing it.

KcLeung514 commented 2 years ago

First why do you use 2 separate instances to manage the same network data?

Because I want to manage to 2 bluetooth connection in one mesh network. One for provisioning device, One for send control message to other devices.

I have no idea how to use one MeshManagerApi instance to manage 2 bluetooth connections.

When I use MeshManagerApi#createMeshPdu, this method will return several responses through MeshManagerCallbacks#onMeshPduCreated. If the application needs to send different message over these connections at the same time, then I will need to call MeshManagerApi#createMeshPdu 2 times. When I receive several responses from MeshManagerCallbacks#onMeshPduCreated, I don't know which connection I should use to send data..

On BleManagerGattCallback#isRequiredServiceSupported, how to know which BluetoothDevice is currently connecting?

roshanrajaratnam commented 2 years ago

I have no idea how to use one MeshManagerApi instance to manage 2 bluetooth connections.

I don't see the need to use two instances of the mesh manager api. Here's a suggestion.

Establish two connections, one to a provisioned device 'A' and an unprovisioned node 'B'
Provision device 'B' and don't reconnect after provisionining.
Continue configuration over the existing connection with device 'A'

When I use MeshManagerApi#createMeshPdu, this method will return several responses through MeshManagerCallbacks#onMeshPduCreated. If the application needs to send different message over these connections at the same time, then I will need to call MeshManagerApi#createMeshPdu 2 times. When I receive several responses from MeshManagerCallbacks#onMeshPduCreated, I don't know which connection I should use to send data..

On BleManagerGattCallback#isRequiredServiceSupported, how to know which BluetoothDevice is currently connecting?

I understand your issue, ususally there is no principle of BluetoothDevice or it's MacAddress in a mesh network eventhough there is a GATT connection. Ideally we should not depend on the MacAddress because the address can change after a reboot. In addition, you are not communicating directly with a device. You are expecting the proxy to propagate your message based on the Unicast/Group/Label address

NordicSemiconductor / Android-nRF-Mesh-Library

Mesh network export has data loss problem #444