search

NabuCasa / silabs-firmware

Silicon Labs firmware for Yellow and SkyConnect
107 stars 2 forks source link

Optimized EmberZNet configuration #1

Closed agners closed 2 years ago

agners commented 2 years ago

For 6.9.x.x/6.10.x.x far I've used the following non-standard customizations:

Zigbee PRO stack macro Value
EMBER_APS_UNICAST_MESSAGE_COUNT 32
EMBER_PACKET_BUFFER_COUNT 128
EMBER_NEIGHBOR_TABLE_SIZE 26
EMBER_ADDRESS_TABLE_SIZE 32
EMBER_SOURCE_ROUTE_TABLE_SIZE 200

With EmberZNet 7.0.0.0, the following values have been chosen:

Zigbee PRO stack macro Default Value
EMBER_APS_UNICAST_MESSAGE_COUNT 10 20
EMBER_MAX_END_DEVICE_CHILDREN 6 32
EMBER_PACKET_BUFFER_COUNT 75 128 (set to 255 in 7.0.1.0)
EMBER_NEIGHBOR_TABLE_SIZE 16 26
EMBER_ADDRESS_TABLE_SIZE 2 16 (via project config)
EMBER_MULTICAST_TABLE_SIZE 8 16 (via project config)
EMBER_SOURCE_ROUTE_TABLE_SIZE 7 200
EMBER_BINDING_TABLE_SIZE⍀ 3 3 (32 in 7.0.1.0, new default)

⍀ not used by zigpy currently

While zigpy tries to maximize EMBER_PACKET_BUFFER_COUNT, the value set in firmware seems to matter, e.g. when configured with 75, I get this output:

2022-01-18 13:15:33 DEBUG (MainThread) [bellows.ezsp.protocol] Send command setConfigurationValue: (<EzspConfigId.CONFIG_PACKET_BUFFER_COUNT: 1>, 255)
...
2022-01-18 13:15:33 DEBUG (MainThread) [bellows.ezsp.protocol] Free/configured buffers after all memory allocation: 74/75
OLD: The new default configuration for `EMBER_ADDRESS_TABLE_SIZE` seems to be just `2`! By default, zigpy seems to try setting it to 16, which fails: ``` 2022-01-18 14:08:59 DEBUG (MainThread) [bellows.ezsp.protocol] Send command setConfigurationValue: (, 16) 2022-01-18 14:08:59 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 83 (setConfigurationValue) received: b'35' 2022-01-18 14:08:59 WARNING (MainThread) [bellows.ezsp.protocol] Couldn't set EzspConfigId.CONFIG_ADDRESS_TABLE_SIZE=16 configuration value: EzspStatus.ERROR_OUT_OF_MEMORY ``` Similarly the default for `CONFIG_MULTICAST_TABLE_SIZE` is now `8`, and zigpy tries to set `16`. ``` 2022-01-18 14:08:59 DEBUG (MainThread) [bellows.ezsp.protocol] Send command setConfigurationValue: (, 16) 2022-01-18 14:08:59 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 83 (setConfigurationValue) received: b'35' 2022-01-18 14:08:59 WARNING (MainThread) [bellows.ezsp.protocol] Couldn't set EzspConfigId.CONFIG_ADDRESS_TABLE_SIZE=16 configuration value: EzspStatus.ERROR_OUT_OF_MEMORY ``` The note "not configureable" means it is no longer configurable through the UI configurator or the project specfic `config/sl_zigbee_pro_stack_config.h`. Instead the values seem to be defined as part of the SDK (it likely could be added manually to the configuration header file still).
agners commented 2 years ago

The complete config as of now

$ bellows -d /dev/ttyUSB2 --baudrate 115200 config --all
NOTE: Configuration changes do not persist across resets
CONFIG_PACKET_BUFFER_COUNT=128
CONFIG_NEIGHBOR_TABLE_SIZE=26
CONFIG_APS_UNICAST_MESSAGE_COUNT=32
CONFIG_BINDING_TABLE_SIZE=3
CONFIG_ADDRESS_TABLE_SIZE=8
CONFIG_MULTICAST_TABLE_SIZE=8
CONFIG_ROUTE_TABLE_SIZE=16
CONFIG_DISCOVERY_TABLE_SIZE=8
CONFIG_STACK_PROFILE=0
CONFIG_SECURITY_LEVEL=5
CONFIG_MAX_HOPS=30
CONFIG_MAX_END_DEVICE_CHILDREN=32
CONFIG_INDIRECT_TRANSMISSION_TIMEOUT=3000
CONFIG_END_DEVICE_POLL_TIMEOUT=8
CONFIG_TX_POWER_MODE=0
CONFIG_DISABLE_RELAY=0
CONFIG_TRUST_CENTER_ADDRESS_CACHE_SIZE=0
CONFIG_SOURCE_ROUTE_TABLE_SIZE=200
CONFIG_FRAGMENT_WINDOW_SIZE=1
CONFIG_FRAGMENT_DELAY_MS=0
CONFIG_KEY_TABLE_SIZE=12
CONFIG_APS_ACK_TIMEOUT=1600
CONFIG_ACTIVE_SCAN_DURATION=3
CONFIG_END_DEVICE_BIND_TIMEOUT=60
CONFIG_PAN_ID_CONFLICT_REPORT_THRESHOLD=2
CONFIG_REQUEST_KEY_TIMEOUT=0
CONFIG_APPLICATION_ZDO_FLAGS=0
CONFIG_BROADCAST_TABLE_SIZE=15
CONFIG_MAC_FILTER_TABLE_SIZE=2
CONFIG_SUPPORTED_NETWORKS=1
CONFIG_SEND_MULTICASTS_TO_SLEEPY_ADDRESS=0
CONFIG_ZLL_GROUP_ADDRESSES=0
CONFIG_ZLL_RSSI_THRESHOLD=216
CONFIG_MTORR_FLOW_CONTROL=1
CONFIG_RETRY_QUEUE_SIZE=16
CONFIG_NEW_BROADCAST_ENTRY_THRESHOLD=9
CONFIG_TRANSIENT_KEY_TIMEOUT_S=300
CONFIG_BROADCAST_MIN_ACKS_NEEDED=255
CONFIG_TC_REJOINS_USING_WELL_KNOWN_KEY_TIMEOUT_S=300
CONFIG_CTUNE_VALUE=127
CONFIG_ASSUME_TC_CONCENTRATOR_TYPE=1
Adminiuga commented 2 years ago

Regarding buffers on 6.10.3.0 firmware it was

2022-01-18 19:26:46 DEBUG (MainThread) [bellows.ezsp.protocol] Send command getConfigurationValue: (
<EzspConfigId.CONFIG_PACKET_BUFFER_COUNT: 1>,)
2022-01-18 19:26:46 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 82 (getConfiguratio
nValue) received: b'008000'
2022-01-18 19:26:46 DEBUG (MainThread) [bellows.ezsp.protocol] Free/configured buffers before any co
nfigurations: 126/128
...
2022-01-18 19:26:46 DEBUG (MainThread) [bellows.ezsp.protocol] Send command getConfigurationValue: (
<EzspConfigId.CONFIG_PACKET_BUFFER_COUNT: 1>,)
2022-01-18 19:26:47 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 82 (getConfiguratio
nValue) received: b'009200'
2022-01-18 19:26:47 DEBUG (MainThread) [bellows.ezsp.protocol] Free/configured buffers after all mem
ory allocation: 144/146

and running it is around those values, between 134 and 146

PTA_LO_PRI_TX_ABORTED = 0, PTA_HI_PRI_TX_ABORTED = 0, ADDRESS_CONFLICT_SENT = 0, EZSP_FREE_BUFFERS = 141

But can confirm your behavior with 7.0.0.0 firmware, EZSP free buffers hovers around 126

_HI_PRI_DENIED = 0, PTA_LO_PRI_TX_ABORTED = 0, PTA_HI_PRI_TX_ABORTED = 0, ADDRESS_CONFLICT_SENT = 0, EZSP_FREE_BUFFERS = 126]
Adminiuga commented 2 years ago

Similarly the default for CONFIG_MULTICAST_TABLE_SIZE is now 8, and zigpy tries to set 16

Bummer. That defines how many groups the NCP can be member of. Unlike deConz and ZNP stack (patched one) EmberZNet filters group messages and to receive any group messages, NCP has to be subscribed to those group.

In the past some of Ikea devices were creating random groups and would send messages only to those group IDs and without adding coordinator to those groups it was impossible getting any messages from those ikea devices.

Adminiuga commented 2 years ago

Check the EmberZNet Release Notes page 6:

NCP Memory Allocation The ability to increase NCP memory allocation sizes at run-time with EZSP commands from the host was incompatible with the SLC architecture and has been removed. The sizes may still be decreased at run-time by host in order to adjust the configuration. When configuring the NCP application all settings must be set to the max that the host will require. The affected configuration items are:

So looks like from now on, at runtime you can only decrease tables/items sizes. So if firmware was compiled with address table size of 8 and bellows tries setting it to 16, then we get OUT_OF_MEMORY status code.

agners commented 2 years ago

So looks like from now on, at runtime you can only decrease tables/items sizes. So if firmware was compiled with address table size of 8 and bellows tries setting it to 16, then we get OUT_OF_MEMORY status code.

Interesting find!

The ability to increase NCP memory allocation sizes at run-time with EZSP commands from the host was incompatible with the SLC architecture and has been removed.

Do you know what SLC refers to exactly?

agners commented 2 years ago

The full configuration when running 6.10.3.0

$ bellows -d /dev/ttyUSB0 --baudrate 115200 config --all
NOTE: Configuration changes do not persist across resets
CONFIG_PACKET_BUFFER_COUNT=128
CONFIG_NEIGHBOR_TABLE_SIZE=26
CONFIG_APS_UNICAST_MESSAGE_COUNT=32
CONFIG_BINDING_TABLE_SIZE=32
CONFIG_ADDRESS_TABLE_SIZE=32
CONFIG_MULTICAST_TABLE_SIZE=8
CONFIG_ROUTE_TABLE_SIZE=16
CONFIG_DISCOVERY_TABLE_SIZE=8
CONFIG_STACK_PROFILE=0
CONFIG_SECURITY_LEVEL=5
CONFIG_MAX_HOPS=30
CONFIG_MAX_END_DEVICE_CHILDREN=32
CONFIG_INDIRECT_TRANSMISSION_TIMEOUT=3000
CONFIG_END_DEVICE_POLL_TIMEOUT=8
CONFIG_TX_POWER_MODE=0
CONFIG_DISABLE_RELAY=0
CONFIG_TRUST_CENTER_ADDRESS_CACHE_SIZE=0
CONFIG_SOURCE_ROUTE_TABLE_SIZE=200
CONFIG_FRAGMENT_WINDOW_SIZE=1
CONFIG_FRAGMENT_DELAY_MS=0
CONFIG_KEY_TABLE_SIZE=12
CONFIG_APS_ACK_TIMEOUT=1600
CONFIG_ACTIVE_SCAN_DURATION=3
CONFIG_END_DEVICE_BIND_TIMEOUT=60
CONFIG_PAN_ID_CONFLICT_REPORT_THRESHOLD=2
CONFIG_REQUEST_KEY_TIMEOUT=0
CONFIG_CERTIFICATE_TABLE_SIZE=0
CONFIG_APPLICATION_ZDO_FLAGS=0
CONFIG_BROADCAST_TABLE_SIZE=15
CONFIG_MAC_FILTER_TABLE_SIZE=0
CONFIG_SUPPORTED_NETWORKS=1
CONFIG_SEND_MULTICASTS_TO_SLEEPY_ADDRESS=0
CONFIG_ZLL_GROUP_ADDRESSES=1
CONFIG_ZLL_RSSI_THRESHOLD=128
CONFIG_MTORR_FLOW_CONTROL=1
CONFIG_RETRY_QUEUE_SIZE=16
CONFIG_NEW_BROADCAST_ENTRY_THRESHOLD=9
CONFIG_TRANSIENT_KEY_TIMEOUT_S=300
CONFIG_BROADCAST_MIN_ACKS_NEEDED=255
CONFIG_TC_REJOINS_USING_WELL_KNOWN_KEY_TIMEOUT_S=300
CONFIG_CTUNE_VALUE=133
CONFIG_ASSUME_TC_CONCENTRATOR_TYPE=1

Note that CONFIG_MULTICAST_TABLE_SIZE was set to 8 there too, but it seems that just like with the CONFIG_PACKET_BUFFER_COUNT setting it was possible to increase that value.

agners commented 2 years ago

Also found this hint in the release notes:

Items marked "Project Configuration" are set to defaults in ember-configuration-defaults.h. These can be overridden by adding a #define to the project configuration.

And CONFIG_MULTICAST_TABLE_SIZE as well as CONFIG_ADDRESS_TABLE_SIZE is marked as "Project Configuration". I am not sure what they exactly mean by "Project Configuration". It sounds like a header file, but they all seem generated to me. I set it now in the Project Properties under C/C++ General -> Paths and Symbols in Symbols -> GNU C. Feels like overkill but works :man_shrugging:

Will push 7.0.0.0 firmwares with increased CONFIG_MULTICAST_TABLE_SIZE and CONFIG_ADDRESS_TABLE_SIZE values.

Adminiuga commented 2 years ago

Do you know what SLC refers to exactly?

Silabs configurator? I have no ideas, but in new SDK everything is prefixed with sl_

I am not sure what they exactly mean by "Project Configuration" Project Properties under C/C++ General -> Paths and Symbols in Symbols -> GNU C. Feels like overkill but works 🤷‍♂️

Yes, that one.

In the project configuration you can set compiler/linker options, defines, includes and a bunch of other things. You should be able to access it via a right click on the project root in the left pane or through the menus in the top of the window edit: need more coffe

Adminiuga commented 2 years ago

BTW, the project default is to use 255 ezsp buffers. Which is that special value to allocate the remainder of the memory. Will try running nthe system with it and see how many actual buffers it allows, i hope that is still accessible via counters

agners commented 2 years ago

At least in Gecko SDK 4.0.0 it would not let me set it to 255 (says its an invalid value), but I set it to it in the header files. In a quick test it seems to work fine here.

2022-02-04 18:21:47 DEBUG (MainThread) [bellows.ezsp.protocol] Send command getConfigurationValue: (<EzspConfigId.CONFIG_PACKET_BUFFER_COUNT: 1>,)
2022-02-04 18:21:47 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 82 (getConfigurationValue) received: b'00ff00'
2022-02-04 18:21:47 DEBUG (MainThread) [bellows.ezsp.protocol] Free/configured buffers before all memory allocation: 254/255
2022-02-04 18:21:47 DEBUG (MainThread) [bellows.ezsp.protocol] Send command setConfigurationValue: (<EzspConfigId.CONFIG_PACKET_BUFFER_COUNT: 1>, 255)
2022-02-04 18:21:47 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 83 (setConfigurationValue) received: b'00'
2022-02-04 18:21:47 DEBUG (MainThread) [bellows.ezsp.protocol] Send command getValue: (<EzspValueId.VALUE_FREE_BUFFERS: 3>,)
2022-02-04 18:21:47 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 170 (getValue) received: b'0001fe'
2022-02-04 18:21:47 DEBUG (MainThread) [bellows.ezsp.protocol] Send command getConfigurationValue: (<EzspConfigId.CONFIG_PACKET_BUFFER_COUNT: 1>,)
2022-02-04 18:21:47 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 82 (getConfigurationValue) received: b'00ff00'
2022-02-04 18:21:47 DEBUG (MainThread) [bellows.ezsp.protocol] Free/configured buffers after all memory allocation: 254/255

Compiled now 7.0.1.0 firmware with the following configurations:

$ bellows -d /dev/ttyUSB1 -b 115200 config --all
NOTE: Configuration changes do not persist across resets
CONFIG_PACKET_BUFFER_COUNT=255
CONFIG_NEIGHBOR_TABLE_SIZE=26
CONFIG_APS_UNICAST_MESSAGE_COUNT=20
CONFIG_BINDING_TABLE_SIZE=32
CONFIG_ADDRESS_TABLE_SIZE=16
CONFIG_MULTICAST_TABLE_SIZE=16
CONFIG_ROUTE_TABLE_SIZE=16
CONFIG_DISCOVERY_TABLE_SIZE=8
CONFIG_STACK_PROFILE=0
CONFIG_SECURITY_LEVEL=5
CONFIG_MAX_HOPS=30
CONFIG_MAX_END_DEVICE_CHILDREN=32
CONFIG_INDIRECT_TRANSMISSION_TIMEOUT=3000
CONFIG_END_DEVICE_POLL_TIMEOUT=8
CONFIG_TX_POWER_MODE=0
CONFIG_DISABLE_RELAY=0
CONFIG_TRUST_CENTER_ADDRESS_CACHE_SIZE=0
CONFIG_SOURCE_ROUTE_TABLE_SIZE=200
CONFIG_FRAGMENT_WINDOW_SIZE=1
CONFIG_FRAGMENT_DELAY_MS=0
CONFIG_KEY_TABLE_SIZE=12
CONFIG_APS_ACK_TIMEOUT=1600
CONFIG_ACTIVE_SCAN_DURATION=3
CONFIG_END_DEVICE_BIND_TIMEOUT=60
CONFIG_PAN_ID_CONFLICT_REPORT_THRESHOLD=2
CONFIG_REQUEST_KEY_TIMEOUT=0
CONFIG_APPLICATION_ZDO_FLAGS=0
CONFIG_BROADCAST_TABLE_SIZE=15
CONFIG_MAC_FILTER_TABLE_SIZE=2
CONFIG_SUPPORTED_NETWORKS=1
CONFIG_SEND_MULTICASTS_TO_SLEEPY_ADDRESS=0
CONFIG_ZLL_GROUP_ADDRESSES=0
CONFIG_ZLL_RSSI_THRESHOLD=216
CONFIG_MTORR_FLOW_CONTROL=1
CONFIG_RETRY_QUEUE_SIZE=16
CONFIG_NEW_BROADCAST_ENTRY_THRESHOLD=9
CONFIG_TRANSIENT_KEY_TIMEOUT_S=300
CONFIG_BROADCAST_MIN_ACKS_NEEDED=255
CONFIG_TC_REJOINS_USING_WELL_KNOWN_KEY_TIMEOUT_S=300
CONFIG_CTUNE_VALUE=129
CONFIG_ASSUME_TC_CONCENTRATOR_TYPE=1
agners commented 2 years ago

With that configuration, I think this zha config customization make sense:

zha:
  zigpy_config:
    source_routing: true
    ezsp_config:
      CONFIG_MAX_END_DEVICE_CHILDREN: 32
      CONFIG_SOURCE_ROUTE_TABLE_SIZE: 200
      CONFIG_NEIGHBOR_TABLE_SIZE: 26

The rest of the configs is default or doesn't need adjustments from what I can tell.