Overcome Domoticz device limit per plugin

[Zorchz]

Even for about 50 physical devices contented to z2m can generate more than 255 logical devices in domoticz. But actuality not all of logical devices really required, is it possible to limit device creation by some filter something like filtered_attributes in z2m byt for devices. An other option can be divide devices between a few plugins.

[schurgan]

+1

[DatafoxChris]

+1

[stas-demydiuk]

How would you imagine this feature implemented?

[DatafoxChris]

+1

Domoticz uses device_ID (string/ 32bit hex) en een unit (255). So every device can have 255 units.

Why would there be a 255 devices limit, please explain

[stas-demydiuk]

Domoticz indexes plugin devices by unit and there the limit of 255 per plugin

[DatafoxChris]

You are using the Device ID, why wouldn't you use the Unit variable from 1 for every individual device ? This way you can use unlimited devices in your plugin.

DeviceID 0x00124b0002cc3bab Unit 1, 2 DeviceID 0x00124b0002cc3c34 Unit 1, 2, 3, 4 DeviceID 0x00124b00061e2607 Unit 1

[stas-demydiuk]

It is not about how I'd like to identify devices it is about plugin interface from domoticz. There is a global array Devices available for plugin which contain all devices associated with this plugin and indexed by unit. So Unit should be unique across plugin.

See https://www.domoticz.com/wiki/Developing_a_Python_plugin for additional information

[DatafoxChris]

You are referring to this array?

Devices[Unit].Update(nValue=nValue, sValue=str(sValue), SignalLevel=5, Image=8)

[DatafoxChris]

I am reading the F*** manual :)

Would this be an option to retrieve a device from domoticz ?

myDev = Domoticz.Device(DeviceID="0x00124b0002cc3bab", Unit=1)

[gizmocuz]

@stas-demydiuk , Firstly I am totally NOT an expert of the plugin system ;) But internal in Domoticz a 'device' consist of:

HardwareID, DeviceID, Unit, Type, Subtype

You can imagine that with this combination there can be many many devices created

From Chris I understand that plugins can only use a index from 1-255, I don't know why, but maybe something should be changed so each plugin can manage at least

DeviceID, Unit, Type, Subtype

To solve Chris his feature problem quickly, he might add the plugin twice in Domoticz and use the second plugin instance for it's lights (for example), so another 255 devices are possible

But maybe something needs to be changed in the core to allow control of 'DeviceID, Unit, Type, Subtype'

[DatafoxChris]

Having the plugin twice would mean i need a second setup of zigbee2mqtt library and a second zigbee usb-stick. This is not a big problem, but it would mean i will be running two separate zigbee networks. Switches and Motion Sensors on one network cannot Bind to lights on the other network. So it would be better to remove the 255 unit-limit (which is now used as the index of the Devices array) or for example use the internal Domoticz DeviceID as the index of this plugin device-array.

Even LidL is going to sell Zigbee devices soon, which i guess will be priced competitively :)

[schurgan]

I already spoke about this problem with 255 devices half a year ago. I read that it is better to be able to erase unnecessary devices and somehow block their appearances again, since they hang in the system to no avail.

[DatafoxChris]

This would be a workaround, not a real solution. If gizmocuz is able to add a second array for the plugins which is indexed by the Domoticz DeviceID, then the limitation of the unit (255) variable is solved.

[kovainfo]

+1

[gizmocuz]

@DatafoxChris , I am not the implementor of this behaviour... Maybe the original author can have a look as well. Otherwise, After the OAuth2 implementation i can have a look at this But i bet there are more experienced developers here that knows the inner workings of the plugin system and why it is designed like this

[gizmocuz]

@DatafoxChris , Why a second zigbee stick ? It is MQTT, you can connect to the broker as many times as you like

[DatafoxChris]

Because z2m delivers all devices it receives from the connected stick to mqtt using topic zigbee2mqtt which is then configured in the plugin in domoticz.

So in order to use two instances of the plugin i would not know of any other way than this setup: USBStick1 - zigbee2mqtt1 -> Broker (using topic zigbee2mqtt1) - plugin1 (using topic zigbee2mqtt1) SUSBtick2 - zigbee2mqtt2 -> Broker (using topic zigbee2mqtt2) - plugin2 (using topic zigbee2mqtt2)

[schurgan]

That’s how I solved this problem.

[stas-demydiuk]

@dnpwwo maybe you can help us with the topic

[dnpwwo]

When I made the framework I wanted to make it work the same way as the C++ supported hardware so used all the same underlying code.

When I wrote it Domoticz had a restriction where the Unit had to be 255 or less because in some of the old data structures it was only allocated a single byte (RFXCom?). Some common functions used these structures from memory.

The Plugin Framework itself would not care if the number of Units was increased but I believe Units above 255 would not work properly as the rest of the system currently stands. Essentially this is a Domoticz issue not a plugin framework one.

For example, if you look for the definition for:

• CSQLHelper::UpdateValue
• CNotificationHelper::CheckAndHandleNotification you will see Unit being passed as 'const unsigned char' which only allows 255. In fact a search for 'unsigned char unit' showed around 20 such functions.

If these functions were changed to work with 'short' then changing the framework would be easy.

[DatafoxChris]

As a quick solution i would opt to change unsigned char to short.

[gizmocuz]

@DatafoxChris , It is fine that everything is delivered to MQTT by the zigbee plugin. But you can still can connect multiple times and add specific hardware each time into the Domoticz system. It is not a good solution to change it to a short

A device is Unique by:

• Hardware_ID
• Device_ID
• Device_Type
• Device_SubType
• Unit

I think with the above 5 combinations you can have enough devices in Domoticz.

In the very beginning I opted for functions like:

Plugin.CreateTempDevice({HardwareID/self},DeviceID, Unit) Plugin.CreateTempHumBaroDevice(DeviceID, Unit) Plugin.CreateLightDevice(DeviceID, Unit) Plugin.CreateRGBWWDevice(DeviceID, Unit) Plugin.CreateWindDevice(DeviceID, Unit)

Plugin.UpdateTempDevice(DeviceID, Unit, Value) Plugin.SwitchLight(DeviceID, Unit, State, Level)

This will also restrict the type of device type/subtype inside the internal plugin system

Maybe it is not to late to do this, It probably breaks all plugins, but it will be better for the feature

[DatafoxChris]

@dnpwwo would you agree this is the best way to go for the future ?

btw: one Zigbee Network can support up to 65.000 devices :)

[dnpwwo]

@DatafoxChris I guess to me there a couple of things:

1. The framework has been around for a while and there are a lot of plugins used by a lot of people. Breaking compatability will cause chaos and be unpopular because some plugin authors are not active anymore so some people will simply lose functionality for ever. Is one use case enough to justify this? I'm not convinced that it is.

2. Python dictionaries require a single key (which was why I chose Unit) for the Devices dictionary, what would that change to in an implementation with a multi-part key? A concatination? a tuple?

3. The framework was designed to shield hobbyist developers from the complexity of things like 5 part unique device keys. I hoped that people with limited coding ability would be able to add support for their personal devices to their own system in any way they saw fit so I am not a huge fan of adding arbitrary restrictions without compelling reasons.

4. I also understood that there would always be some hardware support that would be best done in C and I wonder if this is not the case here? Up to 65K devices is a lot and Python may struggle with that.

@gizmocuz, what are the implications of moving Unit from 1 byte to multiple bytes?

[DatafoxChris]

I would vote for adding new functions which take the proposed parameters, so that old plugins still work, and 'new' plugins can use the extended functionality when needed. This would solve the problem of breaking compatibility, and no need to change Unit to 16 bits. Why not use Domoticz internal Device IDX as the key for the dictionaries?

[gizmocuz]

@dnpwwo , I think the proposed methods are extremely clear (createTempDevice, updateTempDevice) for novice developers as well. Having 255 units seems more then enough, it is also not meant as a ID, it's for (for example) switches that have like 4 buttons, the DeviceID is of course the same, but each button has a different unit value It's not only used by 433Mhz hardware, but should also used by plugins

I agree that keeping the old plugins work would be the best approach for now, and hopefully we can migrate those not maintained plugins later Maybe adding a deprecated message in the log over time ? And in time stop the plugin

[stas-demydiuk]

@gizmocuz what's the benefit having own methods for each device type (Temp, Humidity, Temp+Hum, etc... I guess about 50 different devices) over the universal way we have now? I see that in the first case it will require much more effort from a person who maintains the plugin system in addition to the effort from plugin developers to rewrite their plugins.

I agree with @DatafoxChris that adding another dictionary with deviceID as key would be enough to extend existing limit.

[DatafoxChris]

Sorry, i wrote DeviceID but what i meant was Domoticz internal IDX.

[gizmocuz]

@stas-demydiuk , this Document is very unclear in my perspective:

https://www.domoticz.com/wiki/Developing_a_Python_plugin

It is talking about 'Unit' , but I think it is referring to IDX (Index) correct ? Same IDX value as from the Device page ?

This Index value in Domoticz (Unique index / device) , is not a 8 bit field but 32

[stas-demydiuk]

No, it's referring to unit, not device idx

[gizmocuz]

@stas-demydiuk , Thanks I see. This is a bit strange to me. I would create a new device based on HardwareID (self, the plugin_id), DeviceID (a hex string or a number, not starting with 0x or ending with text like _kwh), Unit, DeviceType/Subtype

Unit's are used for (example):

• A measurement node that has like 4 inputs (1 deviceID, 4 different units)
• A switch that has multiple buttons (also same device_id, multiple units)
• A remote control
• A temperature sensing node with two inputs
• ...

IF you are going to use units in the above screenshot, then this would mean that all 'kwh' devices would have the same ID but different units As fas as I understand, each zigbee device has a unique ID/type, so I would just use this DeviceID/Type (and Unit=1), and after creation use it's IDX value It must be possible to make/store/load a dictionary containing the IDX values of the devices

[stas-demydiuk]

Yes, you are right, a combination of ID+Unit for the specific device would be more sufficient but it is the limitation we currently have from the plugin-system side, as @dnpwwo mentioned it provides a Dictionary with devices associated to plugin and Unit is key in this dictionary.

I guess at this point we need to think about how we can extend the plugin system to increase amount of supported devices from one side and to leave backward compatibility with existing plugins from another side.

I would suggest to add 2 additional methods to Domoticz object to manage devices by DeviceID and Unit

Domoticz.CreateDevice(DeviceID="id", Unit=1, ...) // returns created device
Domoticz.GetDevice(DeviceID="id", Unit=1) // returns device by id and unit or None

and leave Devices dictionary as is now for backward compatibility

In this case, the number of supported logical devices will be increased significantly and we will be able to use own DeviceID for each physical device and Unit for its subdevices

@gizmocuz / @dnpwwo what do you think?

[gizmocuz]

@stas-demydiuk , Yes that would be a good solution, but I'm not good at Python. Will Python allow one dictionary to be used as a single and as a double array ? That would be ideal as for the old way we can still use Devices[1] and the new way Devices["id", 1]

[DatafoxChris]

It seems you can use tuples as the dictionary index

Matrix = {} Matrix[1,2] = 1234

[martin-g-it]

Hi guys, today I ran into the issue of 255 devices on the Z2M plugin. Seems like this has been worked upon through this item. Any update on this feature being resolved? Meanwhile I get errors on Domoticz like either: `Error: (Zigbee2MQTT) Device creation failed, Domoticz settings prevent accepting new devices. - when disabling Accept new Hardware'.. When allowing this, the 255 devices is exceeded and you get the other error about zigbee device object/model not recognized..

[gizmocuz]

No not yet. The python system needs to implement the normal way of device handling (using idx, hardware_id+device_id+unit), and this has to be implemented (new API calls) Hope someone can perform this task Else, first a new Domoticz stable release, and then I could try to do this, But it might break all existing plugins

[martin-g-it]

Just had an idea; based on an additional hardware parameter (Mode2 is still unused I believe): Create 9 additional 'slave' hardware devices for the plugin; then based on the IEEE Address (hex value converted to decimal) bring it under hardware ID 0-9. These hardware ID's are then also listening to the MQTT topics, the same logic can be applied Ie. hardware ID ..3 only listens to messages related to the (converted) IEEE Address ending at ..3.

[dobber81]

Would it be possible to add an exclude-file in the plugin directory? Where each line is a sub-device to be ignored? 0x00158d000104201f_cell for example. I know this is how the plugin for deconz-domoticz does it. Devices in that file would never show up in domoticz and would leave room for other devices.

I'm sorry if this has been suggested before. I read through a few threads about this problem, but could not find anyone suggesteing this as a workaround.

Maybe the list of ignored sub-devices could even be accessed through a tab in "Custom"-"Zigbee2mqtt" alongside devices, groups and network map?

[martin-g-it]

Great idea..! Almost all of my remotes for example I don’t want to be added. Also for eg. Aqara temp sensor there is like 7 devices created (excl. batt lvl and LQI) whereas I would only need 1...

[hughb8on]

May I draw your attention to Issue #594 where I set out my own workaround, coded and tested, which works exactly as @dobber81 suggests.

I started a separate issue because I think two quite separate things needs to be done: firstly the domoticz framework needs to be changed to allow plugins to create more than 255 devices, and secondly the plugin needs to be changed to allow unwanted features to be suppressed (issue #594).

[felix63]

Would be very nice if this was fixed. Btw I don't understand why for ZWave devices the Unit value are not unique and for Zigbee they should be. Is this because Zigbee is a plugin and ZWave is not? Seems to be quite inconsistent

[jkl1337]

Why isn't the key for the devices dictionary and all the callbacks simply the Idx (ie the ID column)? This is the primary key for devices, and is guaranteed to be unique. I am a bit puzzled as to why Unit was ever a better option. Unfortunately there are plugins that depend on Unit being a small integer (like the Tasmoticz as an example). Perhaps the best path forward is to have a flag for plugins in the manifest to use Idx for callbacks and the device dictionary instead.

[gizmocuz]

@jkl1337 , yep, that's what I've been saying quite some time now. Also a device uniqueness is a combination of:

Type + SubType + Unit

and not only 'Unit'

Someone has to make a PR that changes devices creation/updating.

I will soon freeze the code to make a new stable, and we should change this for the upcoming beta version. This does mean that plugins needs to be updated. Also the settings of the plugins should change to use the hardware's Extra and/or Configuration database field and this should be a json array. It should not use the database fields Mode1 till Mode 6

[waltervl]

See also the discussion on the forum: https://www.domoticz.com/forum/viewtopic.php?f=65&t=35452 And also the following PR: https://github.com/domoticz/domoticz/pull/4743

[Dylantje]

Thanks @stas-demydiuk i did not find this clone of my problem..

Only i run v3 how can i go to v4?

[stas-demydiuk]

There is no v4 yet, as well as there is no fix to overcome 255 devices limit

[Dylantje]

mm thats no good news... any way.. Only question, perhaps this problem is a reason for my other problem? https://github.com/stas-demydiuk/domoticz-zigbee2mqtt-plugin/issues/687

And when i remove some idx like link quality etc. Are there then new idx possible?

[waltervl]

There is no v4 yet, as well as there is no fix to overcome 255 devices limit

What about the extended plugin work dnpwwo is doing in beta? Is it too much work to switch over?

[Dylantje]

@stas-demydiuk Domoticz Forum some says: https://www.domoticz.com/forum/viewtopic.php?p=279135#p279135 Is this a option, supported by u? Do you now if this working? I cannot find something like DomoticzEx

I read something there is also a plugin voor zwave to mqtt and perhaps they say possible to use 1 plugin for zwave and zigbee. Are you working with that also?

[heggink]

The current zwavejs2mqtt plugin is using the same framework with the 256 device limit but you can run multiple instances where a config file defines which devices go where (eg sensors in one and switches is another). What's more interesting is that gizmocus is working on getting home assistant auto discovery into domoticz. When that works, we no longer need zwavejs2mqtt. If zigbee2mqtt also supports HASS autodiscovery then this plugin can also be replaced.