Managed counters broken since release 2023.2

[guillaumezin]

Hello,

Since release 2023.2 release, managed counters (for instance P1SmartMeter created with option AddDBLogEntry set to "true" won't show correctly on dashboard (from https://github.com/guillaumezin/DomoticzLinky/issues/211):

To reproduce, create a P1 Smart Meter, add the AddDBLogEntry option (for instance with setused command).

/json.htm?type=command&param=createdevice&idx=200&sensorname=testmanaged&devicetype=250&devicesubtype=1
/json.htm?type=command&param=setused&idx=XXX&used=true&options==QWRkREJMb2dFbnRyeTp0cnVl

If you insert value to show, it will send an error:

/json.htm?type=command&param=udevice&idx=IDX&nvalue=0&svalue=10;0;20;0;30;0

But it will work if you remove AddDBLogEntry option

/json.htm?type=command&param=setused&idx=XXX&used=true&options==IEFkZERCTG9nRW50cnk6ZmFsc2U=
/json.htm?type=command&param=udevice&idx=IDX&nvalue=0&svalue=10;0;20;0;30;0

Now we set AddDBLogEntry to true again

/json.htm?type=command&param=setused&idx=XXX&used=true&options==QWRkREJMb2dFbnRyeTp0cnVl

If I want to insert a data directly in db, I append the date:

/json.htm?type=command&param=udevice&idx=IDX&nvalue=0&svalue=11;0;21;0;31;0;2023-07-25

I have no error, but my dashboard widget is updated (it shouldn't):

My db is updated as expected:

I think the problem is in main/WebServer.cpp, in UpdateManagedValueInt() function. After each UpdateCalendarMeter() call, the function should return ulID immediatly, if at the end of the function and we didn't call any UpdateCalendarMeter() (date not present in sValue), we should call the safe_query(UPDATE DeviceStatus...)

[gizmocuz]

Okey but a p1 meter is not a managed counter, why don't you use it as described in our JSON wiki?

What happens when you use a real managed counter?

[gizmocuz]

Also the P1 meter should be used for Dutch or France energy meters with either the P1 hardware or Teleinfo and nothing else If you created funky hardware and are able to use the Teleinfo or P1 hardware I suggest to use this instead or use real managed counters

[guillaumezin]

Hello,

The problem is the same with sTypeManagedCounter.

DomoticzLinky doesn't really rely on hardware directly, in France, Enedis collects data from French meters and gives access to data the day after. The plugin retrieves the data and inject them in Domoticz DB, no need to add hardware or wires. In first implementation of plugin, we created sTypeManagedCounter. Then, I wanted to add day / night cost and difference between production and consumption, P1 smart meter was closed to what was needed, and instead of adding another sTypeManagedCounter2, I proposed AddDbLogEntry and DisableLogAutoUpdate options, it is more generic and future proof.

[gizmocuz]

I have no issues with the current implementation of the managed counters.... You should not either I suppose but I will have a look at them tomorrow For a managed counter, you have to provide the shortlog entries and the day entries yourself Nothing should be done by domoticz as it is managed somewhere else (hence the name and type)

Now over time you seemed to hack the default sensors in a way that they should act as managed devices I see if I can restore this. But also this will mean that you have to manage it complete yourself as well So no nightly adds to database tables... Correct?

[guillaumezin]

Yes, if AddDbLogEntry and DisableLogAutoUpdate options are set, it means it is the responsabilty of the plugin to manage the sensor completely.

I'm not sure to understand what you mean by not adding to database nightly, but Enedis closes it's servers over night, so yes, the plugin usually adds data on the morning usually, at around 8am.

[gizmocuz]

With nightly I probably think it means AddDbLogEntry… not very clear what this does and ain’t this default and if so, why is this not called DisableDBLogEntry?….

[guillaumezin]

Hello,

From https://www.domoticz.com/wiki/Developing_a_Python_plugin, AddDbLogEntry enables inserting values in history log if you append a date to svalue, DisableLogAutoUpdate disables auto update of week/month/year history log from domoticz.

Feel free to change options names if you have a better idea, I will update my plugin and the wiki, but DisableDBLogEntry makes less sense to me, and I don't know if others plug-ins or scripts are already using these options.

[gizmocuz]

DisableLogAutoUpdate Should maybe logically be 'IsManaged' ... Anyway, seems like we have to parse the options on every device update to check against these options .... That's quite intensive.... Why don't you use a normal P1 meter and update this every xx seconds?

And your plugin real-time?

[gizmocuz]

I will see what I can do to solve this... Could take some time... But you can't get real-time data via Teleinfo?

[gizmocuz]

You had a small type here (double ==)

/json.htm?type=command&param=setused&idx=XXX&used=true&options==QWRkREJMb2dFbnRyeTp0cnVl

Did you wrote the c++ implementation bdw? Why not treat the last svalue as date/time, now it's the contents is split , and checked how large the returned split array is. Who knows in the feature we have a sensors that takes 20 parameters as sValue (... any value)

when setting 'AddDBLogEntry' to true, it of course means that a call without date/time goes wrong, so this one is invalid:

/json.htm?type=command&param=udevice&idx=IDX&nvalue=0&svalue=10;0;20;0;30;0

Sure it is valid when it is set to false.

So far still good!

now calling:

/json.htm?type=command&param=udevice&idx=IDX&nvalue=0&svalue=11;0;21;0;31;0;2023-07-25

This is indeed valid and a database entry is also created!

There is No problem in UpdateManagedValueInt and UpdateCalendarMeter and UpdateManagedValueInt is returning the idx of the updated device correctly.

[gizmocuz]

The problem is inside DeviceStatus as it contains the date/time as well and this should be removed...

Done in the latest source commit, beta is following.

https://github.com/domoticz/domoticz/commit/e04251779538d85decabb0323ab539833796a518

[guillaumezin]

We don't understand each other, and I'm sorry I cannot send a patch, I'm away from my computer now for a few weeks and I have a hard time checking your code from my smartphone.

Yes, I made the C++ implementation.

But calling /json.htm?type=command&param=udevice&idx=IDX&nvalue=0&svalue=10;0;20;0;30;0 shall still be valid, even with AddDBLogEntry set, in that case, we shall update device status only.

If we call /json.htm?type=command&param=udevice&idx=IDX&nvalue=0&svalue=11;0;21;0;31;0;2023-07-25, we shall call UpdateCalendarMeter, but not update device status.

We must not do both at the same time, no need to remove the date, we're not supposed to update the device status if we have a date.

Look at my implementation, before you moved it in UpdateManagedValueInt function, it was what was done.

[gizmocuz]

Okey made some changes and this should now work like before again in the latest code commit Betas are building

[guillaumezin]

Thank you, it seems correct to me, I'll ask plugin users to test it and I'll let you know.

[gizmocuz]

Okey, 15474 is available

[guillaumezin]

Confirmed by many as working as expected, thank you.

[mgrafr]

Bonjour, Comme beaucoup après la maj vers 2023-2 le compteur a affiché INVALID; Je suis sous Docker , j'ai donc aujourd'hui mis à jour DZ vers la build 15525 ; Contrairement à ce que certain ont constaté, j'ai toujours la même erreur. { "ActTime": 1693754878, "AstrTwilightEnd": "22:20", "AstrTwilightStart": "05:40", "CivTwilightEnd": "21:05", "CivTwilightStart": "06:55", "DayLength": "13:10", "NautTwilightEnd": "21:41", "NautTwilightStart": "06:18", "ServerTime": "2023-09-03 17:27:58", "SunAtSouth": "13:60", "Sunrise": "07:25", "Sunset": "20:35", "app_version": "2023.2 (build 15525)", "result": [ { "AddjMulti": 1, "AddjMulti2": 1, "AddjValue": 0, "AddjValue2": 0, "BatteryLevel": 255, "Counter": "0", "CounterDeliv": "0", "CounterDelivToday": "Invalid", "CounterToday": "Invalid", "CustomImage": 0, "Data": "Invalid!: 13652.0;0;0;0;0;0;2023-09-01", "Description": "Compteur Linky 16267727923561",

J'ai fait un git pull mais il m'indique que le plugin est à jour.

Est -ce l'installation sous Docker qui peut poser problème ?

[gizmocuz]

@mgrafr , the international language is English. Could you ask the question again? Also post a http/json call that you use to update the counter