Berry energy.read outputs invalid JSON (nan issue)

[ouinouin]

PROBLEM DESCRIPTION

While issuing the berry command energy.read, and trying to parse it on my comptuter with JQ, i discovered that nan is not a subset of JSON spec, we should use instead "null" or another significant string if needed

REQUESTED INFORMATION

• [x] Read the Contributing Guide and Policy and the Code of Conduct
• [x] Searched the problem in issues
• [x] Searched the problem in discussions
• [x] Searched the problem in the docs
• [x] Searched the problem in the chat
• [x] Device used (e.g., Sonoff Basic): all ESP32devices
• [x] Tasmota binary firmware version number used: 13.4.0
  • [x] Pre-compiled
  • [ ] Self-compiled
• [x] Flashing tools used: webinstaller
• [ ] Provide the output of command: Backlog Template; Module; GPIO 255:

  
  Configuration output here:

- [ ] If using rules, provide the output of this command: `Backlog Rule1; Rule2; Rule3`:
```lua
  Rules output here:

• [ ] Provide the output of this command: Status 0:

  
  STATUS 0 output here:

- [ ] Set `weblog` to 4 and then, when you experience your issue, provide the output of the Console log:
```lua
  Console output here:

TO REPRODUCE

issue the command energy.read() and parse the result in the jq command on a linux computer

EXPECTED BEHAVIOUR

replace nan by a valid json string or a null.

SCREENSHOTS

ADDITIONAL CONTEXT

indeed i discovered this because i have nan on reactive power , and i couldn t see the apparent power in the json despite being available on tasmota

{'data_valid': 0, 'today_delta_kwh': 144, 'frequency_common': 0, 'power_history_0_3': 11, 'today_kwh_3': 0, 'today_offset_init_kwh': 1, 'frequency_3': nan, 'power_history_1_2': 0, 'data_valid_2': 0, 'max_voltage_flag': 0, 'reactive_power': nan, 'energy_active_export': 0, 'start_energy': 0, 'yesterday_sum': 0, 'command_code': 0, 'today_offset_kwh_3': 0, 'frequency_2': nan, 'start_energy_2': 0, 'data_valid_3': 0, 'min_power_flag': 0, 'voltage_available': 1, 'total': 0.00763, 'daily': 0.00763, 'today_kwh': 763, 'max_power_flag': 0, 'frequency': nan, 'voltage_2': 0, 'import_active': 0, 'period': 0.00724, 'import_active_3': 0, 'start_energy_3': 0, 'voltage': 231.435, 'power_history_2_2': 0, 'type_dc': 0, 'total_sum': 0.00763, 'export_active_2': nan, 'total_3': 0, 'power_history_1': 0, 'current_available': 1, 'today_delta_kwh_2': 0, 'max_energy_state': 0, 'total_2': 0, 'import_active_2': 0, 'daily_sum_import_balanced': 0.00763004, 'power_factor': nan, 'today_offset_kwh_2': 0, 'power_factor_3': nan, 'current_2': 0, 'export_active_3': nan, 'power_factor_2': nan, 'mplw_counter': 0, 'today_kwh_2': 0, 'today_delta_kwh_3': 0, 'reactive_power_2': nan, 'phase_count': 1, 'power_history_2_3': 0, 'daily_3': 0, 'current_3': 0, 'current': 0.120211, 'active_power_3': 0, 'apparent_power_3': nan, 'use_overtemp': 1, 'power_steady_counter': 0, 'voltage_3': 0, 'min_current_flag': 0, 'period_2': 0, 'reactive_power_3': nan, 'mplh_counter': 0, 'period_3': 0, 'daily_2': 0, 'daily_sum_export_balanced': 0, 'voltage_common': 0, 'active_power_2': 0, 'active_power': 9.96319, 'apparent_power_2': nan, 'export_active': nan, 'power_history_2': 0, 'today_offset_kwh': 0, 'power_on': 1, 'power_history_0_2': 11, 'power_history_0': 9, 'mplr_counter': 0, 'apparent_power': nan, 'fifth_second': 1, 'daily_sum': 0.00763, 'min_voltage_flag': 0, 'max_current_flag': 0, 'power_history_1_3': 0}

(Please, remember to close the issue when the problem has been addressed)

[sfromis]

You did not post any JSON from Tasmota, but instead something else.

However, it does look like using Berry math.nan does not get handled gracefully when converting to JSON, using json.dump()

import math,json
print(json.dump({'e':math.nan}))
{"e":nan}

[barbudor]

Apparently NaN is invalid in JSON

[sfromis]

Yeah, values can only be strings, numbers, nested objects, arrays, true, false or null.

[Jason2866]

The question is what is a good replacement for nan? null is not correct.

[barbudor]

it's the only thing allowed by JSON standard

[barbudor]

however it appears that python is using "Nan" and that seems to work with parsers https://stackoverflow.com/questions/6601812/sending-nan-in-json

[sfromis]

A question may be if the "message" of math.nan in this case is "really" not-a-number, or maybe is not too far from null/nil, in the meaning of "missing".

Here's a workaround.....

import math,energy,json
var em=energy.read()
for key: em.keys() if math.isnan(em[key]) em[key]=nil end end
var ej=json.dump(em)
print(ej)

[s-hadinger]

This is definitely a bug in the json module. Will fix it.

[sfromis]

Or maybe energy.read() should not be using math.nan from the start?

[s-hadinger]

nan is anyways used in Tasmota

[ouinouin]

seems i lifted a debate :-) indeed, discovered that issue while playing with energy.read, and it doesnt reports the apparent and reactive power ( @s-hadinger shall i open a dedicated issue for that ?) , been reading the code but its above my competencies to track why it reports nan. another point of confusion regarding sticking or not to json, its the simple vs double quoting, to my understanding, normally json requires surrounding strings with double quotes and not single quotes, in commands like energy.read, and i guess all what is a map in berry is surrounded by single quotes. despite map is not advertised as being json compatible, it might be a good idea to stick to json standard :-). I leave it for the doers, since im just a sayer :-).

[sfromis]

As I already mentioned, what you initially posted is indeed not JSON, and Berry makes no claim to it being JSON. It is instead the result of calling the tostring() method of the class, which will be done when printing the instance, or using the str() function. Instead of it being JSON, it is a way of representing a map directly in Berry source code. Well, except for this not working with nan, without using math.nan.

That's why I was careful to talk about "converting to JSON, using json.dump()". Similarly, when you have some JSON in a string, you can convert it to a map using json.load()

I see the usage of single quote in the string representation of a map as a "reminder" that it is not JSON, even through Berry would also support using double quote as string delimiter. However, using double quote would still not result it in being valid JSON, as Berry is using nil instead of null, and also supports nan. Of course, the "JSON-like" representation of a map in Berry source code also supports expressions, including numbers as hex.

[s-hadinger]

The bug still exists in Berry json.dump:

import json
import math
print(json.dump(math.nan))
# outputs 'nan' which is not valid json, hence the bug in json module

[s-hadinger]

The json library is now fixed and converts non-numerical values to null