Null json result gives HA warning every 15 minutes

[gcoan]

I've setup the REST sensors for Octopus Power up REST and Octopus Free Electricity events.

Noticed in the HA logfile that the empty JSON file is resulting in a warning from each of the REST polls every 15 minutes:

2024-08-25 01:11:08.521 WARNING (MainThread) [homeassistant.components.rest.util] JSON result was not a dictionary or list with 0th element a dictionary
2024-08-25 01:11:08.564 WARNING (MainThread) [homeassistant.components.rest.util] JSON result was not a dictionary or list with 0th element a dictionary
2024-08-25 01:26:08.542 WARNING (MainThread) [homeassistant.components.rest.util] JSON result was not a dictionary or list with 0th element a dictionary
2024-08-25 01:26:08.543 WARNING (MainThread) [homeassistant.components.rest.util] JSON result was not a dictionary or list with 0th element a dictionary
2024-08-25 01:41:08.527 WARNING (MainThread) [homeassistant.components.rest.util] JSON result was not a dictionary or list with 0th element a dictionary
2024-08-25 01:41:08.532 WARNING (MainThread) [homeassistant.components.rest.util] JSON result was not a dictionary or list with 0th element a dictionary
2024-08-25 01:56:08.544 WARNING (MainThread) [homeassistant.components.rest.util] JSON result was not a dictionary or list with 0th element a dictionary
2024-08-25 01:56:08.575 WARNING (MainThread) [homeassistant.components.rest.util] JSON result was not a dictionary or list with 0th element a dictionary

Can the JSON file (& possibly REST GET) be changed (e.g. an 'empty' tag) to prevent these warnings which are rather filling up the logfile....

[8none1]

At the moment, if there are no power ups then it returns an empty list - which is valid JSON.

I think returning an empty object (i.e. {}) will probably still cause the same warning.

I tested {"powerups": null} which is (I think) the correct way to indicate a null, but it still generates the same error.

I then tested an empty file, and that resulted in the error REST result could not be parsed as JSON. Which is fair, because it's not valid.

So in short, I don't know. I think the fix is probably in the HA config.

Any ideas?

[8none1]

... and {[]} says:

WARNING (MainThread) [homeassistant.components.rest.util] REST result could not be parsed as JSON

[gcoan]

I did wonder whether {"powerups": []} would be acceptable?

Searching for this error code to see what other solutions there are:

1. https://community.home-assistant.io/t/json-error-from-rest-sensor/383789/

suggests in the template sensor checking if the JSON returned is a single item or an array

2. https://github.com/home-assistant/core/issues/41753

changes around the quotes in json_attributes_path

makes me think the error is in the array parsing bit where it finds a 0 element array its not expecting

3. https://community.home-assistant.io/t/rest-sensor-how-to-avoid-empty-json-dictionary/294749/3

got an empty json result after midnight. Never worked out how to stop the warning 🤔

4. https://community.home-assistant.io/t/rest-sensor-with-empty-json/734182

same problem interrogating Uber Eats orders when no orders present, no solutions posted so far

Can suppress the error in recorder.logger excludes, but that's a mega hack and I don't like it.

Is option 1 maybe the direction to go if we can't find an acceptable null string output? Instead of letting the REST sensor do the JSON parsing, get the data as a long blob and then parse it into the attributes required in custom template code? Hacky but might work?

[gcoan]

Off topic of these null array warnings, but first power up event came through ok-ish today.

{{ states('sensor.octopus_power_up_times') }}

returns [{"start":"2024-08-28T11:00:00.000Z","end":"2024-08-28T13:30:00.000Z"}]

Am able to get the start and end values:

p/up start {{ state_attr('sensor.octopus_power_up_times', 'start') }} p/up end {{ state_attr('sensor.octopus_power_up_times', 'end') }}

returns start: 2024-08-28T11:00:00.000Z end: 2024-08-28T13:30:00.000Z

Was surprised to see the times in Zulu UTC rather than local BST time. Wondered if this was intentional or accidental given they are in local time in the email that's parsed.

Converting these to date objects so I can load them into HA sensors, less successful ☹️

Trying to pull the date out of the start time attribute:

p/up date {{ strptime(state_attr('sensor.octopus_power_up_times', 'start'),'%d/%m/%Y') }}

errors. strptime doesn't like the supplied value '2024-08-28T11:00:00.000Z', it can't convert it to a date time object. I'll fiddle with this a bit more, this is first feedback

[8none1]

Was surprised to see the times in Zulu UTC rather than local BST time. Wondered if this was intentional or accidental given they are in local time in the email that's parsed.

Yes, this is intentional. It simplifies a lot of things, the main one being "what happens if a power up spans a daylight savings change", and once you know the secret is very easy to deal with....

HA's templating has a as_datetime filter which does all the hard work for you.

This example creates a sensor which turns on and off when the power up starts and ends, calculates the duration of the power up and how much time is left:

https://github.com/8none1/octopus_powerups?tab=readme-ov-file#binary-power-up-in-progress-sensor

[gcoan]

Thanks, I was going to go back and look at your sample code to see how you had manipulated it, but ran out of time last night.

Do find date/time stuff complex, there's different ways of approaching what should be the same thing. as_timestamp worked so I was surprised that strptime didn't. Anyway, as_datetime is working as well

[8none1]

as_datetime is working as well

Excellent! Glad to hear it.

Thanks for the testing and feedback!

I'm still pondering the null thing, but will hold until the PU is over as not to break anything....

[gcoan]

Looking just now at the rest API documentation https://www.home-assistant.io/integrations/sensor.rest

it points to https://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html for how XML to JSON conversion is done

Pattern 1, a null entry in JSON is represented "e" : null

so suggests that the null entry should be:

[ "start" : null, "end" : null ]

or similar

[8none1]

Nice, thanks!

I've updated the test file to:

[{"start":null, "end":null}]

Hosted here: https://www.whizzy.org/octopus_powerups/powerup_test.json

I've replaced the other URL with this in my HA set up and I'm now waiting to see if anything complains....

[8none1]

... I think it's working!

I'm going to change the GS script to produce that when empty and let's see what breaks...

[8none1]

Ok, done and deployed.

I'm testing my end, but let me know what happens in your logs too. :rocket:

[gcoan]

I'd configured the test sensor on my test HA, then my host PC crashed, so by the time I rebooted you'd already rolled the change out!

It appears to be working for me too. I got JSON warnings at 13:12 and 13:27, but the sensor changed at 13:42 with no warning in the log

Octopus Power Up Times changed to [{"start":null,"end":null}] 13:42:01 - 10 minutes ago

state_attr('octopus_powerup_times', 'start') and 'end' both return none correctly

Just need to test it when there's a free session again. Not at the moment, very grey, my batteries don't fill during the day but are fortunately still lasting through the night (which is expensive right now on Agile)

[8none1]

Yeah, I could use some wind - which is not something I say very often.

[8none1]

And just like that, a Power Up tomorrow. Everything looks good this end.

[gcoan]

And just like that, a Power Up tomorrow. Everything looks good this end.

Likewise, data came through OK for me. I saw the power up email before the sensor had the value, but was probably between polling periods because when I next looked it was populated OK.

Need to get back on to the code to automate writing the rest interface results to my power up controls, but looks like this is sorted now 👍

[8none1]

I saw the power up email before the sensor had the value, but was probably between polling periods because when I next looked it was populated OK.

The script that populates Github only runs once an hour at 15 mins past. I could run it more frequently but there has never been a Power Up with such short notice that it was required.

Thanks for the assistance in making this improvement, much appreciated. Closing as fixed.