ERROR - RPC request time out (no "°C" unit support?)

jellewie commented 1 year ago

We have a 'Sunny WebBox' version 'G1' and it seems to not be able to run this integration. I have not found out what the issue it yet. It seems that the data is in the wrong format or something.

SMA info: Versie Firmware 1.52 Versie apparaatprofielen 1.21
Versie taalgegevens 1.06
Versie Hardware WEBBOX-G1-150002608

Some log items: (Sorry for Dutch, the HA is setup that way)

Logger: homeassistant.config_entries
Source: config_entries.py:424
First occurred: 10:13:45 (1 occurrences)
Last logged: 10:13:45

Config entry '192.168.1.245' for sma_webbox integration not ready yet: RPC request timed out; Retrying in background

Deze fout is ontstaan door een aangepaste integratie.

Logger: homeassistant
Source: custom_components/sma_webbox/sma_webbox.py:140
Integration: SMA Webbox (documentation, issues)
First occurred: 10:13:40 (14 occurrences)
Last logged: 10:26:21

Error doing job: Exception in callback _SelectorDatagramTransport._read_ready()
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 1027, in _read_ready
    self._protocol.datagram_received(data, addr)
  File "/config/custom_components/sma_webbox/sma_webbox.py", line 140, in datagram_received
    data = json.loads(data.decode().replace("\0", ""))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 574: invalid start byte

Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.sma_webbox.sma_webbox
Source: custom_components/sma_webbox/sma_webbox.py:333
Integration: SMA Webbox (documentation, issues)
First occurred: 10:13:45 (14 occurrences)
Last logged: 10:26:26

RPC request timed out

Traceback (most recent call last):
File "/usr/local/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 1027, in _read_ready
self._protocol.datagram_received(data, addr)
File "/config/custom_components/sma_webbox/sma_webbox.py", line 140, in datagram_received
data = json.loads(data.decode().replace("\0", ""))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 574: invalid start byte
2023-02-22 10:30:54.007 WARNING (MainThread) [custom_components.sma_webbox.sma_webbox] RPC request timed out
2023-02-22 10:30:54.010 WARNING (MainThread) [homeassistant.config_entries] Config entry '192.168.1.yyy' for sma_webbox integration not ready yet: RPC request timed out; Retrying in background
2023-02-22 10:31:08.233 ERROR (MainThread) [homeassistant] Error doing job: Exception in callback _SelectorDatagramTransport._read_ready()

Traceback (most recent call last):
File "/usr/local/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.10/asyncio/selector_events.py", line 1027, in _read_ready
self._protocol.datagram_received(data, addr)
File "/config/custom_components/sma_webbox/sma_webbox.py", line 140, in datagram_received
data = json.loads(data.decode().replace("\0", ""))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 574: invalid start byte
2023-02-22 10:31:13.154 WARNING (MainThread) [custom_components.sma_webbox.sma_webbox] RPC request timed out
2023-02-22 10:31:32.340 ERROR (MainThread) [homeassistant] Error doing job: Exception in callback _SelectorDatagramTransport._read_ready()

jpcornil-git commented 1 year ago

The label (see back of the box) mentions WEBBOX-BT-20-GR1 as type on mine.

Could you:

dump something similar from your webbox http interface:

run the following command (cfr https://github.com/jpcornil-git/HA-sma#sma_webboxpy) from a CLI and report logs ?

python3 custom_components/sma_webbox/sma_webbox.py -d 192.168.1.245

Cheers, jpc

PS: I can handle/speak dutch as well ;-)

jellewie commented 1 year ago

For anyone looking for an answer, I could not find out how to solve this, but since the data was available on the home screen on the loginpage. I have just used the web 'Scrape' integration for now

Scrape:

Just a POST request to the URL "http://192.168.1.1/home.htm"

Looking at the source code of the website we see the values

<tr class="evenrow">
<td class="tdcol0">Power:</td>
<td class="tdcol1" id="Power">123 W</td>
</tr>
<tr class="oddrow">
<td class="tdcol0">Daily yield:</td>
<td class="tdcol1" id="DailyYield">56 kWh</td>
</tr>
<tr class="evenrow">
<td class="tdcol0">Total yield:</td>
<td class="tdcol1" id="TotalYield">789 MWh</td>

So for the first one:

Name: "Power production"
Selector: "td#Power"
value template: "{{value |regex_replace(" W")}}"
Device_class: "power"
state_class: "measurement"
Unit "W"

jellewie commented 1 year ago

Is This something you are looking for?

jellewie commented 1 year ago

I don't know how to run it in HomeAssistant, and it cant seem to tun on Windows

I also tried to play around with the API, but it does not seem to respond well to what I managed to send

jpcornil-git commented 1 year ago

It is definitively a different box/hardware (webbox-G1-150002608 vs webbox-bt-20 GR1 for me)

Wrt the POST request, you forgot to prefix the json dict with "RPC=", i.e. payload should be:

RPC={"version":"1.0","proc":"GetPlantOverview","id":"1","format":"JSON"}

You can also use your web browser and the following URL to check for RPC support http://192.168.1.245/rpc?RPC={%22version%22:%221.0%22,%22proc%22:%22GetPlantOverview%22,%22id%22:%221%22,%22format%22:%22JSON%22}

If nothing is returned, you box unlikely supports the RPC API.

I guess you tried the SMA Solar integration already without success either ?

Cheers, jpc

PS: If you are running the HAOS version, you can access the linux container running HA using the "Terminal & SSH' addon (access via ssh or directly from HA web UI)

jellewie commented 1 year ago

(Tried 'SMA Solar' without luck (wrong authentication, but I can log into the website with credentials) (Also the SSH terminal seemed to run in a sort of docker/vm, because It didnt find the file or python or something, so I gave up on that.

BUT NOW SOME GOOD NEWS! I edited the Talends API and I got some good responses. Will dump some below for you to look at

GetPlantOverview Request GetPlantOverview Response:

{"format":"JSON","result":{"overview":[{"unit":"W","meta":"GriPwr","name":"GriPwr","value":"0"},{"unit":"kWh","meta":"GriEgyTdy","name":"GriEgyTdy","value":"0.000"},{"unit":"kWh","meta":"GriEgyTot","name":"GriEgyTot","value":"521582.599"},{"unit":"","meta":"OpStt","name":"OpStt","value":""},{"unit":"","meta":"Msg","name":"Msg","value":""}]},"proc":"GetPlantOverview","version":"1.0","id":"1"}

GetDevices Response:

{"format":"JSON","result":{"totalDevicesReturned":7,"devices":[{"key":"SENS0500:1173","name":"SENS0500:1173"},{"key":"WR6KTL05:2000237749","name":"WR6KTL05:2000237749"},{"key":"WR6KTL05:2000237809","name":"WR6KTL05:2000237809"},{"key":"WR6KTL05:2000237819","name":"WR6KTL05:2000237819"},{"key":"WR6KTL05:2000237823","name":"WR6KTL05:2000237823"},{"key":"WR7KTL05:2000230787","name":"WR7KTL05:2000230787"},{"key":"WR7KTL05:2000230808","name":"WR7KTL05:2000230808"}]},"proc":"GetDevices","version":"1.0","id":"1"}

I have more, let me know what you need or where I can help

jellewie commented 1 year ago

The "Terminal & SSH" seems to run in sandbox mode, but addon-ssh has an option to run the command python3 config/custom_components/sma_webbox/sma_webbox.py <webbox ip_address>

The error that that it throws out is:

INFO:__main__:Starting SMA Webbox component
INFO:__main__:UDP protocol created
ERROR:asyncio:Exception in callback _SelectorDatagramTransport._read_ready()
handle: <Handle _SelectorDatagramTransport._read_ready()>
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.10/asyncio/selector_events.py", line 1035, in _read_ready self._protocol.datagram_received(data, addr)
  File "/root/config/custom_components/sma_webbox/sma_webbox.py", line 140, in datagram_received 
    data = json.loads(data.decode().replace("\0", ""))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 572: invalid start byte 
WARNING:__main__:RPC request timed out
WARNING:__main__:RPC request timed out
INFO:__main__:-- Overview (Thu Feb 23 09:37:11 2023)
INFO:__main__:meta name value unit
INFO:__main__:GriPwr GriPwr 495 W
INFO:__main__:GriEgyTdy GriEgyTdy 0.246 kWh
......(More data, and repeats itself).....

The byte that is seems to crash on is 0xb0 which is a "°", The "SENS0500:xxxx" unit has two "unit":"°C" in it when asking for "GetProcessData", Is this helpful to you?

If we can talk somewhere I can share some more logs and data if you want?

jellewie commented 1 year ago

Even better news! I found it! kinda...

If I change the unit from "°C" to "K" (yes bodgy solution,,)

SMA Webbox does work and detects the devices properly!!

jpcornil-git commented 1 year ago

Great to read this/finding ... I should have check possible unicode mapping for 0xb0 rather than believe it was garbage ...

"Terminal & SSH' addon gives you access to the supervisor container indeed not the homeassistant one (I'm using a custom docker install running on x86/T2M2 wall screen and ssh directly into it to access HA container); it can be used to access homeassistant configuration directory/edit related files instead of using the file editor addon (in /root/config -> /root/config/custom_components/sma_webbox here)

Given that you are able to run/modify the sma_webbox.py file, we should be able to investigate/validate a possible fix rather quickly. I'll have a look at/more on this tonight.

Great anyway that everything else works apparently !

Cheers, jpc

jpcornil-git commented 1 year ago

Could you already check the returned Content-Type in the response header (text/html;charset=utf-8 ?) May add a print(data) just before L140 to identify the problematic response as well (and check header again as above) Add -d when you run sma_webbox.py to display exchanged messages

jellewie commented 1 year ago

You mean these?

HEADERS

Server: Sunny WebBox
Cache-Control: no-store, no-cache, max-age=0
Date: Thu, 23 Feb 2023 09:48:54 GMT
Pragma: no-cache
Connection: keep-alive
Keep-Alive: 300
Content-Type: text/html
Content-Length: 7502

COMPLETE REQUEST HEADERS

___internal-request-id: 60ee8744-b33b-4745-a99f-c69f16340657
User-Agent: blablabla
content-type: application/json
Accept: */*
Origin: chrome-extension://aejoelaoggembcahagimdiliamlcdmfm
Accept-Encoding: gzip, deflate
Accept-Language: nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7

this is one from "http://192.168.1.yy/rpc" with the application/json data: RPC={"version":"1.0","proc":"GetProcessData","id":"1","format":"JSON","params":{"devices":[{"key":"SENS0500:xxxx","channels":null},{"key":"WR7KTL05:200023xxxx","channels":null}]}}

a piece of the Json corruption is {"unit":"°C","meta":"TmpMdul C","name":"TmpMdul C","value":"8.83"}

==============================

To confirm, the plugin works completely when the SMA module is configured to work only in "K" instead of "°C". Thus only support for that "°C" would need to be added. I agree this sounds simple, but I have no clue how to achieve this

jpcornil-git commented 1 year ago

Yes, thanks. Content-Type in my header is "text/html;charset=utf-8" I'll look at this tonight Cheers, jpc PS : You can format sensor data to remove, e.g. digits after the comma and rename them (I use the key name in the json response as default)

jpcornil-git commented 1 year ago

Could you change line 140 of sma_webbox.py from:

            data = json.loads(data.decode().replace("\0", ""))

to:

            data = json.loads(data.decode('iso-8859-1').replace("\0", ""))

and check again/report results ?

jellewie commented 1 year ago

That seems to work yes

jpcornil-git commented 1 year ago

Thanks,

I'll also have to adapt the code to support new sensor attributes; I don't have sensors for temperature, windspeed, illuminance, ... from my webbox.

Supported units are here sensor.py L37 and attributes set here sensor.py L140 but much more exists cfr. HA const.py

=> This will give you the right default icon/formatting/handling within HA for all of these

Can you handle that and submit a PR or do you prefer that I address this ? (is the list above complete then)

Cheers, jpc

PS: Do you have also a smart meter/are you using the P1 Monitor integration and HA Energy Management ?

jellewie commented 1 year ago

The values are working on the dashboard, and indeed most are missing there "Units", I do not know how to adjust your code to make them work. Do you require more data for this to implement it?

Tried to make the PR request you asked, hope that went well.

PS: P1 is integrated in ESPhome with this

jpcornil-git commented 1 year ago

I saw your PR/it went well :-)

I'll create a WEBBOX-G1 branch and check it in there, then add support for additional sensors/units and ask you to test (as I can't do myself). Once OK for you and existing webbox, I'll merge it back to master and release a new version

Could you run sma_webbox.py script again and attach the output log as a file rather than en embedded image ? (easier/less error-prone to identify missing sensors/exact syntax)

Cheers, jpc

jellewie commented 1 year ago

Part of set log:

INFO:__main__:Starting SMA Webbox component                                                                                                                                                                                         
INFO:__main__:UDP protocol created                                                                                                                                                                                                  
INFO:__main__:-- Overview (Fri Feb 24 13:01:11 2023)                                                                                                                                                                                
INFO:__main__:meta      name                          value     unit                                                                                                                                                                
INFO:__main__:GriPwr    GriPwr                        0000      W                                                                                                                                                                   
INFO:__main__:GriEgyTdy GriEgyTdy                     00.000    kWh                                                                                                                                                                 
INFO:__main__:GriEgyTot GriEgyTot                     000000.000kWh                                                                                                                                                                 
INFO:__main__:OpStt     OpStt                                                                                                                                                                                                       
INFO:__main__:Msg       Msg                                                                                                                                                                                                         
INFO:__main__:Devices                                                                                                                                                                                                               
INFO:__main__: Key:SENS0501:xxxx, Name:SENS0501:xxxx                                                                                                                                                                             
INFO:__main__:  ExlSolIrr ExlSolIrr                     0         W/m^2                                                                                                                                                             
INFO:__main__:  IntSolIrr IntSolIrr                     000       W/m^2                                                                                                                                                             
INFO:__main__:  OpTm      OpTm                          00000.0000000000h                                                                                                                                                          
INFO:__main__:  TmpAmb K  TmpAmb K                      0.00      K                                                                                                                                                                 
INFO:__main__:  TmpMdul K TmpMdul K                     000.00    K                                                                                                                                                                 
INFO:__main__:  WindVel m/sWindVel m/s                   0.0       m/s                                                                                                                                                              
INFO:__main__:Devices                                                                                                                                                                                                               
INFO:__main__: Key:WR6KTL05:xxxxxxxxxx, Name:WR6KTL05:xxxxxxxxxx                                                                                                                                                                    
INFO:__main__:  Balancer  Balancer                      Off                                                                                                                                                                         
INFO:__main__:  E-Total   E-Total                       00000.000 kWh                                                                                                                                                               
INFO:__main__:  Fac       Fac                           00.000    Hz                                                                                                                                                                
INFO:__main__:  Fehler    Fehler                        -------                                                                                                                                                                     
INFO:__main__:  h-On      h-On                          00000.0000000000h                                                                                                                                                           
INFO:__main__:  h-Total   h-Total                       00000.0000000000h                                                                                                                                                           
INFO:__main__:  Iac-Ist   Iac-Ist                       0.000     A                                                                                                                                                                 
INFO:__main__:  Ipv       Ipv                           0.000     A                                                                                                                                                                 
INFO:__main__:  Netz-Ein  Netz-Ein                      00000                                                                                                                                                                       
INFO:__main__:  Pac       Pac                           0000      W                                                                                                                                                                 
INFO:__main__:  Phase     Phase                         -------                                                                                                                                                                     
INFO:__main__:  Riso      Riso                          0000      kOhm                                                                                                                                                              
INFO:__main__:  SeriennummerSeriennummer                  0000000000                                                                                                                                                                
INFO:__main__:  Status    Status                        Mpp                                                                                                                                                                         
INFO:__main__:  Uac       Uac                           000.0     V                                                                                                                                                                 
INFO:__main__:  Upv-Ist   Upv-Ist                       000       V                                                                                                                                                                 
INFO:__main__:  Upv-Soll  Upv-Soll                      000       V                                                                                                                                                                 
INFO:__main__:  Zac       Zac                           0.000     Ohm

Let me know (and how specifically?) I can can try it. I will be out on vacation after today so don't hurry

jpcornil-git commented 1 year ago

I just checked tentative changes to support new WEBBOX-G1 sensors and they doesn't seem to break my setup up to now :-) => HACS -> SMA Sunny Webbox -> three dots -> Reload -> Enable Beta versions and pick main

Waiting for your test results once you are back from vacation !

jellewie commented 1 year ago

It created another Device, where the °C are logged in, but I guess thats fine in the long run.

All the transformers still work, so that's an improvement indeed!

Also here is a overview example of the new values:

jpcornil-git commented 1 year ago

Thanks for the feedback.

Icons (default ones I suppose) & values look good except for TmpAmbC; it looks like 0 is reported but somehow interpreted as °K and converted in °C ... Could you check the unit_of_measurement property [Developper tools -> States, use sma as filter) of that one or may be for all sma entities and see if you see inconsitencies.

Note that unit of measurement is set during initial integration setup, i.e. if you change webbox settings it won't be reflected in HA

jpcornil-git commented 1 year ago

... what version of home assistant are you using (I would expect lx and m/s rather than illuminance and wind_speed respectively when supported) ?

jellewie commented 1 year ago

TmpAmbC is raw at 0.03K thus -273.17C, I suppose this is not connected or broken, I wouldn't worry about it.

Running the one before the latest version or something, but it's set to be Dutch language.

jpcornil-git commented 1 year ago

OK that makes sense in that case (HA is always converting units to your configured region and therefore 0 K is correctly converted to -273 C). You may want to change the webbox setting to report in °C iso °K and remove/add this integration (and check that °C is now correctly handled)

Could you also check wind speed and illuminance sensors properties (I used illuminance iso irradiance for the latter/likely the issue but before fixing that/make a new release I would like to understand the unexpected windspeed unit/display)

jellewie commented 1 year ago

I gave it a look we run "2023.3.1", The External box is completely fogged up with water so that's probably where those weird outside numbers come from. Also the windmeter needs some maintenance and its stuck. So basically all outside numbers of this system are physically broken, but this software seems fine at least.

I have changed the box to "C" already in the last step, and that is still running fine now with the main branch.

you also asked for this

jpcornil-git commented 1 year ago

Thanks => could you check the last version I just checked in on main , it should address the "unit of measurement" issue for irradiance (added recently, cfr. https://github.com/home-assistant/core/pull/83452/files) and "wind speed" as well I believe.

Once confirmed OK, I will make a new tag/release.

jellewie commented 1 year ago

I tried looking into the unit, it was soaked in water. But sadly the RS485 module died, and I can not get any data from it anymore (not even invalid data like before). So I sadly can not confirm what you where asking.

jpcornil-git commented 1 year ago

Oh bad luck ... I'll review all changes again and release as-is then (changes are running OK against my device)

close this case. Once repaired (may be just let it dry first and check again), give me a ping to confirm that it is running OK (or not :-) )

jpcornil-git / HA-sma

ERROR - RPC request time out (no "°C" unit support?) #1

Scrape: