web scraping of the data only works partially

[mathiaslehmann]

Hello, I use the plugin in both mode. Since yesterday there are some problems with querying the data from the website. Apparently things have changed or some are not coming back at all. Could you please check if there have been any changes? Thanks very much.

[mathiaslehmann]

I looked up an example.

"sensor.hydraulik_pufferspeicher_temperatur_oben_0_pufferspeicher_temperatur_unten_aktuell" has become a new sensor. In reality there is a sensor at the top and a sensor at the bottom. "sensor.hydraulik_pufferspeicher_temperatur_oben_0_pufferspeicher_temperatur_oben_aktuell" and "sensor.hydraulik_pufferspeicher_temperatur_unten_0_pufferspeicher_temperatur_unten_aktuell"

[erikkastelec]

Hi,

have you tried reloading the integration and can you provide some logs of errors/warnings you are getting. It would be best to turn on the debug logging for the integration (instructions on how to do this are in the README)

It is possible that they change the website in some way, but the integration works fine for me and I can't see any changes for my unit.

So if I understand you correctly, some sensors went missing and new ones were created, but everything else works as expected?

[mathiaslehmann]

Hi, i first reloaded the integration and later even deleted it and set it up again. Most sensors are available, but the sensors listed below are missing. There may be more, but I didn't use and see all of them. There are some new sensors for this, but not for all the missing ones. Otherwise everything works without problems. The problem seems to have existed since yesterday morning. At first, when I discovered it, all the sensors were not available, so I reloaded the integration, after which I had the known problem. I'll try to collect a piece of log for you.

• sensor.warmwasser_warmwasser_1_status
• sensor.warmwasser_warmwasser_1_warmwassersolltemperatur_aktuell
• sensor.hydraulik_pufferspeicher_temperatur_oben_0_pufferspeicher_temperatur_oben_aktuell
• sensor.hydraulik_pufferspeicher_temperatur_unten_0_pufferspeicher_temperatur_unten_aktuell
• sensor.heizkreis_heizkreise_modtxt_heizkreis_2_status_hzk1
• sensor.heizkreis_heizkreise_modtxt_heizkreis_2_betriebsart_hzk1
• sensor.heizkreis_heizkreise_modtxt_heizkreis_2_pumpe_heizkreis_hzk1
• sensor.heizkreis_heizkreise_modtxt_heizkreis_2_vorlaufsolltemperatur_aktuell_hzk1
• sensor.heizkreis_heizkreise_modtxt_heizkreis_2_vorlauftemperatur_aktuell_hzk1
• sensor.heizkreis_heizkreise_modtxt_heizkreis_2_mischerstellung_ist_aktuell_hzk1
• sensor.heizkreis_heizkreise_modtxt_heizkreis_2_mischerstellung_soll_aktuell_hzk1
• sensor.heizkreis_heizkreise_modtxt_heizkreis_2_aussentemperatur_lokal_gemischt_hzk1

[mathiaslehmann]

Hi, I tried to get some log out. Unfortunately I can only find the following. If I understand correctly, this is only the query of the API. There is nothing in the LOG about web scraping.

2023-05-31 22:20:16.227 DEBUG (SyncWorker_15) [custom_components.wemportal] Fetching fresh api data 2023-05-31 22:20:16.227 DEBUG (SyncWorker_15) [custom_components.wemportal] Sending POST request to https://www.wemportal.com/app/DataAccess/Refresh with headers: {'User-Agent': 'WeishauptWEMApp', 'X-Api-Version': '2.0.0.0', 'Accept': '*/*', 'Content-Type': 'application/json'} and data: {'DeviceID': 13233, 'Modules': [{'ModuleIndex': 0, 'ModuleType': 1, 'Parameters': [{'ParameterID': 'Außentemperatur'}]}, {'ModuleIndex': 0, 'ModuleType': 2, 'Parameters': [{'ParameterID': 'Raumtemperatur'}, {'ParameterID': 'Betriebsart'}, {'ParameterID': 'Heizprogramm1'}, {'ParameterID': 'Heizprogramm2'}, {'ParameterID': 'Heizprogramm3'}, {'ParameterID': 'PP_Funktion'}, {'ParameterID': 'PP_Raumsoll'}, {'ParameterID': 'PP_Beginn'}, {'ParameterID': 'PP_Ende'}, {'ParameterID': 'Komfort'}, {'ParameterID': 'Normal'}, {'ParameterID': 'Absenk'}, {'ParameterID': 'U_Funktion'}, {'ParameterID': 'U_Raumsoll'}, {'ParameterID': 'U_Beginn'}, {'ParameterID': 'U_Ende'}, {'ParameterID': 'AktRaumSoll'}]}, {'ModuleIndex': 0, 'ModuleType': 3, 'Parameters': [{'ParameterID': 'Warmwassertemperatur'}, {'ParameterID': 'NormalWW'}, {'ParameterID': 'AbsenkWW'}, {'ParameterID': 'WW-Push'}, {'ParameterID': 'WW-Programm'}, {'ParameterID': 'Zirkulationsprogramm'}]}, {'ModuleIndex': 0, 'ModuleType': 7, 'Parameters': [{'ParameterID': 'Leistung'}, {'ParameterID': 'Vorlauftemperatur'}, {'ParameterID': 'Anlagendruck'}, {'ParameterID': 'WW-Solltemp'}]}, {'ModuleIndex': 1, 'ModuleType': 2, 'Parameters': [{'ParameterID': 'Raumtemperatur'}, {'ParameterID': 'Betriebsart'}, {'ParameterID': 'Heizprogramm1'}, {'ParameterID': 'Heizprogramm2'}, {'ParameterID': 'Heizprogramm3'}, {'ParameterID': 'PP_Funktion'}, {'ParameterID': 'PP_Raumsoll'}, {'ParameterID': 'PP_Beginn'}, {'ParameterID': 'PP_Ende'}, {'ParameterID': 'Komfort'}, {'ParameterID': 'Normal'}, {'ParameterID': 'Absenk'}, {'ParameterID': 'U_Funktion'}, {'ParameterID': 'U_Raumsoll'}, {'ParameterID': 'U_Beginn'}, {'ParameterID': 'U_Ende'}, {'ParameterID': 'AktRaumSoll'}]}, {'ModuleIndex': 0, 'ModuleType': 4, 'Parameters': [{'ParameterID': 'Kollektortemperatur'}]}]} 2023-05-31 22:20:17.642 DEBUG (SyncWorker_15) [custom_components.wemportal] <Response [200]> 2023-05-31 22:20:17.642 DEBUG (SyncWorker_15) [custom_components.wemportal] Sending POST request to https://www.wemportal.com/app/DataAccess/Read with headers: {'User-Agent': 'WeishauptWEMApp', 'X-Api-Version': '2.0.0.0', 'Accept': '*/*', 'Content-Type': 'application/json'} and data: {'DeviceID': 13233, 'Modules': [{'ModuleIndex': 0, 'ModuleType': 1, 'Parameters': [{'ParameterID': 'Außentemperatur'}]}, {'ModuleIndex': 0, 'ModuleType': 2, 'Parameters': [{'ParameterID': 'Raumtemperatur'}, {'ParameterID': 'Betriebsart'}, {'ParameterID': 'Heizprogramm1'}, {'ParameterID': 'Heizprogramm2'}, {'ParameterID': 'Heizprogramm3'}, {'ParameterID': 'PP_Funktion'}, {'ParameterID': 'PP_Raumsoll'}, {'ParameterID': 'PP_Beginn'}, {'ParameterID': 'PP_Ende'}, {'ParameterID': 'Komfort'}, {'ParameterID': 'Normal'}, {'ParameterID': 'Absenk'}, {'ParameterID': 'U_Funktion'}, {'ParameterID': 'U_Raumsoll'}, {'ParameterID': 'U_Beginn'}, {'ParameterID': 'U_Ende'}, {'ParameterID': 'AktRaumSoll'}]}, {'ModuleIndex': 0, 'ModuleType': 3, 'Parameters': [{'ParameterID': 'Warmwassertemperatur'}, {'ParameterID': 'NormalWW'}, {'ParameterID': 'AbsenkWW'}, {'ParameterID': 'WW-Push'}, {'ParameterID': 'WW-Programm'}, {'ParameterID': 'Zirkulationsprogramm'}]}, {'ModuleIndex': 0, 'ModuleType': 7, 'Parameters': [{'ParameterID': 'Leistung'}, {'ParameterID': 'Vorlauftemperatur'}, {'ParameterID': 'Anlagendruck'}, {'ParameterID': 'WW-Solltemp'}]}, {'ModuleIndex': 1, 'ModuleType': 2, 'Parameters': [{'ParameterID': 'Raumtemperatur'}, {'ParameterID': 'Betriebsart'}, {'ParameterID': 'Heizprogramm1'}, {'ParameterID': 'Heizprogramm2'}, {'ParameterID': 'Heizprogramm3'}, {'ParameterID': 'PP_Funktion'}, {'ParameterID': 'PP_Raumsoll'}, {'ParameterID': 'PP_Beginn'}, {'ParameterID': 'PP_Ende'}, {'ParameterID': 'Komfort'}, {'ParameterID': 'Normal'}, {'ParameterID': 'Absenk'}, {'ParameterID': 'U_Funktion'}, {'ParameterID': 'U_Raumsoll'}, {'ParameterID': 'U_Beginn'}, {'ParameterID': 'U_Ende'}, {'ParameterID': 'AktRaumSoll'}]}, {'ModuleIndex': 0, 'ModuleType': 4, 'Parameters': [{'ParameterID': 'Kollektortemperatur'}]}]} 2023-05-31 22:20:20.569 DEBUG (SyncWorker_15) [custom_components.wemportal] <Response [200]> 2023-05-31 22:20:20.571 DEBUG (MainThread) [custom_components.wemportal] Finished fetching WemPortal update data in 24.419 seconds (success: True)

[erikkastelec]

I would first recommend checking your expert page on the Wemportal website to see if there have been any changes or removals to the sensors. If such changes have occurred, regrettably, there is nothing I can do.

You can do this by selecting the dropdown menu located in the upper left corner labeled "Heating System/Anlagen" and then clicking on "Expert/Fachmann".

If the sensor data is intact, you can save the webpage by right-clicking anywhere on it and selecting "Save Page As" or a similar option.

Next, upload the saved HTML file here or email it to me at erikkastelec@gmail.com. It's worth noting that the HTML file might contain some of your personal information. Once I receive the file, I will inspect the issue and do my best to rectify it.

[mathiaslehmann]

I tried to take some screenshots of the website. I think the data comes from the viewpoint "Fachmann > Info". The problem area is "Hydraulik - Pufferspeicher Temperatur oben 0" and then continues with "Hydraulik - Pufferspeicher Temperatur unten 0" then "Heizkreis/+/Heizkreise - /-modtxt/-Heizkreis 2" and then "Warmwasser - Warmwasser 1" . The value for "Warmwassersolltemperatur Aktuell" under "Warmwasser - Warmwasser 1" is now called "sensor.heizkreis_heizkreise_modtxt_heizkreis_2_warmwassersolltemperatur_aktuell" in Home Assistant, for example, or the value "Warmwassertemperatur Aktuell" under "Warmwasser - Warmwasser 1" is now called "sensor.heizkreis_heizkreise_modtxt_heizkreis_2_warmwassertemperatur_aktuell".

[mathiaslehmann]

Hi, Sensor data are all available. I sent you an email with the html. Many thanks for the help.

[erikkastelec]

New release is ready for you to test

[mathiaslehmann]

Hi, it's working again. Thank you for the quick solution.