search

mdeweerd / MetersToHA

Get Meter Data into Home Automation systems
GNU General Public License v3.0
26 stars 8 forks source link

entités de sensor qui disparaissent #14

Closed leen67 closed 9 months ago

leen67 commented 9 months ago

Après les mises à jour de HA Core et après le redémarrage, les entités sensor.grdf_xxxxxxx_xxx disparaissent et ne reviennent pas après avoir récupéré de nouvelles données depuis GRDF. J'utilise l'Add-on MeterstoHa, pas AppDaemon, HomeAssistant OS. l'option "Force Update" n'est pas disponible dans le configuration de Add-on. Le redémarrage du Add-on n'aide pas. Avez-vous des suggestions?

m2h_exec.log: python3 MetersToHA/apps/meters_to_ha/meters_to_ha.py -l /config/meterstoha --screenshot --keep-output -c "/m2h_config.json" --grdf -r Traceback (most recent call last): File "//MetersToHA/apps/meters_to_ha/meters_to_ha.py", line 3302, in doWork injector.update_grdf_device(gazpar_file) File "//MetersToHA/apps/meters_to_ha/meters_to_ha.py", line 2631, in update_grdf_device current_total_kWh += row_meter_kWh_day TypeError: can only concatenate str (not "int") to str

service.log: 2023-11-28 08:03:00,931 : OK : Using /usr/bin/python3 Version 3.11.6Loading configuration file : /m2h_config.json 2023-11-28 08:03:00,934 : -- : Start loading configuration 2023-11-28 08:03:00,936 : OK : "veolia" = "False" 2023-11-28 08:03:00,937 : OK : "veolia_login" = "None" 2023-11-28 08:03:00,939 : OK : "veolia_password" = "None" 2023-11-28 08:03:00,941 : OK : "veolia_contract" = "None" 2023-11-28 08:03:00,942 : OK : "grdf" = "True" 2023-11-28 08:03:00,943 : OK : "grdf_login" = xxxxxl@gmail.com" 2023-11-28 08:03:00,943 : OK : "grdf_password" = "xxxxxxxx" 2023-11-28 08:03:00,944 : OK : "grdf_pce" = "xxxxxxxxx" 2023-11-28 08:03:00,944 : OK : "screenshot" = "True" 2023-11-28 08:03:00,945 : OK : "skip_download" = "False" 2023-11-28 08:03:00,945 : OK : "keep_output" = "True" 2023-11-28 08:03:00,946 : WW : "geckodriver" not found in config file, using default value 2023-11-28 08:03:00,946 : OK : "geckodriver" = "/MetersToHA/apps/meters_to_ha/geckodriver" 2023-11-28 08:03:00,947 : WW : "firefox" not found in config file, using default value 2023-11-28 08:03:00,947 : OK : "firefox" = "/MetersToHA/apps/meters_to_ha/firefox" 2023-11-28 08:03:00,948 : WW : "chromium" not found in config file, using default value 2023-11-28 08:03:00,948 : OK : "chromium" = "/usr/bin/chromium" 2023-11-28 08:03:00,949 : WW : "chromedriver" not found in config file, using default value 2023-11-28 08:03:00,950 : OK : "chromedriver" = "/usr/bin/chromedriver" 2023-11-28 08:03:00,950 : OK : "chrome_version" = "None" 2023-11-28 08:03:00,951 : OK : "timeout" = "30" 2023-11-28 08:03:00,951 : OK : "download_folder" = "/config/meterstoha/" 2023-11-28 08:03:00,952 : OK : "logs_folder" = "/config/meterstoha/" 2023-11-28 08:03:00,952 : OK : "2captcha_token" = "None" 2023-11-28 08:03:00,953 : OK : "capmonster_token" = "xxxxxxxxx" 2023-11-28 08:03:00,953 : OK : "captchaai_token" = "None" 2023-11-28 08:03:00,954 : OK : "log_level" = "debug" 2023-11-28 08:03:00,954 : OK : End loading configuration 2023-11-28 08:03:00,955 : -- : Start Loading Home Assistant configuration 2023-11-28 08:03:00,955 : OK : "ha_server" = "http://xxxxxxx:8123" 2023-11-28 08:03:00,956 : OK : "ha_token" = "***" 2023-11-28 08:03:00,957 : OK : "veolia_contract" = "None" 2023-11-28 08:03:00,957 : OK : "timeout" = "30" 2023-11-28 08:03:00,958 : OK : "insecure" = "False" 2023-11-28 08:03:00,958 : OK : "state_file" = "/config/meterstoha/meters2ha_state.json" 2023-11-28 08:03:00,959 : OK : End loading Home Assistant configuration 2023-11-28 08:03:00,959 : ~~ : Check availability of "geckodriver"+"firefox" or "chromedriver"+"chromium" 2023-11-28 08:03:00,960 : OK : Found chromium binary 2023-11-28 08:03:00,988 : OK : Check Home Assistant connectivity 2023-11-28 08:03:00,989 : -- : Try starting Chromium. Add nix root user options. 2023-11-28 08:03:01,222 : OK : Start virtual display (Chromium). 2023-11-28 08:03:01,223 : ~~ : LOG LEVEL debug:10:20 ['--verbose'] 2023-11-28 08:03:02,793 : OK : Start the browser 2023-11-28 08:03:02,794 : OK :
2023-11-28 08:03:11,910 : OK : Waiting for cookie popup 2023-11-28 08:03:12,498 : OK : Click on deny 2023-11-28 08:03:13,412 : OK : Connexion au site GRDF 2023-11-28 08:03:14,506 : OK : Waiting for Password 2023-11-28 08:03:14,687 : OK : Waiting for Email 2023-11-28 08:03:19,445 : OK : Type Email 2023-11-28 08:03:19,856 : OK : Type Password 2023-11-28 08:03:20,906 : ~~ : Proceed with captcha resolution. Wait 10s for capmonster 2023-11-28 08:03:31,818 : ~~ : Wait 10s for capmonster 2023-11-28 08:03:42,712 : ~~ : Wait 10s for capmonster 2023-11-28 08:03:53,617 : ~~ : Wait 10s for capmonster 2023-11-28 08:04:04,511 : ~~ : Wait 10s for capmonster 2023-11-28 08:04:15,416 : ~~ : Wait 10s for capmonster 2023-11-28 08:04:28,742 : OK : Automatic resolution succeeded. Grab & Save '/config/meterstoha/screen_before_connection.png'. Wait for Button xpath://input[@value='Connexion']. 2023-11-28 08:04:28,743 : ~~ : Wait before clicking (1.6s). 2023-11-28 08:04:30,977 : OK : Click on connexion 2023-11-28 08:04:39,276 : -- : End of wait after connexion. Get Data URL https://monespace.grdf.fr/api/e-conso/pce/consommation/informatives?dateDebut=2023-11-14&dateFin=2023-11-28&pceList[]=xxxxxxxxxxxGet Data Content. Writing '/config/meterstoha/historique_gazpar.json'. Parsing JSON file. 2023-11-28 08:04:39,290 : -- : From sensor.grdf_xxxxxxxxxxx_kwh: None 2023-11-28 08:04:39,301 : -- : From sensor.gas_consumption_kwh: None 2023-11-28 08:04:39,323 : ~~ : grdf_state: {'entity_id': 'sensor.grdf_xxxxxxxxx_kwh', 'state': '1662.0', 'attributes': {'date_time': '2023-11-24T06:00:00+01:00', 'unit_of_measurement': 'kWh', 'device_class': 'energy', 'state_class': 'total_increasing', 'last_check': '2023-11-25T14:51:05.727037+00:00'}, 'last_changed': '2023-11-25T14:51:05.836121+00:00', 'last_updated': '2023-11-25T14:51:05.836121+00:00', 'context': {'id': '01HG3FCYQCH46A4388VETPYJ8E', 'parent_id': None, 'user_id': '3f052a06225d4d8abc67e980317b8c82'}, 'm3': 12588.0} 2023-11-28 08:04:39,324 : -- : Previous 12588.0 m3 1662.0 kWh 2023-11-24 06:00:00+01:00 from None 2023-11-28 08:04:39,324 : EE : can only concatenate str (not "int") to str 2023-11-28 08:04:39,438 : OK : Close Browser 2023-11-28 08:04:39,448 : OK : Close Display 2023-11-28 08:04:39,449 : EE : Ended with error : // re-run the program with '--debug' option

Log dans l'Add-on: s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service base-addon-banner: starting

Add-on: MetersToHA MetersToHA add-on for Home Assistant

Add-on version: dev You are running the latest version of this add-on. System: Home Assistant OS 11.1 (aarch64 / raspberrypi4-64) Home Assistant Core: 2023.11.3 Home Assistant Supervisor: 2023.11.3

Please, share the above information when looking for help or support in, e.g., GitHub, forums or the Discord chat.

s6-rc: info: service base-addon-banner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service base-addon-log-level: starting s6-rc: info: service fix-attrs successfully started Log level is set to DEBUG s6-rc: info: service base-addon-log-level successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service legacy-services: starting s6-rc: info: service legacy-services successfully started Cloning into 'MetersToHA'... git checkout Your branch is up to date with 'origin/meters-to-ha'. MetersToHA Container version: dev.015 #ac6444fe6ea3619664657995d7eebd23 /run.sh MetersToHA Python GIT version: 6a109f9 on Sat Nov 18 02:41:48 2023 +0100 Generate configuration file Generated configuration file '/m2h_config.json': { "capmonster_token":"2ffffbdf4cd9d08903d93768d1dccfb4", "log_level":"debug", "logs_folder":"/config/meterstoha", "grdf_login":"xxxxxxx@gmail.com", "grdf_password":"xxxxxxxxx", "grdf_pce":"xxxxxxxxxxx", "timeout":"30", "download_folder":"/config/meterstoha", "ha_server": "http://raspberrypi:8123", "ha_token": "***", "type": "ha" } DISPLAY:'' EVENT CONF:grdf:call_grdf === Generated script '/execEvent.sh': ==========

!/bin/bash

{ TARGET_OPT="" [[ "$1" == "call_veolia" ]] && TARGET_OPT=--veolia [[ "$1" == "call_grdf" ]] && TARGET_OPT=--grdf

[[ "$TARGET_OPT" == "" ]] && ( echo "Unrecognized event '$1'" ; exit 1 ) date echo "python3 MetersToHA/apps/meters_to_ha/meters_to_ha.py -l /config/meterstoha --screenshot --keep-output -c \"/m2h_config.json\" $TARGET_OPT -r" python3 MetersToHA/apps/meters_to_ha/meters_to_ha.py -l /config/meterstoha --screenshot --keep-output -c "/m2h_config.json" $TARGET_OPT -r

Copy chrome logs

for i in ~/.config/*/chrome_debug.log ; do [[ -r "$i" ]] || continue SUBDIR="/config/meterstoha/$(basename "$(dirname "$i")")" mkdir -p "${SUBDIR}" cp -p "$i" "${SUBDIR}" done echo "Done $(date)" } >> "/config/meterstoha/m2h_exec.log" 2>&1 === End of Generated script '/execEvent.sh': ===

Test access to Home Assistant API (should show '{"message":"API running."}') curl -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiIxYjk2NDk5M2U1OTM0NjYzOWZlZjhhMGM4MjFmYzI4NiIsImlhdCI6MTY5OTkwMDMzOSwiZXhwIjoyMDE1MjYwMzM5fQ.zpNmVF78svfh79Nnpekxjchb0bS6xzxkpPtWr81d9VU' -H 'Content-Type: application/json' http://xxxxxxxxxxx:8123/api/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 26 100 26 0 0 2018 0 --:--:-- --:--:-- --:--:-- 2166 {"message":"API running."} "./haevent2exec.py" --config-json "/m2h_config.json" --external-program "/execEvent.sh" --log-level="debug" call_veolia call_grdf

leen67 commented 9 months ago

OK, I think I may have found the issue in the code. meters_to_ha.py line 2590 and 2594: previous_kWh = grdf_state["state"] previous_m3 = grdf_state["m3"] Since it is reading from the meters2ha_state.json file, these statements return a string, resulting in the concatenation error on line 2631. Changing these lines to: previous_kWh = float(grdf_state["state"]) previous_m3 = float(grdf_state["m3"]) Then it works again, and re-creates the updated entities in HA.

Hopefully this can be corrected in the source of a next release, for now I just edited it inside the docker container.

mdeweerd commented 9 months ago

Thank you for the feedback - I'll check this out - I'ld expect the JSON to have a numeric field and python to return it as a numeric value, but I'll check what is happening.

leen67 commented 9 months ago

Looking at a previous copy of meters2ha_state.json, it looks like this: { "grdf": { "entity_id": "sensor.grdf_19396092515061_kwh", "state": "1662.0", "attributes": { "date_time": "2023-11-24T06:00:00+01:00", "unit_of_measurement": "kWh", "device_class": "energy", "state_class": "total_increasing", "last_check": "2023-11-25T14:51:05.727037+00:00" }, "last_changed": "2023-11-25T14:51:05.836121+00:00", "last_updated": "2023-11-25T14:51:05.836121+00:00", "context": { "id": "01HG3FCYQCH46A4388VETPYJ8E", "parent_id": null, "user_id": "3f052a06225d4d8abc67e980317b8c82" }, "m3": 12588.0 } } With the state value between quotes. Then gets read as a string I suppose. The current file (written after my mod in the container) doesn't have these quotes: { "grdf": { "state": 890.0, "attributes": { "date_time": "2023-11-29T06:00:00+01:00", "unit_of_measurement": "kWh", "device_class": "energy", "state_class": "total_increasing", "last_check": "2023-11-30T19:41:41.873830+00:00" }, "m3": 12651 } }

Could it be that the meters2ha_state.json file sometimes gets written with the state value as a string instead of a number? I guess next time this happens I can just edit the JSON file and remove the quotes, it's a lot simpler workaround :-)

mdeweerd commented 9 months ago

The workaround to change the json file is easier, but it's not a permanent solution ;-).

I think that the string value is introduced when the previosu state is imported from home assistant. I've followed your suggestion and converted kWh to float when reading the data from home assistant as well.

The code could also be updated to listen to the homeassistant_started event and restore the entities from the json file at that time - and also restore the data from the json file when the addon restarts and the ha entity is not found or has no valid values.

vingerha commented 6 months ago

I have a similar behaviour right now, when running via Alpine for the first time, the 3 entities are created. When I restart HA, the entities disappear. I can restart the docker-compose of course but this does not do anything until new data flows in...which is tomorrow (or when I delete the json). I am not sure if a solution via MQTT is more stable?

EDIT, a not so nice side-effect of restarting/recreating is that I now have re-created the sensor with values of 2 days ago so the statistics get even further incorrect.

EDIT2: will try myself via MQTT in the next few days