search

ondrejvysek / HomeAssistant-CEZDistribuce-PND

11 stars 2 forks source link

CSV jsou stazeny s nevalidnimi daty (obrazek) #43

Open ondrejvysek opened 2 weeks ago

ondrejvysek commented 2 weeks ago

PND se dostalo znatelně dál, nicméně se akce nedaří stáhnout nějaké csv:

link 07 Profil spotřeby za den (+A)
link 08 Profil výroby za den (-A)

a přijde mi, že po nestažení csv už nezpracuje data nebo se zastaví? Přikládám log.

a0d7b954_appdaemon_2024-06-18T05-12-36.356Z.log

Originally posted by @jankorous in https://github.com/ondrejvysek/HomeAssistant-CEZDistribuce-PND/issues/42#issuecomment-2175051030

ondrejvysek commented 2 weeks ago

@jankorous udelam verzi s vetsim debugem verzi modulu, to nam rekne vic

jankorous commented 1 week ago

Koukal jsem, že jste dal na git novou verzi, nicméně ještě než jsem ji nahradil a i potom po její náhradě za novou verzi jsem narazil na to, že se mi nespustí run_pnd, resp. nastartuje mi AppDeamon, dostane se k ukončené inicializaci:

[07:51:32] INFO: Starting AppDaemon...
s6-rc: info: service legacy-services successfully started
2024-06-19 07:51:37.869674 INFO AppDaemon: AppDaemon Version 4.4.2 starting
2024-06-19 07:51:37.870002 INFO AppDaemon: Python version is 3.11.9
2024-06-19 07:51:37.870184 INFO AppDaemon: Configuration read from: /config/appdaemon.yaml
2024-06-19 07:51:37.870345 INFO AppDaemon: Added log: AppDaemon
2024-06-19 07:51:37.870517 INFO AppDaemon: Added log: Error
2024-06-19 07:51:37.870662 INFO AppDaemon: Added log: Access
2024-06-19 07:51:37.870808 INFO AppDaemon: Added log: Diag
2024-06-19 07:51:37.871055 INFO AppDaemon: Added log: pnd
2024-06-19 07:51:38.180184 INFO AppDaemon: Loading Plugin HASS using class HassPlugin from module hassplugin
2024-06-19 07:51:38.704798 INFO HASS: HASS Plugin Initializing
2024-06-19 07:51:38.705137 INFO HASS: HASS Plugin initialization complete
2024-06-19 07:51:38.705953 INFO AppDaemon: Initializing HTTP
2024-06-19 07:51:38.707189 INFO AppDaemon: Using 'ws' for event stream
2024-06-19 07:51:38.712269 INFO AppDaemon: Starting API
2024-06-19 07:51:38.715735 INFO AppDaemon: Starting Admin Interface
2024-06-19 07:51:38.716306 INFO AppDaemon: Starting Dashboards
2024-06-19 07:51:38.731675 INFO HASS: Connected to Home Assistant 2024.6.3
2024-06-19 07:51:38.755880 INFO AppDaemon: App 'pnd' added
2024-06-19 07:51:38.757538 INFO AppDaemon: Found 1 active apps
2024-06-19 07:51:38.758155 INFO AppDaemon: Found 0 inactive apps
2024-06-19 07:51:38.758678 INFO AppDaemon: Found 0 global libraries
2024-06-19 07:51:38.759309 INFO AppDaemon: Starting Apps with 1 workers and 1 pins
2024-06-19 07:51:38.761098 INFO AppDaemon: Running on port 5050
2024-06-19 07:51:38.787854 INFO HASS: Evaluating startup conditions
2024-06-19 07:51:38.793140 INFO HASS: Startup condition met: hass state=RUNNING
2024-06-19 07:51:38.793567 INFO HASS: All startup conditions met
2024-06-19 07:51:38.860671 INFO AppDaemon: Got initial state from namespace default
2024-06-19 07:51:40.769797 INFO AppDaemon: Scheduler running in realtime
2024-06-19 07:51:40.774072 INFO AppDaemon: Adding /homeassistant/appdaemon/apps to module import path
2024-06-19 07:51:40.774738 INFO AppDaemon: Adding /homeassistant/appdaemon/apps/pnd to module import path
2024-06-19 07:51:40.775812 INFO AppDaemon: App initialization complete

Nicméně když v automatizacích spustím run_pnd, ani po 5ti minutách se nic neděje, log mlčí. Pro další pokusy jsem vyčistil složku apps/pnd vyjma apps.yaml, abych viděl přírůstky nebo změny ve stromu filů, ale beze změn, folder zůstane prázdný a png.log taky. Pro jistotu byl i restart HA > Same result.

jankorous commented 1 week ago

Pokud jdu do debugu, točí se up-time HASS:

2024-06-19 08:14:26.816441 DEBUG AppDaemon: get_state: thread.thread-0.None None True
2024-06-19 08:14:26.817055 DEBUG HASS: Utility
2024-06-19 08:14:26.817631 DEBUG AppDaemon: set_state(): sensor.appdaemon_uptime, {'state': '0:00:41'}
2024-06-19 08:14:26.818361 DEBUG AppDaemon: parse_state: sensor.appdaemon_uptime, {'state': '0:00:41'}
2024-06-19 08:14:26.819174 DEBUG AppDaemon: Old state: {'entity_id': 'sensor.appdaemon_uptime', 'state': '0:00:40', 'last_changed': '2024-06-19T08:14:25+02:00', 'attributes': {}}
2024-06-19 08:14:26.819826 DEBUG AppDaemon: New state: {'entity_id': 'sensor.appdaemon_uptime', 'state': '0:00:41', 'last_changed': '2024-06-19T08:14:26+02:00', 'attributes': {}}
2024-06-19 08:14:26.820505 DEBUG AppDaemon: sending event locally
2024-06-19 08:14:26.821356 DEBUG AppDaemon: Util loop compute time: 8.0ms, check_config()=3.0ms, other=5.0ms
2024-06-19 08:14:26.822192 DEBUG AppDaemon: Event type:state_changed:
2024-06-19 08:14:26.822817 DEBUG AppDaemon: {'entity_id': 'sensor.appdaemon_uptime', 'new_state': {'entity_id': 'sensor.appdaemon_uptime', 'state': '0:00:41', 'last_changed': '2024-06-19T08:14:26+02:00', 'attributes': {}}, 'old_state': {'entity_id': 'sensor.appdaemon_uptime', 'state': '0:00:40', 'last_changed': '2024-06-19T08:14:25+02:00', 'attributes': {}}}
2024-06-19 08:14:26.823514 DEBUG AppDaemon: process_event_callbacks() admin {'event_type': 'state_changed', 'data': {'entity_id': 'sensor.appdaemon_uptime', 'new_state': {'entity_id': 'sensor.appdaemon_uptime', 'state': '0:00:41', 'last_changed': '2024-06-19T08:14:26+02:00', 'attributes': {}}, 'old_state': {'entity_id': 'sensor.appdaemon_uptime', 'state': '0:00:40', 'last_changed': '2024-06-19T08:14:25+02:00', 'attributes': {}}}}
2024-06-19 08:14:26.824944 DEBUG AppDaemon: thread_async loop, args={'function': <bound method ADStream.process_event of <appdaemon.stream.adstream.ADStream object at 0x7f76dc4450>>, 'args': ({'event_type': 'state_changed', 'data': {'entity_id': 'sensor.appdaemon_uptime', 'new_state': {'entity_id': 'sensor.appdaemon_uptime', 'state': '0:00:41', 'last_changed': '2024-06-19T08:14:26+02:00', 'attributes': {}}, 'old_state': {'entity_id': 'sensor.appdaemon_uptime', 'state': '0:00:40', 'last_changed': '2024-06-19T08:14:25+02:00', 'attributes': {}}}, 'namespace': 'admin'},), 'kwargs': {}}

Přihodím celý log z HA AppDeamona: appdeamon_log.txt

ondrejvysek commented 1 week ago

To vypada, ze se vubec nenacte pnd.py, je tam init metoda, ktera se spousti pri kazde aktualizaci .py souboru resp startu appdaemon a pise >>>>>>>>>> pnd init.... a ani pred tim neni import pnd.py.

Kdyz neprobehne tohle, neni zaregistrovany listener na udalost run_pnd.

Nova verze dela pouze detailnejsi log se seznamem a verzemi komponent

  1. Kde je pnd.py umisten?
  2. Co je na prvnim radku?

Prip je to nejak dostupne z netu, mohl bych kouknout

jankorous commented 1 week ago

.py soubor mám v homeassistant/appdeamon/app a aktuálně používám ver = "0.9.9"

2024-06-19 10:25:01.408433 INFO AppDaemon: Adding /homeassistant/appdaemon/apps to module import path
2024-06-19 10:25:01.412761 WARNING AppDaemon: No app description found for: /homeassistant/appdaemon/apps/pnd.py - ignoring
2024-06-19 10:25:01.415514 INFO AppDaemon: App initialization complete
ondrejvysek commented 1 week ago

Diky, tohle uz zacina davat smysl. Nenasel definici k pnd.py, tj prosim:

  1. Kde je apps.yaml
  2. Co je v apps.yaml
jankorous commented 1 week ago

apps.yaml je v /homeassistant/appdaemon/

a jeho obsahem je 

---
pnd:
  module: pnd
  class: pnd
  log: pnd
  PNDUserName: "jméno"
  PNDUserPassword: "heslo"
  DataInterval: "01.06.2023 00:00 - 31.06.2024 00:00"
  ELM: "890971"
  DownloadFolder: "/homeassistant/appdaemon/apps/pnd"
ondrejvysek commented 1 week ago

OK, pokud je appdaemon.yaml dle dokumentace, pak tam bude app_dir: /homeassistant/appdaemon/apps, pak je nutné mít apps.yaml v tom adresáři, tj /homeassistant/appdaemon/apps/apps.yaml, tím by se měl vyřešit warning z logu:

2024-06-19 10:25:01.408433 INFO AppDaemon: Adding /homeassistant/appdaemon/apps to module import path 2024-06-19 10:25:01.412761 WARNING AppDaemon: No app description found for: /homeassistant/appdaemon/apps/pnd.py - ignoring

jankorous commented 1 week ago

Yop, solved, nicméně:

2024-06-19 21:09:48: ********************* Starting 0.9.9 *********************
2024-06-19 21:09:48: ----------------------------------------------
2024-06-19 21:09:48: Hello from AppDaemon for Portal Namerenych Dat
2024-06-19 21:09:48.841971 INFO AppDaemon: Removing module /homeassistant/appdaemon/apps/pnd/pnd.py
2024-06-19 21:09:48.842956 INFO AppDaemon: Terminating pnd
2024-06-19 21:09:48.843457 INFO AppDaemon: Calling terminate() for pnd
2024-06-19 21:09:48: >>>>>>>>>>>> PND Terminate
2024-06-19 21:09:49: Driver Loaded
2024-06-19 21:09:51: Website Opened
2024-06-19 21:10:03: App Version: 2.3.5
2024-06-19 21:10:07: H1 tag with text 'Naměřená data' is present.
2024-06-19 21:10:14: Rychla Sestava selected successfully!
2024-06-19 21:10:17: Selecting ELM '890971'
2024-06-19 21:10:17: Valid ELM numbers 'ELM 890971'
2024-06-19 21:10:24: ELM Status: ELM 890971 - 890971
2024-06-19 21:10:24: Iteration 0: Vyhledat Button NOT disabled
2024-06-19 21:10:24: Device ELM '890971' selected successfully!
2024-06-19 21:10:29: Button 'Vyhledat data' clicked successfully!
2024-06-19 21:10:35: 07 Profil spotřeby za den (+A)
2024-06-19 21:10:40: ERROR: Failed to find link 07 Profil spotřeby za den (+A)
2024-06-19 21:10:43: ERROR: Failed to download CSV file for 07 Profil spotřeby za den (+A)
2024-06-19 21:10:44: File downloaded and saved as: /homeassistant/appdaemon/apps/pnd/daily-consumption.csv
2024-06-19 21:10:46: 08 Profil výroby za den (-A)
2024-06-19 21:10:52: ERROR: Failed to find link 08 Profil výroby za den (-A)
2024-06-19 21:10:55: ERROR: Failed to download CSV file for 08 Profil výroby za den (-A)
2024-06-19 21:10:56: File downloaded and saved as: /homeassistant/appdaemon/apps/pnd/daily-production.csv
2024-06-19 21:10:56: All Done - DAILY DATA DOWNLOADED
2024-06-19 21:10:56.643957 WARNING pnd: ------------------------------------------------------------
2024-06-19 21:10:56.644755 WARNING pnd: Unexpected error in worker for App pnd:
2024-06-19 21:10:56.645429 WARNING pnd: Worker Ags: {'id': 'f124e1364f8e4c44b88dc35527575d7b', 'name': 'pnd', 'objectid': '6d69ba68306a42ecb725d50b7dea98c5', 'type': 'event', 'event': 'run_pnd', 'function': <bound method pnd.run_pnd of <pnd.pnd object at 0x7fb0716350>>, 'data': {'metadata': {'origin': 'LOCAL', 'time_fired': '2024-06-19T19:09:48.806894+00:00', 'context': {'id': '01J0RYFEW4WJS8W87CATK2T3AE', 'parent_id': '01J0RYFEW385PYWZF71QEVVQQJ', 'user_id': None}}}, 'pin_app': True, 'pin_thread': 0, 'kwargs': {'__thread_id': 'thread-0'}}
2024-06-19 21:10:56.645757 WARNING pnd: ------------------------------------------------------------
2024-06-19 21:10:56.654850 WARNING pnd: Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/appdaemon/threading.py", line 1095, in worker
    funcref(args["event"], data, self.AD.events.sanitize_event_kwargs(app, args["kwargs"]))
  File "/homeassistant/appdaemon/apps/pnd.py", line 511, in run_pnd
  File "/usr/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 626, in _read
    return parser.read(nrows)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1923, in read
    ) = self._engine.read(  # type: ignore[attr-defined]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 234, in read
    chunks = self._reader.read_low_memory(nrows)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "parsers.pyx", line 838, in pandas._libs.parsers.TextReader.read_low_memory
  File "parsers.pyx", line 905, in pandas._libs.parsers.TextReader._read_rows
  File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows
  File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status
  File "parsers.pyx", line 2061, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 1 fields in line 5, saw 2

2024-06-19 21:10:56.655536 WARNING pnd: ------------------------------------------------------------
2024-06-19 21:10:56.731767 WARNING AppDaemon: Excessive time spent in callback 'run_pnd() in pnd', Thread 'thread.thread-0' - now complete after 68.731095 seconds (limit=60.0)
ondrejvysek commented 1 week ago

Super, posun. Teď prosím tu poslední verzi skriptu 0.9.9, která vypíše v initu (není potřeba pouštět automatizaci, jenom uložit .py) detaily použitých balíčků, poprosím o to. Nejspíše je tam je pravěká verze chromium, pomůže mi identifikovat. Počítám, že nyní ten csv soubor obsahuje stále obrázek.

takto nějak to vypadá, verze chromium je na konci image

jankorous commented 1 week ago

Moduly jsou tyto:

System Information:
Platform: Linux
Platform Release: 6.6.28-haos-raspi
Platform Version: #1 SMP PREEMPT Wed May  8 10:24:45 UTC 2024
Architecture: aarch64
Processor: 
Python Version: 3.11.9

Installed Python Modules:
Package            Version
------------------ ------------
aiohttp            3.8.6
aiohttp-jinja2     1.5.1
aiosignal          1.3.1
appdaemon          4.4.2
astral             3.2
async-timeout      4.0.3
attrs              23.2.0
bcrypt             4.0.1
beautifulsoup4     4.12.3
bidict             0.23.1
bs4                0.0.2
certifi            2024.2.2
charset-normalizer 3.3.2
deepdiff           6.3.0
feedparser         6.0.11
frozenlist         1.4.1
h11                0.14.0
idna               3.7
iso8601            1.1.0
Jinja2             3.1.4
MarkupSafe         2.1.5
multidict          6.0.5
numpy              1.26.4
ordered-set        4.1.0
outcome            1.3.0.post0
packaging          23.2
paho-mqtt          1.6.1
pandas             2.2.2
pid                3.0.4
pip                23.3.1
pyparsing          3.1.1
PySocks            1.7.1
python-dateutil    2.8.2
python-engineio    4.9.1
python-socketio    5.8.0
pytz               2023.3.post1
PyYAML             6.0.1
requests           2.28.2
selenium           4.21.0
setuptools         68.2.2
sgmllib3k          1.0.0
simple-websocket   1.0.0
six                1.16.0
sniffio            1.3.1
sockjs             0.11.0
sortedcontainers   2.4.0
soupsieve          2.5
tomli              2.0.1
tomli_w            1.0.0
trio               0.25.1
trio-websocket     0.11.1
typing_extensions  4.12.2
tzdata             2024.1
urllib3            1.26.18
uvloop             0.17.0
websocket-client   1.5.3
wheel              0.42.0
wsproto            1.2.0
yarl               1.9.4

ChromeDriver Version: ChromeDriver 124.0.6367.78 (a087f2dd364ddd58b9c016ef1bf563d2bc138711-refs/branch-heads/6367@{#954})

... a ano, stále jsou to obrázky :/

ondrejvysek commented 1 week ago

Hm, tohle se zdá být ok.... jdu přemýšlet

ondrejvysek commented 1 week ago

Tu je testovací skript https://github.com/ondrejvysek/HomeAssistant-CEZDistribuce-PND/blob/test/pnd.py problém to 100% neřeší, ale mělo (mohlo) by se dostat přes ty denní data (jsou to cca 1kB soubory). Podařilo se mi nějak nasimulovat, ale zdali to je to samé jako výše netuším. Dejte vědět pak log, zdali se dostalo kousek dál. Pokud ano, opravit dál by mělo být rychlé.

jankorous commented 1 week ago

Otázka - není na PND nějaký limit? Včera jsem se tomu večer už nevěnoval a v 00:30, jak je v automatizaci, proběhl skript a grafy a data naplnil bez chyby...?

Znovu jsem spustil teď ráno:

2024-06-20 07:46:01: ********************* Starting 0.9.9:test *********************
2024-06-20 07:46:01: ----------------------------------------------
2024-06-20 07:46:01: Hello from AppDaemon for Portal Namerenych Dat
2024-06-20 07:46:01.878681 INFO AppDaemon: Removing module /homeassistant/appdaemon/apps/pnd/pnd.py
2024-06-20 07:46:01.879207 INFO AppDaemon: Terminating pnd
2024-06-20 07:46:01.879507 INFO AppDaemon: Calling terminate() for pnd
2024-06-20 07:46:01: >>>>>>>>>>>> PND Terminate
2024-06-20 07:46:02: Driver Loaded
2024-06-20 07:46:04: Website Opened
2024-06-20 07:46:15: App Version: 2.3.5
2024-06-20 07:46:18: H1 tag with text 'Naměřená data' is present.
2024-06-20 07:46:24: Rychla Sestava selected successfully!
2024-06-20 07:46:25: Selecting ELM '890971'
2024-06-20 07:46:26: Valid ELM numbers 'ELM 890971'
2024-06-20 07:46:29: ELM Status:  - 890971
2024-06-20 07:46:29: Iteration 0: Vyhledat Button IS disabled
2024-06-20 07:46:32: ELM Status: ELM 890971 - 890971
2024-06-20 07:46:32: Iteration 1: Vyhledat Button NOT disabled
2024-06-20 07:46:32: Device ELM '890971' selected successfully!
2024-06-20 07:46:35: Button 'Vyhledat data' clicked successfully!
2024-06-20 07:46:39: 07 Profil spotřeby za den (+A)
2024-06-20 07:46:46: Downloading CSV file for 07 Profil spotřeby za den (+A)
2024-06-20 07:46:52: File downloaded and saved as: /homeassistant/appdaemon/apps/pnd/daily-consumption.csv
2024-06-20 07:46:53: 08 Profil výroby za den (-A)
2024-06-20 07:47:00: Downloading CSV file for 08 Profil výroby za den (-A)
2024-06-20 07:47:06: File downloaded and saved as: /homeassistant/appdaemon/apps/pnd/daily-production.csv
2024-06-20 07:47:06: All Done - DAILY DATA DOWNLOADED
2024-06-20 07:47:06: Latest entry: 20.06.2024 23:59 - 0.9217 kWh
2024-06-20 07:47:06: Latest entry: 20.06.2024 23:59 - 8.712 kWh
2024-06-20 07:47:06: All Done - DAILY DATA PROCESSED
2024-06-20 07:47:09: Data Interval Entered - '27.10.2023 00:00 - 27.10.2024 00:00'
2024-06-20 07:47:12: Selecting 07 Profil spotřeby za den (+A)
2024-06-20 07:47:12: 07 Profil spotřeby za den (+A)
2024-06-20 07:47:18: Exporting data
2024-06-20 07:47:21: File downloaded and saved as: /homeassistant/appdaemon/apps/pnd/range-consumption.csv 14.82 KB
2024-06-20 07:47:21: Selecting 08 Profil výroby za den (-A)
2024-06-20 07:47:21: 08 Profil výroby za den (-A)
2024-06-20 07:47:26: Exporting data
2024-06-20 07:47:28: File downloaded and saved as: /homeassistant/appdaemon/apps/pnd/range-production.csv 14.71 KB
2024-06-20 07:47:28: All Done - INTERVAL DATA DOWNLOADED
2024-06-20 07:47:28: All Done - INTERVAL DATA PROCESSED
2024-06-20 07:47:29: All Done - BROWSER CLOSED
2024-06-20 07:47:29: Sensor State Set to OFF
2024-06-20 07:47:29: Debug Files Zipped
2024-06-20 07:47:29: ********************* Duration: 0:01:27.694297 *********************
2024-06-20 07:47:29: ********************* Finished 0.9.9:test *********************
2024-06-20 07:47:29.445986 WARNING AppDaemon: Excessive time spent in callback 'run_pnd() in pnd', Thread 'thread.thread-0' - now complete after 88.445181 seconds (limit=60.0)
ondrejvysek commented 1 week ago

Dobra otazka, na kterou neznam odpoved, alespon neni nikde zmineno. Ale co kontroluji na sitove vrstve?

Data v pnd se aktualizuji tusim 3x za den, tak to stejne nedava smysl.

Ale asi neni dobry napad to poustet nejak intenzivneji uz jenom kvuli mnozstvi dat v ha u rocnich senzoru.

Nechme to zatim zaparkovane, pokud se objevi, test verzi necham pro vyzkouseni.

jankorous commented 1 week ago

Jen drobnost - je ok, že po proběhnutí skriptu mi zmizí pnd.py?

ondrejvysek commented 1 week ago

Neni, ale:

  1. Nesmi byt v /apps/pnd, obsah toho se maze. Musi byt pouze v /apps
  2. U nekolika instalaci jsem zazil ze /apps/pnd.py zmizi a za cca 10min je nazpet - duvod se nedohledal

Neni nic, co hy odstranovalo /apps/pnd.py

Pokud by trvalo dlouhodobe / opakovane, pak je to na debug appdaemona