solaredge reinitialize data when restart HA server and everything is 0

jluc2808 commented 4 years ago

The problem

when i restart HA (server) if i make a modification in configuration.yaml by exemple, the solaredge integration component fail to fetch data and all the data previously displayed are reset to 0

Environment

Home Assistant Core release with the issue: Home Assistant 0.112.0b2
Last working Home Assistant Core release (if known): 0.111
Operating environment (OS/Container/Supervised/Core): docker on synology
Integration causing this issue: last solaregede release branch dev
Link to integration documentation on our website: https://github.com/home-assistant/core/tree/dev/homeassistant/components/solaredge

Problem-relevant `configuration.yaml`

Traceback/Error logs

If you come across any trace or error logs, please provide them. Update for sensor.solaredge_pourcent_autoconso_now fails 16:36:48 – helpers/template.py (ERROR) - message survenu pour la première fois à 16:36:47 et apparu 2 fois

Additional information

probot-home-assistant[bot] commented 4 years ago

solaredge documentation solaredge source _{^{(message by IssueLinks)}}

balloob commented 4 years ago

@terminet85 do you think that this is related to your change?

jluc2808 commented 4 years ago

i think it's because the previous value stored on the each sensor aren't restored when i restart HA , perhaps there is a parameter i forget to tell HA to restore value ? but what i see is : all the sensor value need to have a refresh cycle (request of solaredge API) to display value with Lovelace, in the mean time i get an unexpected numeriqc value could not display till the request has been performed.

terminet85 commented 4 years ago

@terminet85 do you think that this is related to your change?

Never got problems on my local copy, but I doesn't restart hass so often. I'll double check when I'll come back to home.

terminet85 commented 4 years ago

i think it's because the previous value stored on the each sensor aren't restored when i restart HA , perhaps there is a parameter i forget to tell HA to restore value ? but what i see is : all the sensor value need to have a refresh cycle (request of solaredge API) to display value with Lovelace, in the mean time i get an unexpected numeriqc value could not display till the request has been performed.

Can you please enable debug component mode and paste the relevant sections of home-assistant.log?

jluc2808 commented 4 years ago

here a short part of the log (repeat for the solaredge sensor value)

2020-06-30 01:17:22 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.solaredge_cosumption_power fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 272, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 466, in async_device_update
    self.update  # type: ignore
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/solaredge/sensor.py", line 214, in update
    self.data_service.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 239, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/solaredge/sensor.py", line 412, in update
    self.data[energy_type] = row["value"]
KeyError: 'value'

terminet85 commented 4 years ago

Thank you for your feedback. Do you have any power consumption probes? Screenshot_20200630_084932

jluc2808 commented 4 years ago

No consumption problem, i just gave you an abstract with consumption , but the ERROR is repeated for all the value , production, Feed-in, Selfconsumption, GRID, PV, ...... as it's still the same error log i didn't embelded all the log

jluc2808 commented 4 years ago

issue solaredege

terminet85 commented 4 years ago

I've rechecked the code, and restarted HASS. Values aren't persistent, are displayed when API is re-fetched (ASAP in my site aprox 2-3 minutes).

And about your errors, I'm not able to reproduce. Seems like you have problems with API call.

root@sdomotica:/home/homeassistant/.homeassistant# cat home-assistant.log | grep solaredge
2020-06-30 09:37:26 ERROR (MainThread) [homeassistant.components.sensor] The solaredge platform for the sensor integration does not support platform setup. Please remove it from your config.       

2020-06-30 09:37:29 DEBUG (MainThread) [homeassistant.components.solaredge.sensor] Credentials correct and site is active

2020-06-30 09:38:00 DEBUG (SyncWorker_3) [homeassistant.components.solaredge.sensor] Updated SolarEdge overview: (...)

2020-06-30 09:38:01 DEBUG (SyncWorker_2) [homeassistant.components.solaredge.sensor] Updated SolarEdge details: (...)

2020-06-30 09:38:01 DEBUG (SyncWorker_5) [homeassistant.components.solaredge.sensor] Updated SolarEdge inventory: (...)

2020-06-30 09:38:02 DEBUG (SyncWorker_4) [homeassistant.components.solaredge.sensor] Updated SolarEdge power flow: (...)

2020-06-30 09:38:03 DEBUG (SyncWorker_0) [homeassistant.components.solaredge.sensor] Updated SolarEdge energy details: `(...)`

Can you please check if you got '[homeassistant.components.solaredge.sensor] Credentials correct and site is active' line?

jluc2808 commented 4 years ago

i agree, value aren't persistent, so all sensor in configuration.yaml fails to be setted and give an ERROR code like the one i show previously.

Credentials is OK, because all value get back after the 1rst request and return from API.

I don't think that a restart of HA should erase all the sensor value ? (don't display till lovelace is completly alive, yes, but this not only what append)

How could we stay with persistent value ?

jluc2808 commented 4 years ago

just to be precise: the component is directly installed via "Integration" with the last beta of HA, so i have no visible entry in configuration.yaml, my credentials are OK within this component inside "Integration"

I know that it is possible to overwrite this with adding a folder solaredge in my custom-component folder but i didn't know how to proceed ( where i could download the entire folder in the dev branch and does this enough)

terminet85 commented 4 years ago

just to be precise: the component is directly installed via "Integration" with the last beta of HA, so i have no visible entry in configuration.yaml, my credentials are OK within this component inside "Integration"
* I know that it is possible to overwrite this with adding a folder solaredge in my custom-component folder but i didn't know how to proceed ( where i could download the entire folder in the dev branch and does this enough)

However I need you enable debug mode in your configuration.yaml

logger:
   default: warning
   logs:
     homeassistant.components.solaredge: debug

and you attach your home-assistant.log. So we can check solaredge's debug lines.

jluc2808 commented 4 years ago

here is the logs home-assistant.log

terminet85 commented 4 years ago

Daily LimitationUse of the monitoring server API is subject to a query limit of 300 requests for a specific account token and aparallel query limit of 300 requests for each specific site ID from the same source IP. APIs that do not have a specific ID (e.g. Site List, Account List) will be counted as part of the account query limit.Anyadditional site or account level requestwill result in HTTP 429 error –too many requests.

Looking your log I saw [homeassistant.components.solaredge.sensor] Could not retrieve data, skipping update

And if you check the relevant part of sensor.py:

        except (ConnectTimeout, HTTPError):
            _LOGGER.error("Could not retrieve data, skipping update")
            return

Any change you have finished the 300 requests limit? Anyway there are something wrong with the request. The sub-consequential traceback errors are from helpers.entity that fails to update the sensor.

jluc2808 commented 4 years ago

no any chance to have reached the 300 request, because the next resquest is OK few minutes later (10 - 15 minutes) in the same day today. yes i agree , do you need some more trace ? or another sequence ? it's very strange i have some additionnal sensor which are calculated , does these sensor could make some board effect, just in case in post my configuration.yaml (in the zip)

configuration.zip

jluc2808 commented 4 years ago

suggestion : why all the sensor are reset ? does the date_time asked to the API have any influences ? could you store the date_time returned by the API data to avoid problems with the date ? do you need some timer to send requests with delay beetween ? (to many equest at the same time)

terminet85 commented 4 years ago

no any chance to have reached the 300 request, because the next resquest is OK few minutes later (10 - 15 minutes)

But in home-assistant.log you have posted there aren't any ok requests. Anyway there is also a "ConnecTimeout", maybe some network related problems?

does the date_time asked to the API have any influences ?

It's used only on the new class, others requests aren't involved, so I doubt.

could you store the date_time returned by the API data to avoid problems with the date ?

If you're writing about

            now = datetime.now()
            today = date.today()
            midnight = datetime.combine(today, datetime.min.time())

Aren't returned by API. I calculated midnight and now time in order to send get_energy_details's api request, It's need a timeframe.

do you need some timer to send requests with delay beetween ? (to many equest at the same time)

was already built an throttling system in orderd to avoid to reach 300 requests api call. Never got any problems with concurrency call at all

jluc2808 commented 4 years ago

what seems to me even more strange is : i don't see any ok request like you but after 5 hours running 1 - i get the same value with solareedge monitoring and sensor in HA value solaredege

this would say that there OK !!!

2 - i open an SSL session with HA and i can't find any folder : /usr/src/homeassistant/homeassistant/helpers/
i have /usr/src ????

3 - all the error : are with inverters, sensors, batteries, gateways

so, does the use of solaredge with "Integration" have the same reaction than in component ? is the logger setting : homeassistant.components.solaredge: debug enough ?

otherwise : when i talk about date , i ask for time_date within the response for each sensor (there is value and date) to display with the sensor the date_time of the sensor receive

jluc2808 commented 4 years ago

status in the file attached : abstract log solaredge.txt

you have 3 sequences:

at 5:36 (began at 0:11) where 2 requests are OK (overview and power flow) and 1 request down (energy details) for this one the log reproduce the ERROR for each sensor of energy details, this ERROR is the 1rst time at 0:11
at 5:42 there is a : Could not retrieve data
- at 7:25 and 7:42 all requests are OK

i have made no change beetween both, HA is running without interruption and without lovelace display.

I understand that Credentials are OK, request limit (300) aren't reached, solaredge program is running with the last version (include energy details)

terminet85 commented 4 years ago

@balloob https://github.com/home-assistant/core/issues/36970 duplicated. That thread was before you merged my pull request.

I cannot reproduce and I guess isn't related with my commit. Sorry I'll give up.

balloob commented 4 years ago

Alright. Thanks for trying to figure it out ! .

ghost commented 4 years ago

I'm not shure to start a new issue, I have a lot of SolareEdge errors since 122. Am i'm having the same issue as described above?

2020-07-07 19:53:33 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.solaredge_lifetime_energy is taking over 10 seconds

2020-07-07 19:50:26 WARNING (MainThread) [slixmpp.stringprep] Using slower stringprep, consider compiling the faster cython/libidn one.

2020-07-07 19:50:32 ERROR (MainThread) [homeassistant.core] Error doing job: Task was destroyed but it is pending!

2020-07-07 19:53:54 WARNING (MainThread) [homeassistant.components.sensor] Updating solaredge sensor took longer than the scheduled update interval 0:00:30

2020-07-07 19:53:55 ERROR (SyncWorker_14) [homeassistant.components.solaredge.sensor] Could not retrieve data, skipping update

2020-07-07 19:53:55 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.solaredge_lifetime_energy fails Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 272, in async_update_ha_state await self.async_device_update() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 466, in async_device_update self.update # type: ignore File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/solaredge/sensor.py", line 149, in update self._state = self.data_service.data[self._json_key] KeyError: 'lifeTimeData'

2020-07-07 19:53:55 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.solaredge_energy_this_year fails Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 272, in async_update_ha_state await self.async_device_update() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 466, in async_device_update self.update # type: ignore File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/solaredge/sensor.py", line 149, in update self._state = self.data_service.data[self._json_key] KeyError: 'lastYearData'

2020-07-07 19:53:55 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.solaredge_energy_this_month fails Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 272, in async_update_ha_state await self.async_device_update() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 466, in async_device_update self.update # type: ignore File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/solaredge/sensor.py", line 149, in update self._state = self.data_service.data[self._json_key] KeyError: 'lastMonthData'

2020-07-07 19:53:56 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.solaredge_energy_today fails Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 272, in async_update_ha_state await self.async_device_update() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 466, in async_device_update self.update # type: ignore File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/solaredge/sensor.py", line 149, in update self._state = self.data_service.data[self._json_key] KeyError: 'lastDayData'

2020-07-07 19:53:56 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.solaredge_current_power fails Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 272, in async_update_ha_state await self.async_device_update() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 466, in async_device_update self.update # type: ignore File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/solaredge/sensor.py", line 149, in update self._state = self.data_service.data[self._json_key] KeyError: 'currentPower'

2020-07-07 19:53:57 ERROR (SyncWorker_1) [homeassistant.components.solaredge.sensor] Could not retrieve data, skipping update

shows up 150 times....

terminet85 commented 4 years ago

Please enable plugin debug mode and attach home-assistant.log with enough data’s amount.

home-assistant / core