search

anand-p-r / fmi-hass-custom

Home Assistant Custom Component Supporting FMI Based Weather and Sensor Platforms
MIT License
59 stars 23 forks source link

lightning_strikes sensor continuously in "Unavailable" state #33

Closed MikaSakari closed 3 years ago

MikaSakari commented 3 years ago

Hi, wondering if I'm doing something wrong or if there is potentially a bug in the custom component. I'm running this component in RPi 3B in docker, HA version 2021.4.6 and version 0.4.0 of this custom component. After several reboots and now after having it been running for few days, this sensor is showing nothing but "Unavailable". All the other sensors seem to work correctly as they provide sensible values for my location. I have been eagerly waiting for it to start working as the lightning season is going to start this night in my place. I remember this sensor having been working few versions back, but haven't seen it being reporting any values since last few versions. Any idea what might be wrong? Thanks

anand-p-r commented 3 years ago

@MikaSakari Will it be possible to share debug logs from your env? It can be enabled by adding this to configuration.yaml: ''' logger: logs: custom_components.fmi: debug ''' And then restart HA. You can then copy and share the logs related to Fmi integration that show up in your env. Note: For privacy concerns, please do remove lat/long related info from the logs or mask it with something else.

MikaSakari commented 3 years ago

Hi @anand-p-r, this is what I see in the log after having let it run 30min+

anand-p-r commented 3 years ago

Thanks @MikaSakari The latest change reduced the lightning strike search area to 3000kms half-way creating a bounding box based on the lat/long configured in integration. Logs indicate that no lightning strike records were found in this bounding box. That probably means for certain locations, this bounding box logic does not seem to work right. Let me see if there is an alternative.

MikaSakari commented 3 years ago

@anand-p-r Thanks for investigating. In case sharing the coordinates would help you I'm happy to do so (e.g. via LinkedIn messaging). Anyway considering this integration is for Ilmatieteenlaitos and the GPS position is some 150km north from Helsinki I would have assumed 3000km (or even half of it) would easily cover my location (in case I got your comment correct).

MikaSakari commented 3 years ago

Or alternative, in case you would be able to share via this chain (some random) coordinates which are guaranteed to work I could then test my run time environment by using the same coordinates to see if the sensor provides anything by "Unavailable"

anand-p-r commented 3 years ago

@MikaSakari Can you try with the new release and see if this fixes the problem? The coordinate system used for the API requires lat/long to be reversed (=long/lat) and that is the reason for absurd values for the bounding box. This should be fixed now with the new version. Have a good weekend!

MikaSakari commented 3 years ago

@anand-p-r I fear I have both bad and good news. Bad news are that I can't get 0.4.1 to work at all, none of the sensors even appear. Because of that, I went back to 0.4.0 by clearing the fmi folder and copied 0.4.0 back into it. Good news are that to my surprise, that made also lightning_strikes sensor immediately to start reporting values for the first time in my installation. As I didn't trust my eyes, I again cleared fmi folder and saved 0.4.1 into it and same experience as earlier i.e. none of the sensors were even created. Now I'm again back in 0.4.0 and all sensors reporting sensible values and so far I haven't seen "Unavailable" been reported by lightning_strikes. Let's see how things evolve during the night. Right now the integration is reporting closest lightning 772km away in location Onezhsky District, Arkhangelsk Oblast, Northwestern Federal District, 164884, Russia. According to https://www.nordicweather.net/ukkostutka.php, while comparing by eye, that actually might be the closest lightning location from my place right now.

By the way, while adding fmi integration in HA configuration I notice one difference in how 0.4.1 behaved compared to 0.4.0. While adding FMI integration, after having entered location name+long+lat, 0.4.0 opened another window within which one could select in which area to add the FMI device just having been created. But while doing the same with 0.4.1, window having such area-selection didn't appear, there was only text within a window saying that the configuration was created.

Attached you can see what sort of entries were recorded in log while trying to use 0.4.0: log deleted

anand-p-r commented 3 years ago

@MikaSakari It seems the Lightning API is taking 40+secs to complete in your env: Finished fetching fmi data in 40.054 seconds

If I try the same URI above:

DEBUG:URL fetch time - 4.520875s
DEBUG:Looping time - 19.795509s
DEBUG:FMI - Coords retrieved for Lightning Data- 106026

Its close to 25secs. Can you try to clear the fmi folder again including all hidden folders and retry version 0.4.1? On a side note, I cleared my own folder and used version 0.4.1 and did not find any problems.

BTW, you are the only one to provide some feedback for the lightning sensor and hence this question to you - Do we really need 5 days historical lightning strikes' data or can we do with something less. e.g. 3 days or even 2 days? This would considerably reduce the volume of coordinates returned by the API and in turn the sensor startup time. What do you think? If you agree I can update the API call and publish a new version. Let me know.

MikaSakari commented 3 years ago

@anand-p-r I again tried to use 0.4.1 by first clearing the folder and even upgrading HA to the latest stable version (2021.5.5) and taking 0.4.1 into use but with no success (none of the sensors will appear for me, not even the weather one). HA Integration looking like following (I don't have earlier experience on 2021.5.x so may be this is feature of that release):

Screenshot 2021-05-22 at 12 40 27

The log seems to report continuously "Finished fetching fmi data in 40.xxx seconds" ie in 40s range. I'm behind 4G network, speedtest reporting ping 25ms/18Mbps down/16Mbps up so should not be the issue of internet connectivity.

What comes to your question around historical data, I personally don't need long history as I save all data for historical purposes in InfluxDB. Thus even 1 day would do for me. I'm more interested in getting visibility if there are lightnings close to my region and to see if those lightnings are approaching my place or going away.

-->

log deleted

anand-p-r commented 3 years ago

@MikaSakari I reduced the history to single day. For me the sensor starts up in 6 secs now! Try the new version and see if it helps. On a side note you could also try to manually probe this API from your RPI and see how long it takes.

MikaSakari commented 3 years ago

@anand-p-r My first impression is that version 0.4.2 seems to work much better for me. If I interpret the log entries correctly, fetching FMI data now came down to 10 sec ballpark in my RPi environment. Let's see when I will have the next lightning nearby to see how it will behave.

Thanks so much for your effort to resolve these issues.

2021-05-23 12:00:54 WARNING (MainThread) [homeassistant.loader] You are using a custom integration fmi which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant 2021-05-23 12:01:10 DEBUG (MainThread) [custom_components.fmi] Using lat: XX.XXX and long: YY.YYY 2021-05-23 12:01:10 DEBUG (MainThread) [custom_components.fmi] FMI: Data will be updated every 0:30:00 min 2021-05-23 12:01:10 DEBUG (MainThread) [custom_components.fmi] FMI: Best Conditions updated! 2021-05-23 12:01:10 DEBUG (SyncWorker_1) [custom_components.fmi] FMI: Lightning started 2021-05-23 12:01:10 DEBUG (SyncWorker_1) [custom_components.fmi] FMI: Lightning URI - https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::observations::lightning::multipointcoverage&timestep=3600&starttime=2021-05-22T12:01:10Z&bbox=ZZ,ZZ,ZZ,ZZ& 2021-05-23 12:01:18 DEBUG (SyncWorker_1) [custom_components.fmi] FMI - Coords retrieved for Lightning Data- 3124 2021-05-23 12:01:20 DEBUG (SyncWorker_1) [custom_components.fmi] FMI: Lightning ended 2021-05-23 12:01:20 DEBUG (MainThread) [custom_components.fmi] FMI: Lightning Conditions updated! 2021-05-23 12:01:20 DEBUG (SyncWorker_3) [custom_components.fmi] FMI: mareo started 2021-05-23 12:01:20 DEBUG (SyncWorker_3) [custom_components.fmi] FMI: Using Mareo url: http://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::forecast::oaas::sealevel::point::simple&timestep=30&latlon=XX.XXX,YY.YYY&starttime=2021-05-23T12:01:20Z& 2021-05-23 12:01:20 DEBUG (SyncWorker_3) [custom_components.fmi] Sealevel forecast records not in expected format for index - 0 of locstring - latlon=XX.XXX,YY.YYY 2021-05-23 12:01:20 DEBUG (SyncWorker_3) [custom_components.fmi] FMI: Mareo_data not updated. No data available! 2021-05-23 12:01:20 DEBUG (SyncWorker_3) [custom_components.fmi] FMI: mareo ended 2021-05-23 12:01:20 DEBUG (MainThread) [custom_components.fmi] FMI: Mareograph sea level data updated! 2021-05-23 12:01:20 DEBUG (MainThread) [custom_components.fmi] Finished fetching fmi data in 10.594 seconds 2021-05-23 12:01:21 DEBUG (MainThread) [custom_components.fmi] FMI: Sensor Mareo is unavailable 2021-05-23 12:01:21 DEBUG (MainThread) [custom_components.fmi] FMI: Sensor Mareo is unavailable 2021-05-23 12:31:21 DEBUG (MainThread) [custom_components.fmi] FMI: Best Conditions updated! 2021-05-23 12:31:21 DEBUG (SyncWorker_5) [custom_components.fmi] FMI: Lightning started 2021-05-23 12:31:21 DEBUG (SyncWorker_5) [custom_components.fmi] FMI: Lightning URI - https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::observations::lightning::multipointcoverage&timestep=3600&starttime=2021-05-22T12:31:21Z&bbox=ZZ,ZZ,ZZ,ZZ& 2021-05-23 12:31:27 DEBUG (SyncWorker_5) [custom_components.fmi] FMI - Coords retrieved for Lightning Data- 2565 2021-05-23 12:31:29 DEBUG (SyncWorker_5) [custom_components.fmi] FMI: Lightning ended 2021-05-23 12:31:30 DEBUG (MainThread) [custom_components.fmi] FMI: Lightning Conditions updated! 2021-05-23 12:31:30 DEBUG (SyncWorker_1) [custom_components.fmi] FMI: mareo started 2021-05-23 12:31:30 DEBUG (SyncWorker_1) [custom_components.fmi] FMI: Using Mareo url: http://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::forecast::oaas::sealevel::point::simple&timestep=30&latlon=XX.XXX,YY.YYY&starttime=2021-05-23T12:31:30Z& 2021-05-23 12:31:30 DEBUG (SyncWorker_1) [custom_components.fmi] Sealevel forecast records not in expected format for index - 0 of locstring - latlon=XX.XXX,YY.YYY 2021-05-23 12:31:30 DEBUG (SyncWorker_1) [custom_components.fmi] FMI: Mareo_data not updated. No data available! 2021-05-23 12:31:30 DEBUG (SyncWorker_1) [custom_components.fmi] FMI: mareo ended 2021-05-23 12:31:30 DEBUG (MainThread) [custom_components.fmi] FMI: Mareograph sea level data updated! 2021-05-23 12:31:30 DEBUG (MainThread) [custom_components.fmi] Finished fetching fmi data in 9.140 seconds 2021-05-23 12:31:30 DEBUG (MainThread) [custom_components.fmi] FMI: Sensor Mareo is unavailable 2021-05-23 12:50:19 DEBUG (MainThread) [custom_components.fmi] Using lat: XX.XXX and long: YY.YYY 2021-05-23 12:50:19 DEBUG (MainThread) [custom_components.fmi] FMI: Data will be updated every 0:30:00 min 2021-05-23 12:50:19 DEBUG (MainThread) [custom_components.fmi] FMI: Best Conditions updated! 2021-05-23 12:50:19 DEBUG (SyncWorker_4) [custom_components.fmi] FMI: Lightning started 2021-05-23 12:50:19 DEBUG (SyncWorker_4) [custom_components.fmi] FMI: Lightning URI - https://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::observations::lightning::multipointcoverage&timestep=3600&starttime=2021-05-22T12:50:19Z&bbox=ZZ,ZZ,ZZ,ZZ& 2021-05-23 12:50:25 DEBUG (SyncWorker_4) [custom_components.fmi] FMI - Coords retrieved for Lightning Data- 2206 2021-05-23 12:50:27 DEBUG (SyncWorker_4) [custom_components.fmi] FMI: Lightning ended 2021-05-23 12:50:27 DEBUG (MainThread) [custom_components.fmi] FMI: Lightning Conditions updated! 2021-05-23 12:50:27 DEBUG (SyncWorker_5) [custom_components.fmi] FMI: mareo started 2021-05-23 12:50:27 DEBUG (SyncWorker_5) [custom_components.fmi] FMI: Using Mareo url: http://opendata.fmi.fi/wfs?service=WFS&version=2.0.0&request=getFeature&storedquery_id=fmi::forecast::oaas::sealevel::point::simple&timestep=30&latlon=XX.XXX,YY.YYY&starttime=2021-05-23T12:50:27Z& 2021-05-23 12:50:27 DEBUG (SyncWorker_5) [custom_components.fmi] Sealevel forecast records not in expected format for index - 0 of locstring - latlon=XX.XXX,YY.YYY 2021-05-23 12:50:27 DEBUG (SyncWorker_5) [custom_components.fmi] FMI: Mareo_data not updated. No data available! 2021-05-23 12:50:27 DEBUG (SyncWorker_5) [custom_components.fmi] FMI: mareo ended 2021-05-23 12:50:27 DEBUG (MainThread) [custom_components.fmi] FMI: Mareograph sea level data updated! 2021-05-23 12:50:27 DEBUG (MainThread) [custom_components.fmi] Finished fetching fmi data in 8.414 seconds 2021-05-23 12:50:27 DEBUG (MainThread) [custom_components.fmi] FMI: Sensor Mareo is unavailable 2021-05-23 12:50:27 DEBUG (MainThread) [custom_components.fmi] FMI: Sensor Mareo is unavailable