Feature request: Read data that was written within the refresh period

[ei-ke]

Hello Angel,

thank you for writing this integration. I just set it up and it is working properly. My question related to the data created from an LoRa device within the refresh period. The device currently sends data every minute and the refresh period on HA is set to 5 minutes. I thought that this would mean that I don't hammer on TTN's API but still would receive the 5 data points. Currently this doesn't seem to be the case. Would it be feasible to fetch those data points as well without too much work on your end?

Greetings Eike

[angelnu]

Yes, you are right that the current implementation only picks the newest data point during the refresh period. Passing all of them to HA makes sense.

[angelnu]

Added with https://github.com/angelnu/home_assistant_thethingsnetwork/commit/0ed31e8cca3df607ac72f19434ba59c9886a3edc

I will close after confirmation it works as I do not have any sensor updating so frequently.

[ei-ke]

Thank you for this fast work. I updated to the current version and the sensor data was not showing up after a restart. The log was showing:

2021-12-20 18:27:44 ERROR (MainThread) [custom_components.thethingsnetwork] Not authorized for Application ID: attnode-die-erste
2021-12-20 18:27:44 ERROR (MainThread) [custom_components.thethingsnetwork] Unexpected error fetching The Things Network data: 'async for' requires an object with __aiter__ method, got NoneType

I thought a) a missing permission on the API key or b) I replaced the API key while playing aroung. As there was no option to change the API key I removed the integration and set it up again. Still the same error though the API key has all permissions and not just "View devices in application" and "Read application traffic (uplink and downlink)":

2021-12-20 18:33:46 ERROR (MainThread) [custom_components.thethingsnetwork] Not authorized for Application ID: attnode-die-erste
2021-12-20 18:33:46 ERROR (MainThread) [custom_components.thethingsnetwork] Unexpected error fetching The Things Network data: 'async for' requires an object with __aiter__ method, got NoneType
  File "/home/homeassistant/.homeassistant/custom_components/thethingsnetwork/TTN_client.py", line 153, in fetch_data_from_ttn
  File "/home/homeassistant/.homeassistant/custom_components/thethingsnetwork/TTN_client.py", line 191, in __fetch_data_from_ttn
2021-12-20 18:34:40 ERROR (MainThread) [custom_components.thethingsnetwork] Not authorized for Application ID: attnode-die-erste
2021-12-20 18:34:40 ERROR (MainThread) [custom_components.thethingsnetwork] Unexpected error fetching The Things Network data: 'async for' requires an object with __aiter__ method, got NoneType
  File "/home/homeassistant/.homeassistant/custom_components/thethingsnetwork/TTN_client.py", line 153, in fetch_data_from_ttn
  File "/home/homeassistant/.homeassistant/custom_components/thethingsnetwork/TTN_client.py", line 191, in __fetch_data_from_ttn
2021-12-20 18:34:46 ERROR (MainThread) [custom_components.thethingsnetwork] Not authorized for Application ID: attnode-die-erste
2021-12-20 18:34:46 ERROR (MainThread) [custom_components.thethingsnetwork] Unexpected error fetching The Things Network data: 'async for' requires an object with __aiter__ method, got NoneType
  File "/home/homeassistant/.homeassistant/custom_components/thethingsnetwork/TTN_client.py", line 153, in fetch_data_from_ttn
  File "/home/homeassistant/.homeassistant/custom_components/thethingsnetwork/TTN_client.py", line 191, in __fetch_data_from_ttn
2021-12-20 18:35:10 ERROR (MainThread) [custom_components.thethingsnetwork] Not authorized for Application ID: attnode-die-erste
2021-12-20 18:35:10 ERROR (MainThread) [custom_components.thethingsnetwork] Unexpected error fetching The Things Network data: 'async for' requires an object with __aiter__ method, got NoneType
  File "/home/homeassistant/.homeassistant/custom_components/thethingsnetwork/TTN_client.py", line 153, in fetch_data_from_ttn
  File "/home/homeassistant/.homeassistant/custom_components/thethingsnetwork/TTN_client.py", line 191, in __fetch_data_from_ttn
2021-12-20 18:35:16 ERROR (MainThread) [custom_components.thethingsnetwork] Not authorized for Application ID: attnode-die-erste
2021-12-20 18:35:16 ERROR (MainThread) [custom_components.thethingsnetwork] Unexpected error fetching The Things Network data: 'async for' requires an object with __aiter__ method, got NoneType
  File "/home/homeassistant/.homeassistant/custom_components/thethingsnetwork/TTN_client.py", line 153, in fetch_data_from_ttn
  File "/home/homeassistant/.homeassistant/custom_components/thethingsnetwork/TTN_client.py", line 191, in __fetch_data_from_ttn

EDIT: I removed the integration entirely and reinstalled it. At least the data is coming in, but there are no entites.

2021-12-20 18:53:01 DEBUG (MainThread) [custom_components.thethingsnetwork] TTN entry: b'{"result":{"end_device_ids":{"device_id":"attnode-001","application_ids":{"application_id":"attnode-die-erste"},"dev_eui":"~~~snip~~~","dev_addr":"~~~snip~~~"},"received_at":"2021-12-20T17:52:05.165199343Z","uplink_message":{"f_port":1,"f_cnt":605,"frm_payload":"~~~snip~~~,"decoded_payload":{"humidity":54.28,"pressure":1020.59,"temperature":20.21,"v":2.9},"rx_metadata":[{"gateway_ids":{"gateway_id":"~~~snip~~~","eui":"~~~snip~~~"},"time":"2021-12-20T17:52:04.920624Z","timestamp":3501907564,"rssi":-69,"channel_rssi":-69,"snr":8,"location":{"latitude":~~~snip~~~,"longitude":~~~snip~~~,"altitude":65,"source":"SOURCE_REGISTRY"},"channel_index":5}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":7}},"coding_rate":"4/5","frequency":"867500000","timestamp":3501907564,"time":"2021-12-20T17:52:04.920624Z"},"received_at":"2021-12-20T17:52:04.942479981Z","consumed_airtime":"0.061696s","network_ids":{"net_id":"000013","tenant_id":"ttn","cluster_id":"ttn-eu1"}}}}\n'
2021-12-20 18:53:01 DEBUG (MainThread) [custom_components.thethingsnetwork] Finished fetching The Things Network data in 0.750 seconds (success: True)
2021-12-20 18:53:02 DEBUG (SyncWorker_4) [homeassistant.util.json] JSON file not found: /home/homeassistant/.homeassistant/custom_components/thethingsnetwork/translations/sensor.en.json
2021-12-20 18:53:02 DEBUG (SyncWorker_4) [homeassistant.util.json] JSON file not found: /home/homeassistant/.homeassistant/custom_components/thethingsnetwork/translations/binary_sensor.en.json
2021-12-20 18:53:02 DEBUG (SyncWorker_4) [homeassistant.util.json] JSON file not found: /home/homeassistant/.homeassistant/custom_components/thethingsnetwork/translations/device_tracker.en.json
2021-12-20 18:53:02 DEBUG (SyncWorker_4) [homeassistant.util.json] JSON file not found: /home/homeassistant/.homeassistant/custom_components/thethingsnetwork/translations/sensor.de.json
2021-12-20 18:53:02 DEBUG (SyncWorker_4) [homeassistant.util.json] JSON file not found: /home/homeassistant/.homeassistant/custom_components/thethingsnetwork/translations/de.json
2021-12-20 18:53:02 DEBUG (SyncWorker_4) [homeassistant.util.json] JSON file not found: /home/homeassistant/.homeassistant/custom_components/thethingsnetwork/translations/binary_sensor.de.json
2021-12-20 18:53:02 DEBUG (SyncWorker_4) [homeassistant.util.json] JSON file not found: /home/homeassistant/.homeassistant/custom_components/thethingsnetwork/translations/device_tracker.de.json

[angelnu]

I had "cleaned up" too much, including not adding new devices at the end x-)

Please update and give it another try.

[ei-ke]

I can confirm that if within the refresh period a value changes this will also be logged into HA as soon as the next data is loaded from TTN. Thank you!