IoT Edge device storage used for storing pending messages doesn't clear even if messages have already been sent by edgeHub module

anhashia commented 3 years ago

Summary

IoT Edge device storage used for storing pending messages doesn't clear even if messages have already been sent by edgeHub module upstream to Azure IoT Hub. Files holding the storage messages should be deleted and storage needs to be constantly reclaimed as messages are sent to IoT Hub. This will also help user to track if there any pending messages still left to be sent.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug. 1.) Set the storage folders for offline mode. Link module storage to device storage

Below links cover this in detail https://docs.microsoft.com/en-us/azure/iot-edge/offline-capabilities?view=iotedge-2018-06 https://docs.microsoft.com/en-us/azure/iot-edge/how-to-access-host-storage-from-module?view=iotedge-2018-06

2.) Run the simulated temperature sensor module. Set it to send message every 1 second. Set unlimited message count (-1) 3.) After 10 minutes or so temporarily block access to IoT Hub IP address. 4.) Message will start accumulating on edgeHub. edgeHub will also start throwing connectivity exception as it goes into offline mode. 5.) After 30 minutes, unblock the IP address of IoT Hub.

Expected Behavior

1.) All pending accumulated messages during offline mode are sent to IoT Hub 2.) Files used to store pending messages are cleared after 1 hour or so. Storage shouldn’t continuously grow.

Current Behavior

1.) All pending messages go through when edgeHub connection to IoT Hub is restored 2.) Storage continues to grow. Files used to store pending messages are not cleared. There is no flush even after 1 hour

Context (Environment)

Ubuntu 18.04.2 LTS

Device Information

Host OS : Ubuntu 18.04.2 LTS
Architecture: x86_64
Container OS Linux containers

Runtime Versions

iotedged [run iotedge version]: : iotedge 1.0.10
Edge Agent [image tag (e.g. 1.0.0)]:
Edge Hub [image tag (e.g. 1.0.0)]:
Docker/Moby [run docker version]:

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

iotedged logs

``` ```

edge-agent logs

``` ```

edge-hub logs

``` ```

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

github-actions[bot] commented 3 years ago

This issue is being marked as stale because it has been open for 30 days with no activity.

Mike94x commented 2 years ago

Hi guys, i have the same problem (msgs sent to hub when re-connected but not deleted from local storage), so my storage continues to grow. I need to solve this problem because my device may be offline for a long time. How can i solve this?

posterkorn commented 4 months ago

Hi there, we have the same issue on our devices. The messages sent to iot hub re not deleted from the local RocksDB. The storage is rising siginificantly in a few days. At the moment the storage folder contains data from 01.10.2022 till 23.04.2024 (approx. 70 Gb)

Device Information Host OS : Ubuntu 18.04.2 LTS Architecture: x64 Container OS Linux containers

Runtime Versions iotedged [run iotedge version]: : iotedge 1.1.15 Edge Agent: 1.4.18 Edge Hub: 1.4.18 Docker/Moby: 23.0.7+azure-1

Is there any solution available?

Azure / iotedge