search

Waziup / WaziGate

Waziup LoRa Edge gateway
21 stars 20 forks source link

Unreachable wazidev GUI after 10 days runtime of RPi 3b ( swap memory nearly 100%) #144

Open SvenZeisberg opened 3 years ago

SvenZeisberg commented 3 years ago

After 10 days of running a wazigate the swap space of 100M is used to about 97% (according to htop via ssh) ...in an other case after 12 days it was almost 100% and the system become instable due to constantly restarting postgresql container ...wazigate version was v2.1.10 ...a manual restart of the postgresql container did not help, also a restart of all docker containers did not help, and also finally a reboot of the rasperry pi did not solve the problem ...so the remote wazigate is accessible via ssh, but wazigate GUI can not be accessed anymore and also no data are forwarded to the cloud anymore from the three connected wazidevs ...

error_docker_postgresql_screenshot

j-forster commented 3 years ago

Thank you for submitting this issue! To help me identify this issue, please provide additional information: Please run htop or (top) and df via SHH and post the results preferably as text using the Insert code <> button.

SvenZeisberg commented 3 years ago

image This is the same device now showing also two more containers constantly restarting ...

SvenZeisberg commented 3 years ago

error_docker_htop This is a screenshot from htop on the same device.

SvenZeisberg commented 3 years ago

After rebooting the swap was again almost zero ...and after some three hours it was grown to about 10% ...it seems to grow constantly over time ...

SvenZeisberg commented 3 years ago

..and here a log file of the waziup.wazigate-lora container docker_wazigate-lora.log

j-forster commented 3 years ago

Can you please run df and post the result as text? Your htop screenshot does not show 100% swap memory - you you give me a htop that does have 100% swap?

j-forster commented 3 years ago

The wazigate-lora log did not yield any result. Can you collect the logs from MongoDB and PostgreSQL? Also have a look at docker stats and post the results at text here. With your help we can make Waziup better! Thanks in advance!

j-forster commented 3 years ago

@SvenZeisberg sent this file: docker_postgresql_5.log

j-forster commented 3 years ago

@SvenZeisberg sent this file: image_2021_02_23T09_51_09_049Z

@SvenZeisberg sent this file: image_2021_02_23T09_53_42_591Z

@SvenZeisberg sent this file: image_2021_02_23T10_07_07_456Z

SvenZeisberg commented 3 years ago

docker_wazigate-mongo_10min.log This is the log from the last 10min of wazigate-mongo container. This log file is quite large (175MB!!!), so I provide only the last 10min.

SvenZeisberg commented 3 years ago

docker_postgresql_full.log This is the log from postgresql container.

SvenZeisberg commented 3 years ago

The same behavior (wazigate not reachable via GUI anymore) now appeared at the students Pi 3b after running roguhly 10-14 days the installation from image v2.1.10 ... with two wazidev connected by LoRa (each minute) and having a temp/humidity sensor attached to each wazidev.

SvenZeisberg commented 3 years ago

In order to verify this error I would say, any team member might put at least one RPi 3b in his office and run it (with at least two wazidev attached by LoRa receiving values each minute) for at least 2 weeks ...then it will probably become unreachable via GUI as well and you can study the log files in detail. At least in my case this error was the case with two RPi 3b and in case of the student it was the case with one RPi 3b running on wazigate image v2.1.10 with at least two wazidev connected each minute having a new sensor value. And it would be very important, that it is a real user like installation, flashing really a fresh SD card and performing exactly the same steps written at the waziup web site for installation via image and then start ...