cedbossneo / openmower-gui

28 stars 18 forks source link

Container carsh and then failed startup #42

Open 2m opened 4 months ago

2m commented 4 months ago

After full day of mowing today, I noticed that openmower-gui was not responding this evening. I checked the logs of the container and the last logs where:

[GIN] 2024/06/06 - 17:12:16 | 200 |         4m38s |   100.104.187.4 | GET      "/api/openmower/subscribe/pose"
Error #01: websocket: close 1001 (going away)
Error #02: write tcp 100.64.10.87:4006->100.104.187.4:37776: write: connection reset by peer
Error #03: write tcp 100.64.10.87:4006->100.104.187.4:37776: write: connection reset by peer
[GIN] 2024/06/06 - 17:12:16 | 200 |         4m38s |   100.104.187.4 | GET      "/api/openmower/subscribe/map"
Error #01: websocket: close 1001 (going away)
[GIN] 2024/06/06 - 17:17:29 | 200 |     477.092µs |   100.104.187.4 | POST     "/api/config/keys/get"
[GIN] 2024/06/06 - 17:17:29 | 200 |     812.369µs |   100.104.187.4 | GET      "/api/settings"
[GIN] 2024/06/06 - 17:17:29 | 200 |     233.462µs |   100.104.187.4 | POST     "/api/config/keys/get"
[GIN] 2024/06/06 - 17:17:29 | 200 |      288.74µs |   100.104.187.4 | GET      "/api/config/envs"
2024/06/06 17:17:38 http: proxy error: context canceled
2024/06/06 17:17:38 http: proxy error: context canceled
[GIN] 2024/06/06 - 17:17:38 | 502 |  1.087604922s |   100.104.187.4 | GET      "/tiles/rgb/22/2369201/1307072.png?r=red&g=green&b=blue"
[GIN] 2024/06/06 - 17:17:38 | 502 |  1.063002847s |   100.104.187.4 | GET      "/tiles/rgb/22/2369202/1307072.png?r=red&g=green&b=blue"

2024/06/06 17:17:51 [Recovery] 2024/06/06 - 17:17:51 panic recovered:
net/http: abort Handler
/usr/local/go/src/net/http/httputil/reverseproxy.go:519 (0x496c6b)
/app/pkg/api/tiles.go:31 (0x8ba94f)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x8780ef)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 (0x8780d4)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x87748f)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 (0x877458)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x8765c3)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 (0x8762ec)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 (0x875f0f)
/usr/local/go/src/net/http/server.go:2938 (0x2fc86b)
/usr/local/go/src/net/http/server.go:2009 (0x2f8c67)
/usr/local/go/src/runtime/asm_arm64.s:1197 (0x81813)

[GIN] 2024/06/06 - 17:17:51 | 200 | 13.816573681s |   100.104.187.4 | GET      "/tiles/rgb/22/2369202/1307071.png?r=red&g=green&b=blue"

2024/06/06 17:17:51 [Recovery] 2024/06/06 - 17:17:51 panic recovered:
net/http: abort Handler
/usr/local/go/src/net/http/httputil/reverseproxy.go:519 (0x496c6b)
/app/pkg/api/tiles.go:31 (0x8ba94f)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x8780ef)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 (0x8780d4)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x87748f)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 (0x877458)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x8765c3)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 (0x8762ec)
/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 (0x875f0f)
/usr/local/go/src/net/http/server.go:2938 (0x2fc86b)
/usr/local/go/src/net/http/server.go:2009 (0x2f8c67)
/usr/local/go/src/runtime/asm_arm64.s:1197 (0x81813)

[GIN] 2024/06/06 - 17:17:51 | 200 | 13.790320629s |   100.104.187.4 | GET      "/tiles/rgb/22/2369202/1307070.png?r=red&g=green&b=blue"
[GIN] 2024/06/06 - 17:17:51 | 200 | 14.034578109s |   100.104.187.4 | GET      "/tiles/rgb/22/2369201/1307070.png?r=red&g=green&b=blue"
[GIN] 2024/06/06 - 17:17:51 | 200 | 14.293403993s |   100.104.187.4 | GET      "/tiles/rgb/22/2369201/1307071.png?r=red&g=green&b=blue"
[GIN] 2024/06/06 - 17:17:52 | 200 |  1.306853036s |   100.104.187.4 | GET      "/tiles/rgb/22/2369200/1307070.png?r=red&g=green&b=blue"
[GIN] 2024/06/06 - 17:17:52 | 200 |  1.316946493s |   100.104.187.4 | GET      "/tiles/rgb/22/2369200/1307072.png?r=red&g=green&b=blue"
[GIN] 2024/06/06 - 17:17:52 | 200 |  1.329029334s |   100.104.187.4 | GET      "/tiles/rgb/22/2369201/1307072.png?r=red&g=green&b=blue"
[GIN] 2024/06/06 - 17:17:52 | 200 |  1.406972204s |   100.104.187.4 | GET      "/tiles/rgb/22/2369200/1307071.png?r=red&g=green&b=blue"
[GIN] 2024/06/06 - 17:17:59 | 200 |  154.028269ms |   100.104.187.4 | GET      "/tiles/rgb/22/2369202/1307071.png?r=red&g=green&b=blue"
[GIN] 2024/06/06 - 17:17:59 | 200 |  415.291572ms |   100.104.187.4 | GET      "/tiles/rgb/22/2369202/1307070.png?r=red&g=green&b=blue"
2024/06/06 17:18:40 http: proxy error: context canceled
[GIN] 2024/06/06 - 17:18:40 | 502 |     367.129µs |   100.104.187.4 | GET      "/tiles/rgb/22/2369202/1307072.png?r=red&g=green&b=blue"
[GIN] 2024/06/06 - 17:19:29 | 200 |   10.248577ms |   100.104.187.4 | POST     "/api/openmower/call/high_level_control"
[GIN] 2024/06/06 - 17:19:41 | 200 |   18.690325ms |   100.104.187.4 | POST     "/api/openmower/call/high_level_control"

It was a bit strange that systemctl did not start the container up back again. The container was still there, but stopped.

systemctl status gui returned:

● gui.service - Podman container - gui.service
     Loaded: loaded (/etc/systemd/system/gui.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-06-06 18:24:28 BST; 1h 1min ago
       Docs: man:podman-generate-systemd(1)
    Process: 1087 ExecStartPre=/bin/rm -f /run/container-gui.pid /run/container-gui.ctr-id (code=exited, status=0/SUCCESS)
    Process: 1088 ExecStart=/usr/bin/podman run --conmon-pidfile /run/container-gui.pid --cidfile /run/container-gui.ctr-id --cgroups=no-conmon --replace --detach --tty --privileged --name openmower-gui --network=host --env MOWER_CONFIG_FILE=/config/mower_config.sh --env DOCKER_HOST=unix:///run/podman/podman.sock --env ROS_MASTER_URI=http://localhost:11311 --env MAP_TILE_SERVER=http://localhost:5000 --env MAP_TILE_URI=/tiles/rgb/{z}/{x}/{y}.png?r=red&g=green&b=blue --env MAP_TILE_ENABLED=true --env MQTT_ENABLED=true --env MQTT_HOST=:2883 --volume /dev:/dev --volume /run/podman/podman.sock:/run/podman/podman.sock --volume /boot/openmower/db:/app/db --volume /boot/openmower/mower_config.txt:/config/mower_config.sh --label io.containers.autoupdate=image ghcr.io/cedbossneo/openmower-gui:master (code=exited, status=1/FAILURE)
    Process: 1153 ExecStopPost=/usr/bin/podman rm --ignore --force --cidfile /run/container-gui.ctr-id (code=exited, status=125)
        CPU: 514ms

I removed the stopped container and then it started back up normally.