Container stuck on "Removal in Progress" after update

[mdegat01]

I tried to update the signal addon today and it wouldn't start again. When I went to the UI and clicked start I got this message: It seems like the addon did not shut down properly and is now stuck. I can't really seem to get it unstuck either, here's what I've tried: As you can see, no joy. Here's the logs from the dockerd service at the time I updated (around 2022-01-18T09:04:38-05:00) and when I tried to force kill it later (around 2022-01-18T11:52:47-05:00):

2022-01-18T12:01:04-05:00   time="2022-01-18T17:01:04.398353222Z" level=error msg="Handler for DELETE /v1.41/containers/99d130465de0 returned error: container 99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a: driver \"overlay2\" failed to remove root filesystem: unlinkat /mnt/data/docker/overlay2/6e940ced0aba9ebc22e9be604af7ada8c5702a507e86b051d3d0f08ef85af887/merged: device or resource busy"
2022-01-18T12:01:04-05:00   time="2022-01-18T17:01:04.398033750Z" level=error msg="Error removing mounted layer 99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a: unlinkat /mnt/data/docker/overlay2/6e940ced0aba9ebc22e9be604af7ada8c5702a507e86b051d3d0f08ef85af887/merged: device or resource busy"
2022-01-18T11:52:47-05:00   time="2022-01-18T16:52:47.108876796Z" level=error msg="Handler for DELETE /v1.41/containers/99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a returned error: container 99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a: driver \"overlay2\" failed to remove root filesystem: unlinkat /mnt/data/docker/overlay2/6e940ced0aba9ebc22e9be604af7ada8c5702a507e86b051d3d0f08ef85af887/merged: device or resource busy"
2022-01-18T11:52:47-05:00   time="2022-01-18T16:52:47.108573033Z" level=error msg="Error removing mounted layer 99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a: unlinkat /mnt/data/docker/overlay2/6e940ced0aba9ebc22e9be604af7ada8c5702a507e86b051d3d0f08ef85af887/merged: device or resource busy"
2022-01-18T11:51:45-05:00   time="2022-01-18T16:51:45.816048998Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/b392b1e58a7f5a65bf62dba26669c1b6eb926c74301b862544d079fb9d3c4b06 pid=1246545
2022-01-18T11:51:41-05:00   time="2022-01-18T16:51:41.804681343Z" level=error msg="copy shim log" error="read /proc/self/fd/50: file already closed"
2022-01-18T11:51:41-05:00   time="2022-01-18T16:51:41.804386789Z" level=info msg="shim disconnected" id=4f0206b5d068c51114ea212c64a6a4be8edbc1819c23fcce9bbae4e772dc792e
2022-01-18T11:51:41-05:00   time="2022-01-18T16:51:41.804014147Z" level=info msg="ignoring event" container=4f0206b5d068c51114ea212c64a6a4be8edbc1819c23fcce9bbae4e772dc792e module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2022-01-18T11:51:41-05:00   time="2022-01-18T16:51:41.755714583Z" level=info msg="Container 4f0206b5d068c51114ea212c64a6a4be8edbc1819c23fcce9bbae4e772dc792e failed to exit within 10 seconds of signal 15 - using the force"
2022-01-18T11:50:53-05:00   time="2022-01-18T16:50:53.499986503Z" level=error msg="Handler for DELETE /v1.41/containers/99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a returned error: container 99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a: driver \"overlay2\" failed to remove root filesystem: unlinkat /mnt/data/docker/overlay2/6e940ced0aba9ebc22e9be604af7ada8c5702a507e86b051d3d0f08ef85af887/merged: device or resource busy"
2022-01-18T11:50:53-05:00   time="2022-01-18T16:50:53.499445063Z" level=error msg="Error removing mounted layer 99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a: unlinkat /mnt/data/docker/overlay2/6e940ced0aba9ebc22e9be604af7ada8c5702a507e86b051d3d0f08ef85af887/merged: device or resource busy"
2022-01-18T11:49:35-05:00   time="2022-01-18T16:49:35.395935217Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/4f0206b5d068c51114ea212c64a6a4be8edbc1819c23fcce9bbae4e772dc792e pid=1245580
2022-01-18T11:49:34-05:00   time="2022-01-18T16:49:34.547006220Z" level=info msg="ignoring event" container=5eaaa6bc2163f0b37f10fdaaf7cd3ec12f6d06a64797ec8a96098a4d1b359d75 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2022-01-18T11:49:34-05:00   time="2022-01-18T16:49:34.546799628Z" level=error msg="copy shim log" error="read /proc/self/fd/118: file already closed"
2022-01-18T11:49:34-05:00   time="2022-01-18T16:49:34.546643662Z" level=info msg="shim disconnected" id=5eaaa6bc2163f0b37f10fdaaf7cd3ec12f6d06a64797ec8a96098a4d1b359d75
2022-01-18T11:49:34-05:00   time="2022-01-18T16:49:34.416508311Z" level=info msg="Container 5eaaa6bc2163f0b37f10fdaaf7cd3ec12f6d06a64797ec8a96098a4d1b359d75 failed to exit within 10 seconds of signal 15 - using the force"
2022-01-18T09:04:38-05:00   time="2022-01-18T14:04:38.846817553Z" level=error msg="Handler for DELETE /v1.41/containers/99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a returned error: container 99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a: driver \"overlay2\" failed to remove root filesystem: unlinkat /mnt/data/docker/overlay2/6e940ced0aba9ebc22e9be604af7ada8c5702a507e86b051d3d0f08ef85af887/merged: device or resource busy"
2022-01-18T09:04:38-05:00   time="2022-01-18T14:04:38.846509373Z" level=error msg="Error removing mounted layer 99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a: unlinkat /mnt/data/docker/overlay2/6e940ced0aba9ebc22e9be604af7ada8c5702a507e86b051d3d0f08ef85af887/merged: device or resource busy"
2022-01-18T09:04:33-05:00   time="2022-01-18T14:04:33.517519874Z" level=error msg="Handler for DELETE /v1.41/containers/99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a returned error: container 99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a: driver \"overlay2\" failed to remove root filesystem: unlinkat /mnt/data/docker/overlay2/6e940ced0aba9ebc22e9be604af7ada8c5702a507e86b051d3d0f08ef85af887/merged: device or resource busy"
2022-01-18T09:04:33-05:00   time="2022-01-18T14:04:33.517165109Z" level=error msg="Error removing mounted layer 99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a: unlinkat /mnt/data/docker/overlay2/6e940ced0aba9ebc22e9be604af7ada8c5702a507e86b051d3d0f08ef85af887/merged: device or resource busy"
2022-01-18T09:04:28-05:00   time="2022-01-18T14:04:28.261937439Z" level=error msg="copy shim log" error="read /proc/self/fd/50: file already closed"
2022-01-18T09:04:28-05:00   time="2022-01-18T14:04:28.244749930Z" level=info msg="shim disconnected" id=99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a
2022-01-18T09:04:28-05:00   time="2022-01-18T14:04:28.243743135Z" level=info msg="ignoring event" container=99d130465de0ee5cbe3aee703b8126413585025bacdbe58e68232e494bf6b84a module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

You may remember I submitted an issue for this a while ago when this addon was in the other repo ( https://github.com/haberda/hassio_addons/issues/29 ). Truth be told it really never went away. I've just been restarting the host every update, it's the only way I've been able to get the addon going again after it gets stuck like this. I decided to submit the issue this time because its a bit annoying and I'm just hoping someone has an idea.

And it happens to me so consistently I feel like it must happen to someone else, right? Fwiw I actually have like 20 addons running at all times and this addon is the only one this seems to happen with, its very strange.

[haberda]

Hey @mdegat01, I remember the issue. Like before I can't get it to reproduce and no-one else has reported an issue like this and I don't really have an explanation.

Quick question: are you using home assistant OS or a supervised install? I have used it on both just fine, but if you have a supervised install maybe you need to update docker?

[mdegat01]

Home Assistant OS, I have an HA Blue actually. That's so strange, its pretty consistently reproducible for me.

I guess if I'm the only one I'll keep poking at it when I have time and see if I can come up with an explanation. I did some googling of the error and it has been seen with other containers too. The solution always seems to be restart docker (which in the case of HAOS kind of just means reboot the host). Haven't yet found a reason I can tie back to this addon though.

[github-actions[bot]]

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!