search

cascandaliato / docker-restarter

MIT License
14 stars 0 forks source link

Got exception when needed to restart #3

Open karserasl opened 1 year ago

karserasl commented 1 year ago

restarter | 2023-09-24T12:30:02.509038316Z Traceback (most recent call last): restarter | 2023-09-24T12:30:02.509065623Z File "/usr/src/app/./restarter.py", line 32, in <module> restarter | 2023-09-24T12:30:02.509143644Z service = event["Actor"]["Attributes"]["com.docker.compose.service"] restarter | 2023-09-24T12:30:02.509219177Z ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ restarter | 2023-09-24T12:30:02.509233587Z KeyError: 'com.docker.compose.service'

karserasl commented 1 year ago

Im using socket-proxy for docker host

`restarter: <<: *common-keys-core # See EXTENSION FIELDS at the top image: ghcr.io/cascandaliato/docker-restarter container_name: restarter networks:

cascandaliato commented 1 year ago

I pushed a fix (0c6057ace05bf9c2ae4750c3b1046daafb5f7abe). You probably started or stopped a container defined outside of a docker compose file.

Is Tecnativa/docker-socket-proxy the proxy you are talking about? If so, this project needs the following permissions: EVENTS (enabled by default?), CONTAINERS, POST, ALLOW_RESTARTS.

karserasl commented 1 year ago

Thanks alot Gonna try

karserasl commented 1 year ago

I think using ALLOW_RESTARTS=1 and the latest version fixed everything

Great work

karserasl commented 1 year ago

Talked too fast. Im getting this exception now.

restarter | 2023-09-24T15:01:52.654079529Z Restarting the following container(s) in 10 seconds because container qbittorrent (re)started: cross-seed, cross-seed-search restarter | 2023-09-24T15:33:28.054311911Z Restarting the following container(s) in 10 seconds because container gluetun (re)started: autobrr, joal, portcheck, qbittorrent, thelounge restarter | 2023-09-24T15:33:29.823397734Z Traceback (most recent call last): restarter | 2023-09-24T15:33:29.823410523Z File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status restarter | 2023-09-24T15:33:29.833985297Z response.raise_for_status() restarter | 2023-09-24T15:33:29.834000573Z File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status restarter | 2023-09-24T15:33:29.839697374Z raise HTTPError(http_error_msg, response=self) restarter | 2023-09-24T15:33:29.840563152Z requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://socket-proxy:2375/v1.43/containers/4f6a65c0047b48d46bc1a962696dea9226fc7f5a7938bb81c815755e0ec0d6c8/restart?t=10 restarter | 2023-09-24T15:33:29.840567853Z restarter | 2023-09-24T15:33:29.840569867Z The above exception was the direct cause of the following exception: restarter | 2023-09-24T15:33:29.840571908Z restarter | 2023-09-24T15:33:29.840573848Z Traceback (most recent call last): restarter | 2023-09-24T15:33:29.840575881Z File "/usr/src/app/./restarter.py", line 40, in restarter | 2023-09-24T15:33:29.840639762Z docker_client.containers.get(depender).restart() restarter | 2023-09-24T15:33:29.840687543Z File "/usr/local/lib/python3.11/site-packages/docker/models/containers.py", line 395, in restart restarter | 2023-09-24T15:33:29.851685047Z return self.client.api.restart(self.id, *kwargs) restarter | 2023-09-24T15:33:29.851714893Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ restarter | 2023-09-24T15:33:29.851739577Z File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped restarter | 2023-09-24T15:33:29.856188061Z return f(self, resource_id, args, **kwargs) restarter | 2023-09-24T15:33:29.856212190Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ restarter | 2023-09-24T15:33:29.856216523Z File "/usr/local/lib/python3.11/site-packages/docker/api/container.py", line 1089, in restart restarter | 2023-09-24T15:33:29.858447220Z self._raise_for_status(res) restarter | 2023-09-24T15:33:29.858459696Z File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status restarter | 2023-09-24T15:33:29.858578592Z raise create_api_error_from_http_exception(e) from e restarter | 2023-09-24T15:33:29.858594269Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ restarter | 2023-09-24T15:33:29.858608526Z File "/usr/local/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception restarter | 2023-09-24T15:33:29.868615961Z raise cls(e, response=response, explanation=explanation) from e restarter | 2023-09-24T15:33:29.868635785Z docker.errors.APIError: 500 Server Error for http://socket-proxy:2375/v1.43/containers/4f6a65c0047b48d46bc1a962696dea9226fc7f5a7938bb81c815755e0ec0d6c8/restart?t=10: Internal Server Error ("Cannot restart container 4f6a65c0047b48d46bc1a962696dea9226fc7f5a7938bb81c815755e0ec0d6c8: No such container: c6205526a4ce82f0d4ff8b090d1138d701ecf9910dd571e02a5ab221004b066d")

cascandaliato commented 1 year ago

I've added more log traces. The code doesn't expect (yet) any container to disappear so it won't work if there are services that remove containers (watchtower?) but I suspect it's something else because the proxy API is logging two different container ids in the same error message:

docker.errors.APIError: 500 Server Error for http://socket-proxy:2375/v1.43/containers/4f6a65c0047b48d46bc1a962696dea9226fc7f5a7938bb81c815755e0ec0d6c8/restart?t=10: Internal Server Error ("Cannot restart container 4f6a65c0047b48d46bc1a962696dea9226fc7f5a7938bb81c815755e0ec0d6c8: No such container: c6205526a4ce82f0d4ff8b090d1138d701ecf9910dd571e02a5ab221004b066d")

Do you mind trying again and sharing the container logs? Thank you for your patience.

vdrover commented 1 year ago

Just adding a note that watchtower support would be wonderful.

karserasl commented 1 year ago

Yeah, it was the watchtower that updated Gluetun. So, that's the main problem.

Gluetun pushed a new release and Watchtower updates it, which caused the whole stack depending on it to lose connection.

cascandaliato commented 1 year ago

Ok, that makes sense. I'll make the script handle this case and only log an error if a container is not available anymore when we try to restart it, for example because watchtower has updated it, the stack has been rebuilt or the user just removed it with docker rm.

cascandaliato commented 1 year ago

After commit 7295fcaa4b8b71f648430fd1ba5a6a984a5fd209 I don't expect the container to crash if it's trying to restart a container while watchtower is removing and updating it. Please give it a try when possible.

karserasl commented 1 year ago

im still seeing this

restarter  | 2023-09-28T15:08:03.586274819Z Container gluetun (id cef21da8dee3, service gluetun) restarted
restarter  | 2023-09-28T15:08:03.606773656Z The following containers depend on service gluetun and will be restarted in 30 seconds:
restarter  | 2023-09-28T15:08:03.608356355Z   qbittorrent (id d2543234000c, service qbittorrent)

restarter  | 2023-09-28T15:08:38.071354033Z Restarting container qbittorrent (id d2543234000c, service qbittorrent)
restarter  | 2023-09-28T15:08:49.117722348Z Could not restart container qbittorrent (id d2543234000c, service qbittorrent) because of error: 500 Server Error for http://socket-proxy:2375/v1.43/containers/d2543234000cc6be03349c23270e65c22135499cdfa6cbd8a1703ac4e2f95406/restart?t=10: Internal Server Error ("Cannot restart container d2543234000cc6be03349c23270e65c22135499cdfa6cbd8a1703ac4e2f95406: No such container: a7a0622a5331c1e838b668c6e7c5ad808c0ca8f78213f95eec29e3d07331b6e6")

was after gluetun update from watchtower

cascandaliato commented 1 year ago

Thank you for taking the time to test the latest version.

Do you know if watchtower also updated the qbittorrent container?

Here are some things that might help me:

If you prefer, you can reach me on Twitter or Reddit (same username as here).

cascandaliato commented 1 year ago

Nevermind, I have replicated the issue so there is no need to collect logs, etc. It's the same issue qdm12/deunhealth#11 that Quentin has been working on. I'll read his notes and see if I can do my own implementation.