Open benoit74 opened 2 months ago
What we see is that the scraper container have finished (containers f506be0a1d31
and 32424149f271
gone).
But the task containers which are responsible to manage the whole task execution are still there (containers ad7dfd16fe54
and 4aed9b3cd658
).
@kevinmcmurtrie could you please give us the last lines of these two containers as well?
They don't seem to be doing anything.
$ docker logs -t -n 25 ad7dfd16fe54
2024-06-21T20:53:53.607318797Z username=pixelmemory
2024-06-21T20:53:53.607327547Z webapi_uris=['https://api.farm.openzim.org/v1']
2024-06-21T20:53:53.607335687Z workdir=/data
2024-06-21T20:53:53.607350767Z task_id=4436db27-adcb-49c4-be0f-d74af0164bf0
2024-06-21T20:53:53.607360177Z [2024-06-21 20:53:53,607: INFO] testing workdir at /data…
2024-06-21T20:53:53.607369237Z [2024-06-21 20:53:53,607: INFO] workdir is available and writable
2024-06-21T20:53:53.607377407Z [2024-06-21 20:53:53,607: INFO] testing private key at /etc/ssh/keys/zimfarm…
2024-06-21T20:53:53.691983592Z [2024-06-21 20:53:53,691: INFO] private key is available and readable (a3121794f8e27e40c2f20782fae70eab)
2024-06-21T20:53:53.692023802Z [2024-06-21 20:53:53,691: INFO] testing authentication with https://api.farm.openzim.org/v1…
2024-06-21T20:53:54.752378767Z [2024-06-21 20:53:54,752: INFO] authentication successful
2024-06-21T20:53:54.752433407Z [2024-06-21 20:53:54,752: INFO] testing docker API on /var/run/docker.sock…
2024-06-21T20:53:54.770756971Z [2024-06-21 20:53:54,770: INFO] docker API access successful
2024-06-21T20:53:54.782102019Z [2024-06-21 20:53:54,782: INFO] Host hardware resources:
2024-06-21T20:53:54.782126059Z CPU : 20 (total) ; 11 (avail)
2024-06-21T20:53:54.782144659Z RAM : 160 GiB (total) ; 134 GiB (avail)
2024-06-21T20:53:54.782151699Z Disk: 3 TiB (configured) ; 9.59 TiB (avail) ; 203.13 GiB (reserved) ; 2.8 TiB (remain)
2024-06-21T20:53:54.782161509Z [2024-06-21 20:53:54,782: INFO] registering exit signals
2024-06-21T20:53:54.782169919Z [2024-06-21 20:53:54,782: INFO] Fetching task details for 4436db27-adcb-49c4-be0f-d74af0164bf0
2024-06-21T20:53:55.310394299Z [2024-06-21 20:53:55,310: INFO] Updating task-status=started
2024-06-21T20:53:55.846879537Z [2024-06-21 20:53:55,846: INFO] Setting-up workdir
2024-06-21T20:53:55.849309102Z [2024-06-21 20:53:55,849: INFO] Starting DNS cache
2024-06-21T20:53:57.033792153Z [2024-06-21 20:53:57,033: DEBUG] DNS Cache started using IPs: ['172.17.0.13']
2024-06-21T20:53:57.033822502Z [2024-06-21 20:53:57,033: INFO] Starting scraper. Expects files at: /Yosemite/zimfarm/data/4436db27-adcb-49c4-be0f-d74af0164bf0
2024-06-21T20:53:57.035737170Z [2024-06-21 20:53:57,035: DEBUG] Pulling image ghcr.io/openzim/ted:dev
2024-06-21T20:54:24.173183862Z [2024-06-21 20:54:24,173: INFO] Updating task-status=scraper_started
$ docker logs -t -n 25 4aed9b3cd658
2024-06-21T10:01:00.779871640Z webapi_uris=['https://api.farm.openzim.org/v1']
2024-06-21T10:01:00.779880010Z workdir=/data
2024-06-21T10:01:00.779895310Z task_id=12c2a80a-f0be-4fed-9d36-1078ab4b440c
2024-06-21T10:01:00.779906280Z [2024-06-21 10:01:00,779: INFO] testing workdir at /data…
2024-06-21T10:01:00.779918120Z [2024-06-21 10:01:00,779: INFO] workdir is available and writable
2024-06-21T10:01:00.779926470Z [2024-06-21 10:01:00,779: INFO] testing private key at /etc/ssh/keys/zimfarm…
2024-06-21T10:01:00.863235667Z [2024-06-21 10:01:00,863: INFO] private key is available and readable (a3121794f8e27e40c2f20782fae70eab)
2024-06-21T10:01:00.863266247Z [2024-06-21 10:01:00,863: INFO] testing authentication with https://api.farm.openzim.org/v1…
2024-06-21T10:01:01.973726177Z [2024-06-21 10:01:01,973: INFO] authentication successful
2024-06-21T10:01:01.973772017Z [2024-06-21 10:01:01,973: INFO] testing docker API on /var/run/docker.sock…
2024-06-21T10:01:01.989067831Z [2024-06-21 10:01:01,989: INFO] docker API access successful
2024-06-21T10:01:01.997389094Z [2024-06-21 10:01:01,997: INFO] Host hardware resources:
2024-06-21T10:01:01.997401284Z CPU : 20 (total) ; 14 (avail)
2024-06-21T10:01:01.997415134Z RAM : 160 GiB (total) ; 148 GiB (avail)
2024-06-21T10:01:01.997420814Z Disk: 3 TiB (configured) ; 9.62 TiB (avail) ; 173.13 GiB (reserved) ; 2.83 TiB (remain)
2024-06-21T10:01:01.997427184Z [2024-06-21 10:01:01,997: INFO] registering exit signals
2024-06-21T10:01:01.997447344Z [2024-06-21 10:01:01,997: INFO] Fetching task details for 12c2a80a-f0be-4fed-9d36-1078ab4b440c
2024-06-21T10:01:02.509530841Z [2024-06-21 10:01:02,509: INFO] Updating task-status=started
2024-06-21T10:01:02.997001147Z [2024-06-21 10:01:02,996: INFO] Setting-up workdir
2024-06-21T10:01:02.999647909Z [2024-06-21 10:01:02,999: INFO] Starting DNS cache
2024-06-21T10:01:03.766344035Z [2024-06-21 10:01:03,766: DEBUG] DNS Cache started using IPs: ['172.17.0.10']
2024-06-21T10:01:03.766374615Z [2024-06-21 10:01:03,766: INFO] Starting scraper. Expects files at: /Yosemite/zimfarm/data/12c2a80a-f0be-4fed-9d36-1078ab4b440c
2024-06-21T10:01:03.767748795Z [2024-06-21 10:01:03,767: DEBUG] Pulling image ghcr.io/openzim/mwoffliner:1.13.0
2024-06-21T10:01:05.176917791Z [2024-06-21 10:01:05,176: INFO] Updating task-status=scraper_started
2024-06-21T19:15:49.220726258Z [2024-06-21 19:15:49,220: ERROR] ConnectionError (attempt 1) for PATCH https://api.farm.openzim.org/v1/tasks/12c2a80a-f0be-4fed-9d36-1078ab4b440c -- HTTPSConnectionPool(host='api.farm.openzim.org', port=443): Read timed out. (read timeout=None)
Sending them a CTRL-C:
$ docker attach ad7dfd16fe54
^C[2024-06-24 06:49:43,950: INFO] received exit signal (Interrupt), shutting down…
[2024-06-24 06:49:43,950: INFO] Stopping all containers and actions
[2024-06-24 06:49:43,950: INFO] Stopping and removing monitor
[2024-06-24 06:49:43,950: INFO] Stopping and removing dnscache
[2024-06-24 06:49:44,314: INFO] Stopping and removing scraper
[2024-06-24 06:49:44,554: INFO] Stopping and removing log_uploader
[2024-06-24 06:49:44,554: INFO] Stopping and removing artifacts_uploader
[2024-06-24 06:49:44,554: INFO] Stopping and removing uploader
[2024-06-24 06:49:44,554: INFO] Stopping and removing checker
[2024-06-24 06:49:44,554: INFO] Removing task workdir /data
[2024-06-24 06:49:44,554: WARNING] ZIM files exists. removing anyway: [('ted_mul_activism_2024-06.zim', '4.66 GiB')]
[2024-06-24 06:49:44,556: INFO] Updating task-status=canceled
$ docker attach 4aed9b3cd658
^C[2024-06-24 06:50:52,176: INFO] received exit signal (Interrupt), shutting down…
[2024-06-24 06:50:52,176: INFO] Stopping all containers and actions
[2024-06-24 06:50:52,176: INFO] Stopping and removing monitor
[2024-06-24 06:50:52,176: INFO] Stopping and removing dnscache
[2024-06-24 06:50:52,530: INFO] Stopping and removing scraper
[2024-06-24 06:50:52,756: INFO] Stopping and removing log_uploader
[2024-06-24 06:50:52,756: INFO] Stopping and removing artifacts_uploader
[2024-06-24 06:50:52,756: INFO] Stopping and removing uploader
[2024-06-24 06:50:52,756: INFO] Stopping and removing checker
[2024-06-24 06:50:52,756: INFO] Removing task workdir /data
[2024-06-24 06:50:52,764: INFO] Updating task-status=canceled
So clearly it looks like we have a concern on the task manager which is not detecting that scraper container has finished / stopped (or not checking anymore maybe ...)
Thank you @kevinmcmurtrie ; next time please include the inspect
output (at least the State
part).
Task manager queries docker API for scraper container status and condiders the following as running:
created
and running
and restarting
from what was shared. It's possible the containers were paused
(external cause, zimfarm doesnt pause containers) or removing
(something preventing removal from finishing) though.I'd wait for another occurrence (with the status clue) before taking action
Thank you @rgaudin for the wise input!
@kevinmcmurtrie reported: