celery / celery

Distributed Task Queue (development branch)
https://docs.celeryq.dev
Other
24.72k stars 4.66k forks source link

Task status doesn't change when completed after issued worker shutdown #4548

Closed erkib closed 6 years ago

erkib commented 6 years ago

Checklist

Versions used

cx_Oracle==6.0.2 Django==1.11.6 celery==4.1.0 django-celery-results==1.0.1

Steps to reproduce

  1. Execute a task that doesn't complete so quickly that you can issue shutdown of the worker during the execution of the task.
  2. When task finishes, worker shuts down
  3. Check the status of the task from backend: in my case using django-db, its still STARTED. I have a hook for on_success where I write some more details of the task to separate database table and this hook is called.

Expected behavior

After issuing a worker shutdown, task running at that time are completed either successfully or with errors and the corresponding status of the task is written to backend (SUCCESS, FAILURE) and correct counters updated.

Actual behavior

After issuing a worker shutdown, task running at that time are completed either successfully or with errors, but the status of the task in backend is not updated and corresponding counters are not updated. Because I use track_started = True, tasks remain in STARTED state.

Log from Celery worker

[2018-02-20 08:15:39,552: DEBUG/MainProcess] | Consumer: Closing event loop... [2018-02-20 08:15:39,552: DEBUG/MainProcess] | Consumer: Stopping event loop... [2018-02-20 08:15:39,552: DEBUG/MainProcess] | Consumer: Stopping Gossip... [2018-02-20 08:15:39,561: DEBUG/MainProcess] Closed channel #3 [2018-02-20 08:15:39,561: DEBUG/MainProcess] | Consumer: Stopping Heart... [2018-02-20 08:15:39,561: DEBUG/MainProcess] | Consumer: Stopping Control... [2018-02-20 08:15:39,563: DEBUG/MainProcess] Closed channel #2 [2018-02-20 08:15:39,563: DEBUG/MainProcess] | Consumer: Stopping Tasks... [2018-02-20 08:15:39,563: DEBUG/MainProcess] Canceling task consumer... [2018-02-20 08:15:39,564: DEBUG/MainProcess] | Consumer: Stopping Mingle... [2018-02-20 08:15:39,564: DEBUG/MainProcess] | Consumer: Stopping Events... [2018-02-20 08:15:39,564: DEBUG/MainProcess] | Consumer: Stopping Connection... [2018-02-20 08:15:39,564: DEBUG/MainProcess] | Worker: Stopping Pool... [2018-02-20 08:15:48,663: INFO/ForkPoolWorker-17] puppet.run[ce6980ce-7e3a-4aa5-bc25-1891e262d065]: Running Puppet on foo.test.ee using timeout 1 minutes as user erkib [2018-02-20 08:16:48,791: INFO/ForkPoolWorker-17] Task puppet.run[ce6980ce-7e3a-4aa5-bc25-1891e262d065] succeeded in 84.34898755000904s: True [2018-02-20 08:16:49,806: DEBUG/MainProcess] | Worker: Stopping Hub... [2018-02-20 08:16:49,806: DEBUG/MainProcess] | Consumer: Shutdown Gossip... [2018-02-20 08:16:49,806: DEBUG/MainProcess] | Consumer: Shutdown Heart... [2018-02-20 08:16:49,806: DEBUG/MainProcess] | Consumer: Shutdown Control... [2018-02-20 08:16:49,806: DEBUG/MainProcess] | Consumer: Shutdown Tasks... [2018-02-20 08:16:49,806: DEBUG/MainProcess] Canceling task consumer... [2018-02-20 08:16:49,806: DEBUG/MainProcess] Closing consumer channel... [2018-02-20 08:16:49,807: DEBUG/MainProcess] | Consumer: Shutdown Events... [2018-02-20 08:16:49,808: DEBUG/MainProcess] Closed channel #1 [2018-02-20 08:16:49,808: DEBUG/MainProcess] | Consumer: Shutdown Connection... [2018-02-20 08:16:49,809: DEBUG/MainProcess] Closed channel #1 [2018-02-20 08:16:49,810: DEBUG/MainProcess] removing tasks from inqueue until task handler finished

auvipy commented 6 years ago

please use celery 4.2.1 and report again.