Management command autostart not working from admin panel

pchevali commented 2 weeks ago

Expected Behavior

Triggering a Management Commands Over HTTP job with autostart should run the task

Actual Behavior

The task is marked as Queued, there is an error in celery task result with the following error : {"exc_type": "NotFound", "exc_message": ["ManagementCommandJob with id 1 was not found."], "exc_module": "rest_framework.exceptions"}

Steps to Reproduce the Problem

Open admin panel
Create a management command job over http ( ping_mngmt_commands_http for exemple)
Check autostart
Click save

Specifications

GeoNode version: 4.3
Installation type (vanilla, geonode-project): geonode project
Installation method (manual, docker): Docker
Additional details: Creating the management command with api works well.

mattiagiupponi commented 1 week ago

HI @pchevali I tried a couple of times on development demo and the async is correctly started and celery is logging the execution as expected

[2024-09-06 09:03:15,267: INFO/MainProcess] Task geonode.management_commands_http.tasks.run_management_command_async[e9218e9b-b534-423e-a33a-d86fc51c1801] received
[2024-09-06 09:03:15,268: DEBUG/MainProcess] TaskPool: Apply <function fast_trace_task at 0x7f3aa1103130> (args:('geonode.management_commands_http.tasks.run_management_command_async', 'e9218e9b-b534-423e-a33a-d86fc51c1801', {'lang': 'py', 'task': 'geonode.management_commands_http.tasks.run_management_command_async', 'id': 'e9218e9b-b534-423e-a33a-d86fc51c1801', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'e9218e9b-b534-423e-a33a-d86fc51c1801', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'job_id': 3}", 'origin': 'gen80004@d3626ae4f8dc', 'ignore_result': False, 'replaced_task_nesting': 0, 'stamped_headers': None, 'stamps': {}, 'properties': {'content_type': 'application/json', 'content_encoding': 'utf-8', 'application_headers': {'lang': 'py', 'task': 'geonode.management_commands_http.tasks.run_management_command_async', 'id': 'e9218e9b-b534-423e-a33a-d86fc51c1801', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id':... kwargs:{})
[2024-09-06 09:03:15,308: INFO/ForkPoolWorker-2855] Task geonode.management_commands_http.tasks.run_management_command_async[e9218e9b-b534-423e-a33a-d86fc51c1801] succeeded in 0.03869563527405262s: None

Can you try again from the master branch?

pchevali commented 1 week ago

Hi,

Actually it fails randomly on my production setup. Sometimes it works and sometimes no. I noticed that it has more probability to fail just after a restart of the docker compose stack. Locally it always works for me, but maybe it's due to the fact that the celery setup is not the same ( synchronous calls ?) I tried to dig into the django admin part as it is not the same path for creating the job in the admin page ( in admin.py) and in the api ( mixins.py:CreateJobMixin), but didn't find an answer. By any chance can the obj.save() call in admin.py at line 77 be an asynchronous call ? I supposed that it was a direct save call to the model ManagementCommandJob but cannot confirm that.

I will try to test with master branch when I find the time.

GeoNode / geonode