Django based tool for monitoring huey task queue
Current implementation will just store all Huey task signals into the database and display them in the Django admin.
pip install django-huey-monitor
INSTALLED_APPS = [
#...
'bx_django_utils', # https://github.com/boxine/bx_django_utils
'huey_monitor',
#...
]
Note: You must normally not change your Huey tasks.
Huey monitor model can store information about task hierarchy. But this information can't be collected automatically. You have to store these information in your Task code.
e.g.:
@task(context=True)
def sub_task(task, parent_task_id, chunk_data):
# Save the task hierarchy by:
TaskModel.objects.set_parent_task(
main_task_id=parent_task_id,
sub_task_id=task.id,
)
# ... do something with e.g.: chunk_data ...
@task(context=True)
def main_task(task):
for chunk_data in something:
sub_task(parent_task_id=task.id, chunk_data=chunk_data)
Working example can be found in the test app here: huey_monitor_tests/test_app/tasks.py
Running task can store progress information in a similar way as tqdm. So it's possible to see the progress in admin.
Minimal example:
@task(context=True)
def foobar_task(task, list_to_process):
process_info = ProcessInfo(
task,
desc='A description of this Job',
total=len(list_to_process)
)
for item in list_to_process:
# ...to something with the {item}...
process_info.update(n=1) # add the information that one item was processed
It is also possible to divide the work to several tasks and collect information about the processing of main-/sub-tasks.
Working example can be found in the test app here: huey_monitor_tests/test_app/tasks.py
It is possible to override list_filter
of SignalInfoModelAdmin
and TaskModelAdmin
via settings.
e.g.:
HUEY_MONITOR_SIGNAL_INFO_MODEL_LIST_FILTER = ('task__name', 'signal_name')
HUEY_MONITOR_TASK_MODEL_LIST_FILTER = ('name', 'state__signal_name')
Note: This both settings are optional. In this example is the "hostname" filter not present ;)
Note: You can quickly test Huey Monitor with the test project, e.g:
~/django-huey-monitor$ ./manage.sh run_testserver
or in an isolated Docker container:
~/django-huey-monitor$ make up
More info see below.
(More Screenshots here: boxine.github.io/django-huey-monitor/)
To start developing e.g.:
~$ git clone https://github.com/boxine/django-huey-monitor.git
~$ cd django-huey-monitor
~/django-huey-monitor$ ./manage.py
~/django-huey-monitor$ make help
~/django-huey-monitor$ make up
Point our browser to: http://localhost:8000/
Our Makefile contains the following targets:
help List all commands
install install huey monitor package
update Update the dependencies as according to the pyproject.toml file
run-dev-server Run Django's developer server
test Run unittests
tox Run unittests via tox
makemessages Make and compile locales message files
clean Remove created files from the test project (e.g.: SQlite, static files)
build Update/Build docker services
up Start docker containers
down Stop all containers
shell-django go into a interactive bash shell in Django container
run-shell-django Build and start the Django container and go into shell
shell-huey1 go into a interactive bash shell in Huey worker container 1
shell-huey2 go into a interactive bash shell in Huey worker container 2
shell-huey3 go into a interactive bash shell in Huey worker container 3
shell-redis go into a interactive bash shell in Redis container
logs Display and follow docker logs
logs-django Display and follow docker logs only from "django" container
reload-django Reload the Django dev server
reload-huey Reload the Huey worker
restart Restart the containers
fire-test-tasks Call "fire-test-tasks" manage command to create some Huey Tasks
fire-many-test-tasks Call "fire-test-tasks" with --count 10000 to create many task entries ;)
fire-parallel-processing-task Just fire "parallel processing" Huey Task
delete-all-tasks-data Delete all Task/Signal database enties
It's also possible to run the test setup with SQLite and Huey immediate setup without docker:
~$ git clone https://github.com/boxine/django-huey-monitor.git
~$ cd django-huey-monitor
~/django-huey-monitor$ ./manage.py run_dev_server
Huey Monitor | Django | Python |
---|---|---|
>v0.7.0 | v3.2, v4.1, v4.2 | v3.9, v3.10, v3.11 |
>v0.6.0 | v3.2, v4.0, v4.1 | v3.9, v3.10, v3.11 |
>v0.5.0 | v2.2, v3.1, v3.2 | v3.7, v3.8, v3.9 |
<=v0.4.0 | v2.2, v3.0, v3.1 | v3.7, v3.8, v3.9 |
We refactor the project setup: Developer must reinit the repository.
Change CI and remove tests against Django 3.0, but add test run with Django v3.2
bx_py_utils was split and Django related stuff moved into: bx_django_utils
You must change your Django settings and replace the app name:
INSTALLED_APPS = [
#...
- 'bx_py_utils',
+ 'bx_django_utils',
'huey_monitor',
#...
]
DisallowedModelAdminLookup
in SignalInfoModelAdmin
, too.DisallowedModelAdminLookup
in TaskModelAdmin
ZeroDivisionError
in adminlist_filter
of SignalInfoModelAdmin
and TaskModelAdmin
via settingsmain
branchpytest
to normal unittests
poetry
to pip-tools
ProcessInfo.__init__
(contributed by Skrattoune)TaskModel.desc
max. length and make SignalInfoModel.exception_line
optionalTaskProgressModel
and store progress information into TaskModel
huey_monitor.css
filecumulate2parents
flag to ProcessInfo
(contributed by formacube)ProcessInfo.__str__()
ProcessInfo
tox
runs and use different Python versionsGPL. Patches welcome!