Describe the bug
In setups using Docker Compose, health checks work only for dependency container (and only if they are update containers with an AL service inside ;)). Health check for actual service containers are broken by the scaler, as it doesn't set the required environmental variable.
It's due to the difference between _start_container and _start methods in the docker_ctl.py. The first one is used by start_stateful_container (used for dependency containers) and the second by set_target (used for scaling services).
Both sets the same health check (link 1) (link 2), but only the first sets AL_SERVICE_NAME (link 1) (link 2).
Finally, the health check results from services looks like this:
Traceback (most recent call last):\n File \"<frozen runpy>\", line 198, in _run_module_as_main
File \"<frozen runpy>\", line 88, in _run_code\n File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_v4_service/healthz.py\", line 35, in <module>
perform_check()\n File \"/var/lib/assemblyline/.local/lib/python3.11/site-packages/assemblyline_v4_service/healthz.py\", line 17, in perform_check
forge.get_service_queue(service=environ['AL_SERVICE_NAME'])
~~~~~~~^^^^^^^^^^^^^^^^^^^
File \"<frozen os>\", line 679, in __getitem__
KeyError: 'AL_SERVICE_NAME'
To Reproduce
Steps to reproduce the behavior:
Set up AL as docker compose.
Configure a service, e.g. YARA
Wait for scaler to set up containers
Use docker ps to see the health status - observe unhealthy service containers
Use docker inspect <container name> to see results of health check
Expected behavior
Health check for containers works for all containers.
Screenshots
Environment (please complete the following information if pertinent):
Assemblyline Version: 4.5.0.18, Yara 4.5.0.5
Additional context
In addition, if a service sets up a dependency that isn't an AL updater (e.g. Redis), the health check won't work at all, as _start_container forces the AL service health check. Interestingly, the _start method has the logic to determine whether to set the health check or not. I'd welcome it in the _set_container too :)
cccs-rs
commented
7 months ago
Describe the bug In setups using Docker Compose, health checks work only for dependency container (and only if they are update containers with an AL service inside ;)). Health check for actual service containers are broken by the scaler, as it doesn't set the required environmental variable.
It's due to the difference between
_start_containerand_startmethods in thedocker_ctl.py. The first one is used bystart_stateful_container(used for dependency containers) and the second byset_target(used for scaling services). Both sets the same health check (link 1) (link 2), but only the first setsAL_SERVICE_NAME(link 1) (link 2).Finally, the health check results from services looks like this:
To Reproduce Steps to reproduce the behavior:
docker psto see the health status - observe unhealthy service containersdocker inspect <container name>to see results of health checkExpected behavior Health check for containers works for all containers.
Screenshots
Environment (please complete the following information if pertinent):
Additional context In addition, if a service sets up a dependency that isn't an AL updater (e.g. Redis), the health check won't work at all, as
_start_containerforces the AL service health check. Interestingly, the_startmethod has the logic to determine whether to set the health check or not. I'd welcome it in the_set_containertoo :)