services in the docker containers stop working

[schulze-b1]

• multiple times we had the issue that the service in a container was not working, even if the container was still running and where healthchecks exist, it was showing as healthy
• containers on control nodes where we have seen this: ironic_dnsmasq, ironic_inspector, neutron_dhcp_agent
• only solution is to restart the containers
• where logs exist, we don't see anything suspicious
• in some cases the only indicator is, that logging has stopped altogether and the last change to the logfile as hours or days old
• how can this be prevented? it makes deploying very difficult and time consuming, if we do not know that a container is broken and needs a restart

[berendt]

If the health check says the health is OK when it is not, then the health check is broken and needs to be fixed in kolla-ansible.

Do the processes just get stuck?

[berendt]

Can you please attach a strace to the affected processes? We suspect it is related to the RabbitMQ Broker. For this, however, you would have to take a look at exactly where the processes are stuck.

[berendt]

I am just clearing the issues. There are a few notes on strace at https://docs.osism.tech/notes/openstack.html#debugging-services. Please reopen the issue when the problem has reoccurred and we can take a closer look.