tiangolo / full-stack

Full stack, modern web application generator. Using Flask, PostgreSQL DB, Docker, Swagger, automatic HTTPS and more.
MIT License
523 stars 81 forks source link

How to troubleshoot a service when it fails to run [workflow for docker swarm deploy] #19

Closed AYEG closed 5 years ago

AYEG commented 5 years ago

Hi!

I'm trying to deploy a cluster using these instructions . https://github.com/tiangolo/full-stack/blob/master/docker-swarm-cluster-deploy.md

It seem like the DB is not coming online and causing the backend to also fail after many attempts from tenacity.

$ docker service logs db tells me "no such task or service: db", but I can see it is listed with $ docker service ls, with replicas 0/1

I don't have previous experience with swarm, is there a way to access failed service logs? I tried following advice like print journalctl, but that doesnt seem very useful in this case. Should I include something extra to output logs?

AYEG commented 5 years ago

One more thing I tried is edit docker-stack.yml to include

logging: driver: journald

Then run this command: $ journalctl -u docker CONTAINER_NAME=db

As described here: https://stackoverflow.com/questions/43663013/access-logs-of-a-killed-docker-container/43663253

It tells me: -- Logs begin at Mon 2018-11-12 21:37:13 CET, end at Thu 2018-12-27 14:15:03 CET. -- -- No entries --

AYEG commented 5 years ago

Found a way to print errors here is a possible solution for anyone else wondering how to do this:

docker service ls // list the services docker service ps <id> // shows container id and a truncated error message in second to last column docker inspect<container id from first column> | grep Err // print out the error message belonging to the service container

"Err": "no suitable node (scheduling constraints not satisfied on 1 node)"

Running these commands no longer requires enabling experimental docker features, it works by default

tiangolo commented 5 years ago

Sorry for the delay.

When you create a Docker Swarm mode stack, all the "services" get a name prefixed by the stack name.

So, if you create a stack my-super-app-com, your backend service will be named my-super-app-com_backend.

Then you can see the logs with:

docker service logs my-super-app-com_backend

To see only the last 100 logs:

docker service logs my-super-app-com_backend --tail=100

To see only the last 100 logs and keep showing any new logs live, "following" the logs:

docker service logs my-super-app-com_backend --tail=100 -f

(This last one is what I use the most).