search

gliderlabs / logspout

Log routing for Docker container logs
MIT License
4.65k stars 679 forks source link

Not seeing logs on all containers? #139

Closed diranged closed 8 years ago

diranged commented 8 years ago

We have some containers that are launched through Rancher and it seems like their logs are not making it into Logspout. Here's an example host:

[root@dev-rancher-node-playground-uswest2-4-i-5125f795:/var/log]# docker ps
CONTAINER ID        IMAGE                                                            COMMAND                CREATED             STATUS                      PORTS                                          NAMES
db8ec79f1f14        xxx/elasticsearch:latest       "/root/run"            46 hours ago        Up 46 hours                 9200/tcp                                       d7b45a5b-2bfb-4429-a71a-636fe874d0f1   
057c5dca9f4b        xxxdynamodb-local:latest      "/root/run"            46 hours ago        Up 46 hours                 9334/tcp                                       bd06e093-d3de-4fa7-a498-1a8ae85f1cc8   
90fab2af57eb        xxx/memcached:latest           "/root/run"            46 hours ago        Up 46 hours                 11211/tcp                                      b9d0c9c7-3ffc-4c1d-8620-a9a0f4cd96ab   
b827f1862ff2        xxxx/elasticmq:latest           "/root/run"            47 hours ago        Up 47 hours                 9324/tcp                                       308fd0b8-2fd8-4648-91b8-7a3d8f37534d   
605fb417d665        xxx/echo_server/v0004:latest   "/bin/sh -c 'cd /app   3 days ago          Up 2 days                   5000-5001/tcp                                  1eab7853-ea8e-4458-b490-6c06f8228b64   
a7febea96c20        rancher/agent:v0.8.2                                             "/run.sh run"          4 days ago          Up 4 days                                                                  rancher-agent                          
574821cbe34f        redis:2.6.17                                                     "/entrypoint.sh redi   10 days ago         Up 10 days                  6379/tcp                                       2cd543cf-1dc1-4286-8784-782b266b9f31   
7dbc7a264189        rancher/agent                                                    "/run.sh https://ran   2 weeks ago         Restarting (0) 4 days ago                                                  rancher-node                           
332d27e660fd        xxx/harry-rtd:1                "/bin/sh -c '/usr/sh   2 weeks ago         Up 2 weeks                                                                 e7056c0c-3ef8-446e-9b77-422dcee4c5c2   
fd1d024081fb        rancher/agent-instance:v0.4.1                                    "/etc/init.d/agent-i   2 weeks ago         Up 2 weeks                  0.0.0.0:500->500/udp, 0.0.0.0:4500->4500/udp   b410b62d-a1d7-4fea-9c58-4819d47571fe   
d563f3bea6c6        gliderlabs/logspout                                              "/bin/logspout syslo   2 weeks ago         Up 2 weeks                  8000/tcp                                       logspout

Some of our logs make it into Logspout properly:

[root@dev-rancher-node-playground-uswest2-4-i-5125f795:/var/log:130]# grep b410b62d-a1d7-4fea-9c58-4819d47571fe /var/log/syslog  | wc
    166    1490   21618
[root@dev-rancher-node-playground-uswest2-4-i-5125f795:/var/log]# docker logs b410b62d-a1d7-4fea-9c58-4819d47571fe | wc
    163     632    8646

However, others don't:

[root@dev-rancher-node-playground-uswest2-4-i-5125f795:/var/log]# grep 308fd0b8-2fd8-4648-91b8-7a3d8f37534d /var/log/syslog 
[root@dev-rancher-node-playground-uswest2-4-i-5125f795:/var/log:130]# docker logs 308fd0b8-2fd8-4648-91b8-7a3d8f37534d | wc
     19     167    3900

Any thoughts on how to start debugging this?

janeczku commented 8 years ago

@diranged Logspout (for whatever intentions) ignores all containers that have TTY option enabled. Since TTY is set to true by default when creating a new service in Rancher you won't see any logs for them. cc/ @progrium

jbergknoff commented 8 years ago

I'm having a similar issue in ECS. Some container logs make it to papertrail and some do not. Any thoughts or suggestions for how to debug?

progrium commented 8 years ago

@jbergknoff as the previous comments suggest, try running without TTY.

jbergknoff commented 8 years ago

Yeah, the containers in question didn't have TTY allocated. I switched from latest to master and so far things look okay.