Several questions about container.logs / certain container hangs

[danqing]

Hi,

I have several questions about container.logs. I'm using Python 3 with docker-py 2.5.1.

• What's the difference between stream and follow? It looks like if I specify stream, it will wait for more logs to come, which is essentially follow. On the other hand, if I specify follow, then it has to be stream (how else will I get results?)
• What's the return type. In the docs it says that string is returned if not streaming. However, I'm getting bytes. I am getting strings (in the generator) when streaming at some time - I haven't been able to identify when it returns string vs. bytes, but I seem to be getting strings in a ThreadPoolExecutor environment, and bytes if it's in its dedicated process.

Also, I can't get streaming logs for certain containers, consistently, at least if I start the stream as soon as the container is started. There's no error, but it simply hangs forever. (It take the container say 5 minutes to produce the first line) But there is log, and I can get it after it's produced. Unfortunately I'm not able to reproduce with trivial examples.

Any help is much appreciated. Thanks!

[shin-]

stream affects the way you will receive the response. If it's set to False, you'll get a single chunk of data once the request completes. If True, it'll be a generator that streams data as it is received.

For most use-cases, people should use stream=True, unless you're looking at a very few lines of logs that you expect to get immediately.

If you set follow=True and stream=False, the method will wait for the request to complete (container exits or is stopped) or until the HTTP timeout is hit, whichever comes first. That combination is not the most useful, but those 2 parameters have separate meanings and they should stay separate.

Is it possible that the containers you're getting strings for are TTY-enabled? That's admittedly something we've been fairly bad at keeping consistent.

I'm happy to look at repro cases if you manage to provide one on that hanging logs thing. It's not something I've run in so far.

[danqing]

So I struggle to get the relation between TTY and stream/follow. When I turn off TTY, I don't seem to get anything (it's stuck forever) - not sure if that's the correct behavior or not.

It seems to me that if I have stream=True but follow=False, the iterator still waits for future results, behaving like "follow". That is, I found whether I follow or not the behavior seems to be the same.

I couldn't come up with a minimal example that hangs, but I do have one additional piece of info that may be helpful: if I do a while loop first that's not streaming, waiting for the first bit of log to show up, and only then jump out of the loop and start streaming, then it will work correctly.

I don't know how docker manages the underlying file, but there's a chance that streaming didn't find the file/it didn't exist until the first line of log is generated.

[danqing]

For the hanging logs, it seems that if no new logs come out in a minute or two it will start to hang and not recover. Logs need to constantly come out in order for streaming to work.

[danqing]

any suggestions as of what to do when stream hangs?

[danqing]

@shin- apparently the issue is that the connection times out after a minute or so. Can we have custom timeouts for these APIs? When there's no log produced for a few minutes the connection becomes broken and no logs can go through.

[danqing]

Friendly ping @shin-

[danqing]

Ping again @shin-

[Spriithy]

Bumping this : is there a way to bypass the HTTP request timeout ? I have a long-blocking process in my docker and when the http request times-out.

[Spriithy]

For future reference : I solved this issue by using subprocess.Popen with the docker logs -f <id> command.

import subprocess
import docker

dockercli = docker.from_env()
container = docker.containers.run(image, command, detach=True)

cid = container.id
logs = subprocess.Popen(['docker', 'logs', '-f', cid], stdout=subprocess.PIPE)
for c in iter(lambda: logs.stdout.read(1).decode('utf-8'), ''):
    sys.stdout.write(c)
    sys.stdout.flush()