use socket from exec_run in python3 (python2 is working) -- 'SocketIO' object has no attribute 'sendall'

[christianbur]

The following code works fine with pyhton2 (docker-py 2.5.1), but not with python3 (docker-py 3.7.0)

import docker

client = docker.DockerClient(base_url='unix://var/run/docker.sock', version='auto')
container = client.containers.run("gliderlabs/alpine", command= "sleep 100", detach = True)
socket = container.exec_run(cmd="sh", stdin=True, socket = True)
socket.sendall(b"ls\n")

# a read block after a send
try:
    unknown_byte=socket.recv(1)
    while 1:
        # note that os.read does not work
        # because it does not TLS-decrypt
        # but returns the low-level encrypted data
        # one must use "socket.recv" instead
        data = socket.recv(16384)
        if not data: break
        print(data.decode('utf8'))
except so.timeout: pass

socket.sendall(b"exit\n")

in python3 "container.exec_run" returns ExecResult(exit_code=None, output=<socket.SocketIO object at 0x7f5bb7f7c160>).

So my idea was to use socket.output.sendall(b "ls\n"), but then I always get the error AttributeError: 'SocketIO' object has no attribute 'sendall'"". The documentation says only If socket=True, a socket object for the connection, and sendall() is a socket method for me.

How to use exec_run in python3?

Source: https://github.com/docker/docker-py/issues/983 https://stackoverflow.com/questions/46521166/how-to-write-to-stdin-in-a-tls-enabled-docker-using-the-python-docker-api https://github.com/mailcow/mailcow-dockerized/pull/2297

Version: docker-host: {'Platform': {'Name': ''}, 'Components': [{'Name': 'Engine', 'Version': '18.03.1-ce', 'Details': {'ApiVersion': '1.37', 'Arch': 'amd64', 'BuildTime': '2018-04-26T07:15:30.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9ee9f40', 'GoVersion': 'go1.9.5', 'KernelVersion': '4.15.0-45-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}], 'Version': '18.03.1-ce', 'ApiVersion': '1.37', 'MinAPIVersion': '1.12', 'GitCommit': '9ee9f40', 'GoVersion': 'go1.9.5', 'Os': 'linux', 'Arch': 'amd64', 'KernelVersion': '4.15.0-45-generic', 'BuildTime': '2018-04-26T07:15:30.000000000+00:00'}

conatiner Alpine 3.9 with docker-py 3.7.0

[christianbur]

if i check the available methods of SocketIO i get the following call: print ([method_name for method_name in dir(worker_socket) if callable(getattr(worker_socket, method_name))]) output:

['__class__', '__del__', '__delattr__', '__dir__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', 
'__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__lt__', '__ne__', '__new__', 
'__next__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 
'_checkClosed', '_checkReadable', '_checkSeekable', '_checkWritable', 'close', 'fileno', 'flush', 'isatty', 
'read', 'readable', 'readall', 'readinto', 'readline', 'readlines', 'seek', 'seekable', 'tell', 'truncate', 
'writable', 'write', 'writelines']

i checked if the stream is writable, unfortunately i get False print (worker_socket.writable()) -> False

to me, it sounds like IO, but that's for "implementation of I/O streams" and not a socket ??

[jrcichra]

Having the exact same issue, did someone introduce Socket.IO into this instead of a python socket?

[jrcichra]

Here's what I came up with after a day of trying to figure this out:

import docker, tarfile
from io import BytesIO

client = docker.APIClient()

# create container
container = client.create_container(
    'ubuntu',
    stdin_open = True,
    # environment=["PS1=#"],
    command    = 'bash')
client.start(container)

# attach stdin to container and send data
s = client.attach_socket(container, params={'stdin': 1, 'stream': 1,'stdout':1,'stderr':1})

while True:
    original_text_to_send = input("$") + '\n'
    if(original_text_to_send == "exit\n"):
        s.close()
        break
    else:
        s._sock.send(original_text_to_send.encode('utf-8'))
        msg = s._sock.recv(1024)
        print(msg)
        print(len(msg))
        print('==================')
        print(msg.decode()[8:])

print("We're done here")
client.stop(container)
client.wait(container)
client.remove_container(container)

This creates an ubuntu container that starts up bash. It then attaches stdin, stdout, stderr, in a stream form. I made a little while loop to handle commands. I noticed the first 8 bytes are something special, maybe header data for SocketIO? Not sure. The formatted output strips off those 8 bytes.

[adalric]

Response does nothing on 101 status code, does it?

[christianbur]

so the above example with python3

import docker

client = docker.DockerClient(base_url='unix://var/run/docker.sock', version='auto')
container = client.containers.run("gliderlabs/alpine", command= "sleep 100", detach = True)
socket = container.exec_run(cmd="sh", stdin=True, socket = True)
print(socket)
socket.output._sock.send(b"ls\n")

#print socket
#socket.sendall(b"ls\n")

# a read block after a send
try:
    unknown_byte=socket.output._sock.recv(1)
    while 1:
        # note that os.read does not work
        # because it does not TLS-decrypt
        # but returns the low-level encrypted data
        # one must use "socket.recv" instead
        data = socket.output._sock.recv(16384)
        if not data: break
        print(data.decode('utf8'))
except so.timeout: pass

socket.output._sock.send(b"exit\n")

methodes for "socket.output._sock"

['__class__', '__del__', '__delattr__', '__dir__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_accept', '_check_sendfile_params', '_decref_socketios', '_real_close', '_sendfile_use_send', '_sendfile_use_sendfile', 'accept', 'bind', 'close', 'connect', 'connect_ex', 'detach', 'dup', 'fileno', 'get_inheritable', 'getpeername', 'getsockname', 'getsockopt', 'gettimeout', 'listen', 'makefile', 'recv', 'recv_into', 'recvfrom', 'recvfrom_into', 'recvmsg', 'recvmsg_into', 'send', 'sendall', 'sendfile', 'sendmsg', 'sendmsg_afalg', 'sendto', 'set_inheritable', 'setblocking', 'setsockopt', 'settimeout', 'shutdown']

[hellais]

I also ran into issues using the socket=True option to send data via as if it were coming from stdin.

I am using docker==3.7.1, where the syntax of container.exec_run is also different.

Here is a minimal code snippet to do what I needed:

    _, socket = container.exec_run(cmd="sh", stdin=True, socket=True)
    socket._sock.sendall(b"ls\n")

[Bustel]

Looking at the tests in api_exec_test.py the "unknown byte tells you whether its stdout (1), stdin (0) or stderr(2) and it is followed by a frame size.

    def test_exec_start_socket(self):
        container = self.client.create_container(TEST_IMG, 'cat',
                                                 detach=True, stdin_open=True)
        container_id = container['Id']
        self.client.start(container_id)
        self.tmp_containers.append(container_id)

        line = 'yay, interactive exec!'
        # `echo` appends CRLF, `printf` doesn't
        exec_id = self.client.exec_create(
            container_id, ['printf', line], tty=True)
        assert 'Id' in exec_id

        socket = self.client.exec_start(exec_id, socket=True)
        self.addCleanup(socket.close)

        (stream, next_size) = next_frame_header(socket)
        assert stream == 1  # stdout (0 = stdin, 1 = stdout, 2 = stderr)
        assert next_size == len(line)
        data = read_exactly(socket, next_size)
        assert data.decode('utf-8') == line

IMO This should be documented somewhere.

[Alphasite]

I think this is a bug with how _get_raw_response_socket is implemented, it does:

    def _get_raw_response_socket(self, response):
        self._raise_for_status(response)
        if self.base_url == "http+docker://localnpipe":
            sock = response.raw._fp.fp.raw.sock
        elif self.base_url.startswith('http+docker://ssh'):
            sock = response.raw._fp.fp.channel
        else:
            sock = response.raw._fp.fp.raw
            if self.base_url.startswith("https://"):
                sock = sock._sock

Where it gets the underlying socket for https urls, but when you get an http url it doesn't get the right socket. On docker for Mac, 'http+docker://localhost' also needs this check, since the actual socket of interest is the underlying ._sock.

One possible alternative this this may be to use:

    if hasattr(sock, "send"):
        return sock
    else:
        return sock._sock

[mschwager]

Could this issue be related to https://github.com/docker/docker-py/issues/1507 or https://github.com/docker/docker-py/issues/983? I've done some investigation there, but haven't come to any conclusions.

[matan1008]

Any news about this issue? why not just change

            sock = response.raw._fp.fp.raw
            if self.base_url.startswith("https://"):
                sock = sock._sock

to sock = response.raw._fp.fp.raw._sock?

I face this on Ubuntu 20.04, where response.raw._fp.fp.raw is s SocketIO object and not a socket.

[mschwager]

Any news about this issue? why not just change ... to sock = response.raw._fp.fp.raw._sock?

In my testing (https://github.com/docker/docker-py/issues/1507#issuecomment-765684307), _sock hangs when using "stdin": 1 in Python 3.

[matan1008]

I tried it with stdin=True, socket=True, tty=True and it worked (python 3.9), in my humble opinion the api here is really half baked.

[mschwager]

I found a fix here: https://github.com/docker/docker-py/issues/1507#issuecomment-901300117.

If you're calling any sock._sock.send, sock._sock.sendall, sock._sock.recv, etc, you must call sock._sock.close before sock.close. This will ensure that the underlying socket is closed correctly :+1:

[matan1008]

Accessing an internal variable is more of an hack than a solution, I would really expect the library to return a socket or at least a writable SocketIO object

[mschwager]

Could also change this to something like:

if self.base_url.startswith("https://") or self.base_url == "http+docker://localhost".

[xintenseapple]

So we are still sitting with an unwritable SocketIO object being returned. Neither the fact that the return is a SocketIO object nor the fact that it is unwritable is documented. There is no reason to be returning an unwritable SocketIO anyway, it defeats the purpose of having a socket to begin with.

[eriky]

Still not fixed? I'd like to use this lower level access but I'd prefer a less hacky method. Unfortunately, containerd libs for Python are also lacking / non-existant.