Reconnection issue

[andreyohiienko]

Hi, I really enjoy your TWS image, but I've faced with one problem recently. At some point in time connections start failing. The UI keeps running ok, but I can't connect to API. Here is my container logs:

2024/01/25 15:07:07 socat[12] N accepting connection from AF=2 100.100.1.137:47806 on AF=2 100.100.207.15:8888 2024/01/25 15:07:07 socat[12] N forked off child process 835 2024/01/25 15:07:07 socat[12] N listening on AF=2 0.0.0.0:8888 2024/01/25 15:07:07 socat[835] N opening connection to AF=2 127.0.0.1:7497 2024/01/25 15:07:07 socat[835] N successfully connected from local address AF=2 127.0.0.1:51930 2024/01/25 15:07:07 socat[835] N starting data transfer loop with FDs [6,6] and [5,5] 2024/01/25 15:07:11 socat[835] N socket 1 (fd 6) is at EOF 2024/01/25 15:07:11 socat[835] N exiting with status 0 2024/01/25 15:07:11 socat[12] N childdied(): handling signal 17

and ib insync returns:

2024-01-25 15:07:07,189 ib_insync.client INFO Connecting to 52.139.19.233:8888 with clientId 101... 2024-01-25 15:07:07,242 ib_insync.client INFO Connected 2024-01-25 15:07:11,246 ib_insync.client INFO Disconnecting 2024-01-25 15:07:11,246 ib_insync.client ERROR API connection failed: TimeoutError() 2024-01-25 15:07:11,247 ib_insync.client INFO Disconnected. Traceback (most recent call last): File "C:\Python310\lib\asyncio\tasks.py", line 650, in _wrap_awaitable return (yield from awaitable.await()) asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Python310\lib\asyncio\tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\AndriiO18\projects\tws-tests\ib.py", line 63, in ds = DataStream(1) File "C:\Users\AndriiO18\projects\tws-tests\ib.py", line 38, in init self.ib = IB().connect(c_id) File "C:\Users\AndriiO18\projects\tws-tests\ib.py", line 20, in connect self.ib.connect( File "C:\Users\AndriiO18\projects\tws-tests.venv\lib\site-packages\ib_insync\ib.py", line 279, in connect return self._run(self.connectAsync( File "C:\Users\AndriiO18\projects\tws-tests.venv\lib\site-packages\ib_insync\ib.py", line 318, in _run return util.run(*awaitables, timeout=self.RequestTimeout) File "C:\Users\AndriiO18\projects\tws-tests.venv\lib\site-packages\ib_insync\util.py", line 341, in run result = loop.run_until_complete(task) File "C:\Python310\lib\asyncio\base_events.py", line 646, in run_until_complete return future.result() File "C:\Users\AndriiO18\projects\tws-tests.venv\lib\site-packages\ib_insync\ib.py", line 1748, in connectAsync await self.client.connectAsync(host, port, clientId, timeout) File "C:\Users\AndriiO18\projects\tws-tests.venv\lib\site-packages\ib_insync\client.py", line 217, in connectAsync await asyncio.wait_for(self.apiStart, timeout) File "C:\Python310\lib\asyncio\tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError

The problem is I can't catch the real reason of this problem because it can run without any problems for hours. I used recent ghcr.io/extrange/ibkr:stable then downgraded to ghcr.io/extrange/ibkr:10.19.2d but the problem appeared there too. And only container restart fixes the problem. Both TWS and IB GW have this issue.

Will be grateful for any advice. Thanks.

[andreyohiienko]

https://github.com/erdewit/ib_insync/issues/386#issuecomment-1001161053 most likely it is connected to this issue But how to prevent it is a question.

[andreyohiienko]

lsof -i :4002 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 50 root 56u IPv6 137291 0t0 TCP localhost:4002->localhost:46418 (ESTABLISHED) java 50 root 60u IPv6 48900 0t0 TCP *:4002 (LISTEN) java 50 root 78u IPv6 238033 0t0 TCP localhost:4002->localhost:43734 (ESTABLISHED) java 50 root 80u IPv6 310694 0t0 TCP localhost:4002->localhost:39618 (CLOSE_WAIT) < ==== THIS ONE socat 260 root 5u IPv4 138298 0t0 TCP localhost:46418->localhost:4002 (ESTABLISHED) socat 1618 root 5u IPv4 237438 0t0 TCP localhost:43734->localhost:4002 (ESTABLISHED)

is there any way to prevent this?

[dchien234]

Encountered the same issue, not sure how to solve this. When I tried to kill the process that has CLOSE_WAIT connection, the docker container died 😭. Have u managed to find a solution for this issue?

[andreyohiienko]

@dchien234 No ( I think killing the process is not a solution. The process should not block the port after tws disconnection.

[extrange]

erdewit/ib_insync#386 (comment) most likely it is connected to this issue But how to prevent it is a question.

Yea, it is probably related to ib_insync, and not related to wrapping IBKR in docker. Closing for now.