Previously working implementations timeout as soon as TWS asks to allow

I had a previously working implementation of ib_insync that started giving "Failed to connect to IB gateway/ Timeout Error" as soon as the TWS popup asks if I want to allow the connection about 2 or 3 weeks ago. I've been going crazy trying to track down the exact cause and was shocked to find the news about ib_insync's creator, and then even more shock to have the same issue when using this new and updated library. Whitelisting doesn't help and as a way to rule out my implementation, I have the same issue with @mattsta 's icli (and also I ran the example code in a jupiter notebook to triple check, same results output follows:

2024-05-06 14:04:21,259 ib_async.client INFO Connecting to 192.168.50.203:7497 with clientId 5280...
2024-05-06 14:04:21,266 ib_async.client INFO Connected
2024-05-06 14:04:25,271 ib_async.client INFO Disconnecting
2024-05-06 14:04:25,274 ib_async.client ERROR API connection failed: TimeoutError()

---------------------------------------------------------------------------
CancelledError                            Traceback (most recent call last)
File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py:495](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py#line=494), in wait_for(fut, timeout)
    494 try:
--> 495     return fut.result()
    496 except exceptions.CancelledError as exc:

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/futures.py:204](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/futures.py#line=203), in Future.result(self)
    203     exc = self._make_cancelled_error()
--> 204     raise exc
    205 if self._state != _FINISHED:

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py:274](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py#line=273), in Task.__step(***failed resolving arguments***)
    273     else:
--> 274         result = coro.throw(exc)
    275 except StopIteration as exc:

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py:689](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py#line=688), in _wrap_awaitable(awaitable)
    684 """Helper for asyncio.ensure_future().
    685 
    686 Wraps awaitable (an object with __await__) into a coroutine
    687 that will later be wrapped in a Task by ensure_future().
    688 """
--> 689 return (yield from awaitable.__await__())

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/futures.py:293](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/futures.py#line=292), in Future.__await__(self)
    292     self._asyncio_future_blocking = True
--> 293     yield self  # This tells Task to wait for completion.
    294 if not self.done():

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py:344](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py#line=343), in Task.__wakeup(self, future)
    343 try:
--> 344     future.result()
    345 except BaseException as exc:
    346     # This may also be a cancellation.

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/futures.py:204](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/futures.py#line=203), in Future.result(self)
    203     exc = self._make_cancelled_error()
--> 204     raise exc
    205 if self._state != _FINISHED:

CancelledError: 

The above exception was the direct cause of the following exception:

TimeoutError                              Traceback (most recent call last)
Cell In[13], line 2
      1 ib = IB()
----> 2 ib.connect('192.168.50.203', 7497, clientId=5280)

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/ib_async/ib.py:337](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/ib_async/ib.py#line=336), in IB.connect(self, host, port, clientId, timeout, readonly, account, raiseSyncErrors)
    305 def connect(
    306     self,
    307     host: str = "127.0.0.1",
   (...)
    313     raiseSyncErrors: bool = False,
    314 ):
    315     """
    316     Connect to a running TWS or IB gateway application.
    317     After the connection is made the client is fully synchronized
   (...)
    335           When ``False`` the error will only be logged at error level.
    336     """
--> 337     return self._run(
    338         self.connectAsync(
    339             host, port, clientId, timeout, readonly, account, raiseSyncErrors
    340         )
    341     )

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/ib_async/ib.py:381](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/ib_async/ib.py#line=380), in IB._run(self, *awaitables)
    380 def _run(self, *awaitables: Awaitable):
--> 381     return util.run(*awaitables, timeout=self.RequestTimeout)

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/ib_async/util.py:357](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/ib_async/util.py#line=356), in run(timeout, *awaitables)
    355 globalErrorEvent.connect(onError)
    356 try:
--> 357     result = loop.run_until_complete(task)
    358 except asyncio.CancelledError as e:
    359     raise globalErrorEvent.value() or e

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/nest_asyncio.py:98](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/nest_asyncio.py#line=97), in _patch_loop.<locals>.run_until_complete(self, future)
     95 if not f.done():
     96     raise RuntimeError(
     97         'Event loop stopped before Future completed.')
---> 98 return f.result()

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/futures.py:209](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/futures.py#line=208), in Future.result(self)
    207 self.__log_traceback = False
    208 if self._exception is not None:
--> 209     raise self._exception.with_traceback(self._exception_tb)
    210 return self._result

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py:272](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py#line=271), in Task.__step(***failed resolving arguments***)
    268 try:
    269     if exc is None:
    270         # We use the `send` method directly, because coroutines
    271         # don't have `__iter__` and `__next__` methods.
--> 272         result = coro.send(None)
    273     else:
    274         result = coro.throw(exc)

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/ib_async/ib.py:1973](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/ib_async/ib.py#line=1972), in IB.connectAsync(self, host, port, clientId, timeout, readonly, account, raiseSyncErrors)
   1970 timeout = timeout or None
   1971 try:
   1972     # establish API connection
-> 1973     await self.client.connectAsync(host, port, clientId, timeout)
   1975     # autobind manual orders
   1976     if clientId == 0:

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/ib_async/client.py:228](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/site-packages/ib_async/client.py#line=227), in Client.connectAsync(self, host, port, clientId, timeout)
    219     msg = b"API\0" + self._prefix(
    220         b"v%d..%d%s"
    221         % (
   (...)
    225         )
    226     )
    227     self.conn.sendMsg(msg)
--> 228     await asyncio.wait_for(self.apiStart, timeout)
    229     self._logger.info("API connection ready")
    230 except BaseException as e:

File [~/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py:497](http://localhost:8888/home/lc/.pyenv/versions/3.11.0rc1/lib/python3.11/asyncio/tasks.py#line=496), in wait_for(fut, timeout)
    495             return fut.result()
    496         except exceptions.CancelledError as exc:
--> 497             raise exceptions.TimeoutError() from exc
    498 finally:
    499     timeout_handle.cancel()

TimeoutError: 

2024-05-06 14:04:25,965 ib_async.client INFO Disconnected.

Any feedback would be VERY appreciated, strangely the ruby IBKR library that hasn't been updated since 2016 connects properly to my TWS (and even submits trades successfully) so I know it TWS still set up correctly, but I am starting to think I need to reach out and see if I'm the only one.

P.S. I was previously using it with older python versions (3.10) but as you can see above, 3.11 also didn't work.

Oh also I did try hardcoding a longer timeout (double and triple), but that didn't make a difference at all since the Timeout comes the instant the connection opens (attempts to open) on TWS

Those do sound like odd errors. The systems are currently working for me, so I don't think anything is critically broken with the python libraries themselves. Sometimes the TWS or IBKR gateway breaks and requires a full exit/restart cycle to fix itself, but it sounds like you tried across multiple days already.

Are you sure the client id is unique everywhere?

Did you try maybe running the gateway only and not a full TWS?

I assume your ruby tests are also using the same network address? So you're connecting to a remote system and not just something over localhost?

Initial suggestions:

verify the remote host is actually reachable (no firewall rules causing problems, etc)
try different client ids (or even client id 0 which controls everything)

upgrade your python version from 3.11.0rc1 to the latest 3.11 or 3.12 and install the dependencies again:

$ pyenv install -l |grep 3.1\[12]
3.11.0
3.11-dev
3.11.1
3.11.2
3.11.3
3.11.4
3.11.5
3.11.6
3.11.7
3.11.8
3.11.9
3.12.0
3.12-dev
3.12.1
3.12.2
3.12.3

Thank you for the quick response, I had suspected I should/could be using a newer version of python but my pyenv even after updating (and installing again by following the docs) shows this as the result

  3.11.0rc1
  3.11-dev
  3.12-dev
  miniconda3-4.3.11

I'm going to keep digging, but if you have any idea why this would be happening. My guess was whatever debian or ubuntu distro I'm using here is too old and it was installed from the pkg manager, which is why I tried installing from scratch as per the docs. Maybe I'll nuke ~.pyenv and start over...

Ok, scratch that last comment, I have confirmed I get the same error on 3.11.9

Traceback (most recent call last):
  File "/home/lou/.pyenv/versions/3.11.9/lib/python3.11/asyncio/tasks.py", line 500, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/home/lou/.pyenv/versions/3.11.9/lib/python3.11/asyncio/tasks.py", line 694, in _wrap_awaitable
    return (yield from awaitable.__await__())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lou/.pyenv/versions/3.11.9/lib/python3.11/site-packages/ib_async/ib.py", line 337, in connect
    return self._run(
           ^^^^^^^^^^
  File "/home/lou/.pyenv/versions/3.11.9/lib/python3.11/site-packages/ib_async/ib.py", line 381, in _run
    return util.run(*awaitables, timeout=self.RequestTimeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lou/.pyenv/versions/3.11.9/lib/python3.11/site-packages/ib_async/util.py", line 357, in run
    result = loop.run_until_complete(task)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lou/.pyenv/versions/3.11.9/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/lou/.pyenv/versions/3.11.9/lib/python3.11/site-packages/ib_async/ib.py", line 1973, in connectAsync
    await self.client.connectAsync(host, port, clientId, timeout)
  File "/home/lou/.pyenv/versions/3.11.9/lib/python3.11/site-packages/ib_async/client.py", line 228, in connectAsync
    await asyncio.wait_for(self.apiStart, timeout)
  File "/home/lou/.pyenv/versions/3.11.9/lib/python3.11/asyncio/tasks.py", line 502, in wait_for
    raise exceptions.TimeoutError() from exc
TimeoutError

I haven't yet tried the standalone gateway, I'd like to ideally avoid it since I would use my implementation side by side with TWS running.

Also, to answer your other questions I do usually use random to generate the clientId, and I always try a few times and also hard code it to something unique.

Yes, this is over my local network so not just localhost but I know there are no firewall issues because the ruby library connects from all the different machines I've tried (just like ib_insync did for over 2 years lol).

Looks like I need to really dig into the asyncio lib and figure this out :-|

I also tried python v 3.12.3 by using icli and poetry and reproduced the problem there.

2024-05-15 23:39:02.093 | INFO     | icli.helpers:<module>:53 - Futures Next Roll-Forward Date: 2024-06-17
2024-05-15 23:39:02.922 | INFO     | icli.cli:reconnect:2680 - Connecting to IBKR API...
2024-05-15 23:39:07.039 | ERROR    | icli.cli:reconnect:2741 - [] Failed to connect to IB Gateway, trying again...
2024-05-15 23:39:14.050 | ERROR    | icli.cli:reconnect:2741 - [] Failed to connect to IB Gateway, trying again...
2024-05-15 23:39:21.062 | ERROR    | icli.cli:reconnect:2741 - [] Failed to connect to IB Gateway, trying again...
2024-05-15 23:39:28.072 | ERROR    | icli.cli:reconnect:2741 - [] Failed to connect to IB Gateway, trying again...
^C2024-05-15 23:39:33.335 | ERROR    | icli.cli:reconnect:2741 - [] Failed to connect to IB Gateway, trying again...
^C2024-05-15 23:39:34.196 | WARNING  | icli.cli:reconnect:2751 - Exit requested during sleep. Goodbye.

Just baffled because it does seem like it should be some misconfiguration in my TWS but I haven't changed the settings in months (and they are backed up) plus the whole ruby library working in the same situation leads me to think its something in asyncio

maybe its better if you share part of your code that relates to connecting.. and also share your TWS API config settings..

from what i can gather above i'm still not very sure what's the steps to replicate your scenario.. is the prompt asking you to allow connection to TWS still popping up? or since 2 weeks ago you seen it its no longer showing?

ib-api-reloaded / ib_async

Previously working implementations timeout as soon as TWS asks to allow #18