I'm currently using the GithubRepositoryReader with llama-hub version 0.0.66 to read documents from a private Github repo which has a lot of markdown documents in. I'm finding that I'm often getting httpx.ConnectTimeout errors and it seems like the default timeout for httpx is only 5 seconds (https://www.python-httpx.org/advanced/#timeout-configuration).
Is there any way which we can set the timeout easily?
Version
llama-index: 0.9.25.post1, llama-hub: 0.0.66
Steps to Reproduce
I'm only getting this in times where my connection is poor. But I'm not 100% sure if that is the cause of this issue.
Relevant Logs/Tracbacks
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 189, in connect_tcp
addr_obj = ip_address(remote_host)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/ipaddress.py", line 54, in ip_address
raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: 'api.github.com' does not appear to be an IPv4 or IPv6 address
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 192, in connect_tcp
gai_res = await getaddrinfo(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 339, in __wakeup
future.result()
File "/usr/local/lib/python3.11/asyncio/futures.py", line 198, in result
raise exc
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
yield
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp
with anyio.fail_after(timeout):
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/anyio/_core/_tasks.py", line 119, in __exit__
raise TimeoutError
TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py", line 67, in map_httpcore_exceptions
yield
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py", line 371, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 268, in handle_async_request
raise exc
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 251, in handle_async_request
response = await connection.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
raise exc
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
stream = await self._connect(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_async/connection.py", line 124, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_backends/auto.py", line 30, in connect_tcp
return await self._backend.connect_tcp(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 112, in connect_tcp
with map_exceptions(exc_map):
File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectTimeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/app/reginald/models/create_index.py", line 86, in main
data_creator.create_index()
File "/app/reginald/models/models/llama_index.py", line 471, in create_index
self.prep_documents()
File "/app/reginald/models/models/llama_index.py", line 223, in prep_documents
self._load_turing_way(gh_token)
File "/app/reginald/models/models/llama_index.py", line 349, in _load_turing_way
self.documents.extend(turing_way_loader.load_data(branch="main"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/llama_hub/github_repo/base.py", line 287, in load_data
return self._load_data_from_branch(branch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/llama_hub/github_repo/base.py", line 254, in _load_data_from_branch
blobs_and_paths = self._loop.run_until_complete(self._recurse_tree(tree_sha))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/nest_asyncio.py", line 99, in run_until_complete
return f.result()
^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/futures.py", line 203, in result
raise self._exception.with_traceback(self._exception_tb)
File "/usr/local/lib/python3.11/asyncio/tasks.py", line 267, in __step
result = coro.send(None)
^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/llama_hub/github_repo/base.py", line 351, in _recurse_tree
await self._recurse_tree(
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/llama_hub/github_repo/base.py", line 351, in _recurse_tree
await self._recurse_tree(
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/llama_hub/github_repo/base.py", line 351, in _recurse_tree
await self._recurse_tree(
[Previous line repeated 1 more time]
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/llama_hub/github_repo/base.py", line 321, in _recurse_tree
tree_data: GitTreeResponseModel = await self._github_client.get_tree(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/llama_hub/github_repo/github_client.py", line 376, in get_tree
await self.request(
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/llama_hub/github_repo/github_client.py", line 320, in request
raise excp
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/llama_hub/github_repo/github_client.py", line 315, in request
response = await _client.request(
^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1559, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1646, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1674, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1711, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpx/_client.py", line 1748, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py", line 370, in handle_async_request
with map_httpcore_exceptions():
File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/root/.cache/pypoetry/virtualenvs/reginald-9TtSrW0h-py3.11/lib/python3.11/site-packages/httpx/_transports/default.py", line 84, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectTimeout
HTTP Exception for https://api.github.com/repos/the-turing-way/the-turing-way/git/trees/0537ea188689c1aebe741777b557f3da02a5df37 -
Bug Description
I'm currently using the
GithubRepositoryReaderwithllama-hubversion 0.0.66 to read documents from a private Github repo which has a lot of markdown documents in. I'm finding that I'm often gettinghttpx.ConnectTimeouterrors and it seems like the default timeout forhttpxis only 5 seconds (https://www.python-httpx.org/advanced/#timeout-configuration).Is there any way which we can set the timeout easily?
Version
llama-index: 0.9.25.post1,llama-hub: 0.0.66Steps to Reproduce
I'm only getting this in times where my connection is poor. But I'm not 100% sure if that is the cause of this issue.
Relevant Logs/Tracbacks