search

run-llama / llama_index

LlamaIndex is a data framework for your LLM applications
https://docs.llamaindex.ai
MIT License
35.35k stars 4.98k forks source link

[Question]: ImportError: cannot import name 'Trainer' from partially initialized module 'span_marker.trainer' #15155

Open aclifton314 opened 1 month ago

aclifton314 commented 1 month ago

Question Validation

Question

llama-index: 0.10.59 python: 3.11.9 Ubunutu 22.04

I'm getting the following error and I'm not really sure how to correct it.

Traceback (most recent call last):
  File "/home/aclifton/thdt04/llamaindex-tool-routing/evaluate.py", line 41, in <module>
    queries = flare.gen_queries()
              ^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/thdt04/llamaindex-tool-routing/thor_flare.py", line 103, in gen_queries
    data_generator = DatasetGenerator.from_documents(self.docs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/evaluation/dataset_generation.py", line 212, in from_documents
    node_with_scores = node_postprocessor.postprocess_nodes(node_with_scores)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/postprocessor/types.py", line 56, in postprocess_nodes
    return self._postprocess_nodes(nodes, query_bundle)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 235, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/postprocessor/node.py", line 47, in _postprocess_nodes
    nlp = spacy.blank(self.lang)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/spacy/__init__.py", line 79, in blank
    return LangClass.from_config(config, vocab=vocab, meta=meta)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/spacy/language.py", line 1855, in from_config
    nlp = lang_cls(
          ^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/spacy/language.py", line 188, in __init__
    util.registry._entry_point_factories.get_all()
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/catalogue/__init__.py", line 110, in get_all
    result.update(self.get_entry_points())
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/catalogue/__init__.py", line 125, in get_entry_points
    result[entry_point.name] = entry_point.load()
                               ^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/span_marker/__init__.py", line 16, in <module>
    from span_marker.trainer import Trainer
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/span_marker/trainer.py", line 19, in <module>
    from span_marker.evaluation import compute_f1_via_seqeval
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/span_marker/evaluation.py", line 4, in <module>
    import evaluate
  File "/home/aclifton/thdt04/llamaindex-tool-routing/evaluate.py", line 41, in <module>
    queries = flare.gen_queries()
              ^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/thdt04/llamaindex-tool-routing/thor_flare.py", line 103, in gen_queries
    data_generator = DatasetGenerator.from_documents(self.docs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/evaluation/dataset_generation.py", line 212, in from_documents
    node_with_scores = node_postprocessor.postprocess_nodes(node_with_scores)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/postprocessor/types.py", line 56, in postprocess_nodes
    return self._postprocess_nodes(nodes, query_bundle)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 235, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/postprocessor/node.py", line 47, in _postprocess_nodes
    nlp = spacy.blank(self.lang)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/spacy/__init__.py", line 79, in blank
    return LangClass.from_config(config, vocab=vocab, meta=meta)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/spacy/language.py", line 1855, in from_config
    nlp = lang_cls(
          ^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/spacy/language.py", line 188, in __init__
    util.registry._entry_point_factories.get_all()
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/catalogue/__init__.py", line 110, in get_all
    result.update(self.get_entry_points())
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/catalogue/__init__.py", line 125, in get_entry_points
    result[entry_point.name] = entry_point.load()
                               ^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/span_marker/__init__.py", line 16, in <module>
    from span_marker.trainer import Trainer
ImportError: cannot import name 'Trainer' from partially initialized module 'span_marker.trainer' (most likely due to a circular import) (/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/span_marker/trainer.py)
/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpx9_9e4t2'>
  _warnings.warn(warn_message, ResourceWarning)
logan-markewich commented 1 month ago

Seems like an issue with space or span marker, unrelated to llama-index?

aclifton314 commented 1 month ago

@logan-markewich Ok. I seem to be getting the error on a line:

data_generator = DatasetGenerator.from_documents(llama_docs)

After I uninstalled spacy, I get the following error (to be expected):

ImportError: Spacy is not installed, please install it with `pip install spacy`.

However, when I do the following:

data_generator = DatasetGenerator.from_documents(llama_docs, llm=Settings.llm)

where Setting.llm is set to an OpenAILike LLM, I get the same error above. Any ideas about what might be going on?

aclifton314 commented 1 month ago

I am coming across some strange behavior that I hope others might be able to chime in on. Here is some code I am running from a file called scratch.py:

from llama_index.core.evaluation import DatasetGenerator
from llama_index.core import SimpleDirectoryReader
from llama_index.llms.openai_like import OpenAILike

myllm = OpenAILike(
    model="microsoft/Phi-3-medium-128k-instruct",
    api_base="http://my/model/location",
    api_key="fake",
    max_tokens=512,
    logprobs=False,
    additional_kwargs={"stop": ["Observation:"]},
)

print("SimpleDirectoryReader...")
reader = SimpleDirectoryReader("/path/to/my/data")

print("loading data...")
documents = reader.load_data()

print("DatasetGenerator.from_documents...")
data_generator = DatasetGenerator.from_documents(documents, llm=myllm)

# print("generating questions from nodes...")
# eval_questions = data_generator.generate_questions_from_nodes()
# print(eval_questions)

Here is the output I get:

SimpleDirectoryReader...
loading data...
DatasetGenerator.from_documents...
setting embedding model
setting llm
creating SimpleDirectoryReader
creating weaviate client
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/urllib3/connection.py", line 196, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/urllib3/connectionpool.py", line 495, in _make_request
    conn.request(
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/urllib3/connection.py", line 398, in request
    self.endheaders()
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/http/client.py", line 1298, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/http/client.py", line 1058, in _send_output
    self.send(msg)
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/http/client.py", line 996, in send
    self.connect()
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/urllib3/connection.py", line 236, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/urllib3/connection.py", line 211, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fec81321b50>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /v1/.well-known/ready (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fec81321b50>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/weaviate/connect/connection.py", line 652, in wait_for_weaviate
    requests.get(
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /v1/.well-known/ready (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fec81321b50>: Failed to establish a new connection: [Errno 111] Connection refused'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aclifton/thdt04/llamaindex-tool-routing/scratch.py", line 111, in <module>
    data_generator = DatasetGenerator.from_documents(documents, llm=myllm)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/evaluation/dataset_generation.py", line 212, in from_documents
    node_with_scores = node_postprocessor.postprocess_nodes(node_with_scores)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/postprocessor/types.py", line 56, in postprocess_nodes
    return self._postprocess_nodes(nodes, query_bundle)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 235, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/postprocessor/node.py", line 47, in _postprocess_nodes
    nlp = spacy.blank(self.lang)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/spacy/__init__.py", line 79, in blank
    return LangClass.from_config(config, vocab=vocab, meta=meta)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/spacy/language.py", line 1855, in from_config
    nlp = lang_cls(
          ^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/spacy/language.py", line 188, in __init__
    util.registry._entry_point_factories.get_all()
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/catalogue/__init__.py", line 110, in get_all
    result.update(self.get_entry_points())
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/catalogue/__init__.py", line 125, in get_entry_points
    result[entry_point.name] = entry_point.load()
                               ^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/span_marker/__init__.py", line 16, in <module>
    from span_marker.trainer import Trainer
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/span_marker/trainer.py", line 19, in <module>
    from span_marker.evaluation import compute_f1_via_seqeval
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/span_marker/evaluation.py", line 4, in <module>
    import evaluate
  File "/home/aclifton/thdt04/llamaindex-tool-routing/evaluate.py", line 77, in <module>
    flare = ThorFlare()
            ^^^^^^^^^^^
  File "/home/aclifton/thdt04/llamaindex-tool-routing/thor_flare.py", line 58, in __init__
    self.client = weaviate.Client("http://localhost:8080")
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/weaviate/client.py", line 150, in __init__
    self._connection = Connection(
                       ^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/weaviate/connect/connection.py", line 166, in __init__
    self.wait_for_weaviate(startup_period)
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/weaviate/connect/connection.py", line 657, in wait_for_weaviate
    raise WeaviateStartUpError(
weaviate.exceptions.WeaviateStartUpError: Weaviate did not start up in 5 seconds. Either the Weaviate URL http://localhost:8080 is wrong or Weaviate did not start up in the interval given in 'startup_period'.
/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/tempfile.py:934: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpvf1aquvu'>
  _warnings.warn(warn_message, ResourceWarning)

I understand the error as I don't have the weaviate docker running. But what is interesting to me are these lines:

SimpleDirectoryReader...
loading data...
DatasetGenerator.from_documents...
setting embedding model
setting llm
creating SimpleDirectoryReader
creating weaviate client

because the lines:

setting embedding model
setting llm
creating SimpleDirectoryReader
creating weaviate client

come from a completely different file that is in the same directory as scratch.py. I shutdown my machine and restarted it just to see if that might fix anything. I then started my venv and ran python scratch.py and get the same result above.

Any thoughts about what might be going on here? I would expect that scratch.py runs independently and doesn't try to call the other file.

Thanks in advance for your help!

aclifton314 commented 1 month ago

I've got an update. I moved the scratch.py file to a different directory and that seemed to fix the issue of calling the other file. I am now getting a new error below:

SimpleDirectoryReader...
loading data...
DatasetGenerator.from_documents...
/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/evaluation/dataset_generation.py:215: DeprecationWarning: Call to deprecated class DatasetGenerator. (Deprecated in favor of `RagDatasetGenerator` which should be used instead.)
  return cls(
generating questions from nodes...
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpcore/_async/http11.py", line 143, in handle_async_request
    raise exc
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpcore/_async/http11.py", line 113, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpcore/_async/http11.py", line 186, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpcore/_async/http11.py", line 224, in _receive_event
    data = await self._network_stream.read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 32, in read
    with map_exceptions(exc_map):
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/_base_client.py", line 1563, in _request
    response = await self._client.send(
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpx/_client.py", line 1661, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpx/_client.py", line 1689, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpx/_client.py", line 1763, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpx/_transports/default.py", line 372, in handle_async_request
    with map_httpcore_exceptions():
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aclifton/thdt04/scratch.py", line 24, in <module>
    eval_questions = data_generator.generate_questions_from_nodes(1)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/evaluation/dataset_generation.py", line 331, in generate_questions_from_nodes
    return asyncio_run(self.agenerate_questions_from_nodes(num=num))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/async_utils.py", line 33, in asyncio_run
    return loop.run_until_complete(coro)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/evaluation/dataset_generation.py", line 316, in agenerate_questions_from_nodes
    dataset = await self._agenerate_dataset(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/evaluation/dataset_generation.py", line 270, in _agenerate_dataset
    responses = await async_module.gather(*query_tasks)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/base/base_query_engine.py", line 64, in aquery
    query_result = await self._aquery(str_or_query_bundle)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/query_engine/retriever_query_engine.py", line 206, in _aquery
    response = await self._response_synthesizer.asynthesize(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/response_synthesizers/base.py", line 316, in asynthesize
    response_str = await self.aget_response(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/response_synthesizers/compact_and_refine.py", line 23, in aget_response
    return await super().aget_response(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/response_synthesizers/refine.py", line 363, in aget_response
    response = await self._agive_response_single(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/response_synthesizers/refine.py", line 482, in _agive_response_single
    structured_response = await program.acall(
                          ^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/response_synthesizers/refine.py", line 99, in acall
    answer = await self._llm.apredict(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/llms/llm.py", line 652, in apredict
    response = await self.acomplete(formatted_prompt, formatted=True)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/llms/openai_like/base.py", line 127, in acomplete
    return await super().acomplete(prompt, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in async_wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/core/llms/callbacks.py", line 334, in wrapped_async_llm_predict
    f_return_val = await f(_self, *args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/llms/openai/base.py", line 652, in acomplete
    return await acomplete_fn(prompt, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/llama_index/llms/openai/base.py", line 770, in _acomplete
    response = await aclient.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/resources/completions.py", line 1059, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/_base_client.py", line 1836, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/_base_client.py", line 1524, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/_base_client.py", line 1572, in _request
    return await self._retry_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/_base_client.py", line 1657, in _retry_request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/_base_client.py", line 1572, in _request
    return await self._retry_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/_base_client.py", line 1657, in _retry_request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/_base_client.py", line 1572, in _request
    return await self._retry_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/_base_client.py", line 1657, in _retry_request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/aclifton/anaconda3/envs/thdt04/lib/python3.11/site-packages/openai/_base_client.py", line 1582, in _request
    raise APITimeoutError(request=request) from err
openai.APITimeoutError: Request timed out.

Any ideas about what might be happening? It looks to be coming from the LLM but I can't tell for sure.