search

h2oai / h2ogpt

Private chat with local GPT with document, images, video, etc. 100% private, Apache 2.0. Supports oLLaMa, Mixtral, llama.cpp, and more. Demo: https://gpt.h2o.ai/ https://gpt-docs.h2o.ai/
http://h2o.ai
Apache License 2.0
11.38k stars 1.25k forks source link

segfaults and Could not allocate tuple object! when multiple threads access duckdb #378

Open pseudotensor opened 1 year ago

pseudotensor commented 1 year ago 
Generation Failed: Could not allocate tuple object!   File "/home/jon/h2ogpt/iterators/timeout_iterator.py", line 99, in __lookahead
    self._buffer.put(next(self._iterator))
  File "/home/jon/h2ogpt/gradio_runner.py", line 1221, in get_response
    for output_fun in fun1():
  File "/home/jon/h2ogpt/generate.py", line 1536, in evaluate
    for r in run_qa_db(query=query,
  File "/home/jon/h2ogpt/gpt_langchain.py", line 1831, in _run_qa_db
    docs, chain, scores, use_context = get_similarity_chain(**sim_kwargs)
  File "/home/jon/h2ogpt/gpt_langchain.py", line 2005, in get_similarity_chain
    db_get = db._collection.get(where=filter_kwargs.get('filter'))
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/api/models/Collection.py", line 141, in get
    return self._client._get(
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/api/local.py", line 358, in _get
    db_result = self._db.get(
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/db/clickhouse.py", line 494, in get
    val = self._get(where=where_str, columns=columns)
  File "/home/jon/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/db/duckdb.py", line 271, in _get
    ).fetchall()
Fatal Python error: Segmentation fault

Thread 0x00007f5c5c034700 (most recent call first):
  <no Python frame>

Thread 0x00007f5c5e835700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/socket.py", line 232 in __init__
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/util/connection.py", line 77 in create_connection
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/connection.py", line 169 in _new_conn
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/connection.py", line 200 in connect
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/http/client.py", line 975 in send
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/http/client.py", line 1037 in _send_output
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/http/client.py", line 1277 in endheaders
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/http/client.py", line 1328 in _send_request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/http/client.py", line 1282 in request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/connection.py", line 234 in request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/connectionpool.py", line 394 in _make_request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/connectionpool.py", line 699 in urlopen
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/adapters.py", line 489 in send
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/sessions.py", line 701 in send
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/sessions.py", line 587 in request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/api.py", line 59 in request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/api.py", line 73 in get
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 635 in get_llm
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1790 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c9304a700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/http/client.py", line 998 in send
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/http/client.py", line 1037 in _send_output
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/http/client.py", line 1277 in endheaders
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/http/client.py", line 1328 in _send_request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/http/client.py", line 1282 in request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/connection.py", line 234 in request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/connectionpool.py", line 394 in _make_request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/connectionpool.py", line 699 in urlopen
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/adapters.py", line 489 in send
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/sessions.py", line 701 in send
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/sessions.py", line 587 in request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/api.py", line 59 in request
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/api.py", line 73 in get
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 635 in get_llm
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1790 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c84044700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/db/duckdb.py", line 269 in _get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/db/clickhouse.py", line 494 in get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/api/local.py", line 358 in _get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/api/models/Collection.py", line 141 in get
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 2005 in get_similarity_chain
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1831 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f49c4ff9700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/db/duckdb.py", line 269 in _get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/db/clickhouse.py", line 494 in get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/api/local.py", line 358 in _get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/api/models/Collection.py", line 141 in get
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 2005 in get_similarity_chain
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1831 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Current thread 0x00007f5c7a040700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/db/duckdb.py", line 271 in _get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/db/clickhouse.py", line 494 in get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/api/local.py", line 358 in _get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/chromadb/api/models/Collection.py", line 141 in get
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 2005 in get_similarity_chain
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1831 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c7503e700 (most recent call first):
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 419 in _call
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/llms/base.py", line 499 in _generate
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/llms/base.py", line 199 in generate
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/llms/base.py", line 141 in generate_prompt
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/chains/llm.py", line 84 in generate
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/chains/llm.py", line 74 in _call
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/chains/base.py", line 143 in __call__
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/chains/llm.py", line 218 in predict
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/chains/combine_documents/stuff.py", line 87 in combine_docs
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/chains/combine_documents/base.py", line 84 in _call
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/chains/base.py", line 143 in __call__
  File "/home/ubuntu/h2ogpt/utils.py", line 338 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c8b046700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 320 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/queue.py", line 171 in get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 857 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c61036700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/filelock/_api.py", line 190 in acquire
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/filelock/_api.py", line 220 in __enter__
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 2026 in get_similarity_chain
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1831 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c63837700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/filelock/_api.py", line 190 in acquire
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/filelock/_api.py", line 220 in __enter__
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 2026 in get_similarity_chain
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1831 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c66038700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/filelock/_unix.py", line 36 in _acquire
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/filelock/_api.py", line 177 in acquire
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/filelock/_api.py", line 220 in __enter__
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 2026 in get_similarity_chain
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1831 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c68839700 (most recent call first):
  File "/home/ubuntu/h2ogpt/utils_langchain.py", line 57 in __next__
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1856 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c6b03a700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 489 in unshape
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 571 in forward
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 600 in forward
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 693 in forward
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1090 in forward
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1938 in forward
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/InstructorEmbedding/instructor.py", line 269 in forward
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217 in forward
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/InstructorEmbedding/instructor.py", line 539 in encode
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/embeddings/huggingface.py", line 171 in embed_query
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/langchain/vectorstores/chroma.py", line 229 in similarity_search_with_score
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 2027 in get_similarity_chain
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1831 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c6d83b700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/filelock/_api.py", line 190 in acquire
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/filelock/_api.py", line 220 in __enter__
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 2026 in get_similarity_chain
  File "/home/ubuntu/h2ogpt/gpt_langchain.py", line 1831 in _run_qa_db
  File "/home/ubuntu/h2ogpt/generate.py", line 1536 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c8b847700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 320 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/queue.py", line 171 in get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 857 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c8b847700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 320 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/queue.py", line 171 in get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 857 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c7283d700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/ssl.py", line 1130 in read
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/ssl.py", line 1274 in recv_into
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/socket.py", line 705 in readinto
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/response.py", line 694 in _update_chunk_length
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/response.py", line 764 in read_chunked
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/response.py", line 572 in stream
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/models.py", line 816 in generate
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/models.py", line 865 in iter_lines
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/openai/api_requestor.py", line 115 in parse_stream
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/openai/api_requestor.py", line 692 in <genexpr>
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 166 in <genexpr>
  File "/home/ubuntu/h2ogpt/generate.py", line 1653 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c7783f700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/socket.py", line 705 in readinto
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/response.py", line 694 in _update_chunk_length
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/response.py", line 764 in read_chunked
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/response.py", line 572 in stream
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/models.py", line 816 in generate
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/models.py", line 865 in iter_lines
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/text_generation/client.py", line 238 in generate_stream
  File "/home/ubuntu/h2ogpt/generate.py", line 1822 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c7c841700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/socket.py", line 705 in readinto
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/response.py", line 694 in _update_chunk_length
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/response.py", line 764 in read_chunked
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/urllib3/response.py", line 572 in stream
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/models.py", line 816 in generate
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/requests/models.py", line 865 in iter_lines
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/text_generation/client.py", line 238 in generate_stream
  File "/home/ubuntu/h2ogpt/generate.py", line 1822 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c88845700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 320 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 607 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 940 in start
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 34 in __init__
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1295 in all_bot
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/utils.py", line 317 in run_sync_iterator_async
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5c8e048700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 320 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/queue.py", line 171 in get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 857 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5cc485d700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f566bc71700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5659fff700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 324 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 607 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/fsspec/asyn.py", line 88 in sync
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio_client/utils.py", line 479 in synchronize_async
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio_client/client.py", line 659 in _predict
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio_client/client.py", line 630 in _inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f57777cb700 (most recent call first):
  File "/home/ubuntu/h2ogpt/generate.py", line 1769 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f588292c700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 324 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/queue.py", line 180 in get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/posthog/consumer.py", line 104 in next
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/posthog/consumer.py", line 73 in upload
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/posthog/consumer.py", line 62 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f57777cb700 (most recent call first):
  File "/home/ubuntu/h2ogpt/generate.py", line 1769 in evaluate
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 1221 in get_response
  File "/home/ubuntu/h2ogpt/iterators/timeout_iterator.py", line 99 in __lookahead
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f588292c700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 324 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/queue.py", line 180 in get
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/posthog/consumer.py", line 104 in next
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/posthog/consumer.py", line 73 in upload
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/posthog/consumer.py", line 62 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5891d66700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 324 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 607 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f589affd700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f58997fa700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5892ffd700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5b5d6fa700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f589bfff700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5b5eefd700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f589bfff700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5b5eefd700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5b8242d700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/asyncio/runners.py", line 44 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/uvicorn/server.py", line 59 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5b83c51700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 324 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 607 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/apscheduler/schedulers/blocking.py", line 30 in _main_loop
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5b87581700 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 324 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 607 in wait
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/fsspec/asyn.py", line 88 in sync
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio_client/utils.py", line 479 in synchronize_async
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio_client/client.py", line 659 in _predict
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio_client/client.py", line 630 in _inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 953 in run
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f5ce0dac280 (most recent call first):
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/gradio/blocks.py", line 2058 in block_thread
  File "/home/ubuntu/h2ogpt/gradio_runner.py", line 2064 in go_gradio
  File "/home/ubuntu/h2ogpt/generate.py", line 633 in main
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/fire/core.py", line 691 in _CallAndUpdateTrace
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/fire/core.py", line 475 in _Fire
  File "/home/ubuntu/miniconda3/envs/h2ollm/lib/python3.10/site-packages/fire/core.py", line 141 in Fire
  File "/home/ubuntu/h2ogpt/generate.py", line 2514 in entrypoint_main
  File "/home/ubuntu/h2ogpt/generate.py", line 2518 in <module>

Extension modules: simplejson._speedups, charset_normalizer.md, psutil._psutil_linux, psutil._psutil_posix, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_int>
pseudotensor commented 1 year ago

Still occurs despite lock, but a single model doesn't hit. So still some kind of concurrency issue of some kind.