search

katanaml / sparrow

Data processing with ML, LLM and Vision LLM
https://katanaml.io
GNU General Public License v3.0
3.71k stars 378 forks source link

indexSelectLargeIndex: block: [523,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed. #74

Closed ciobanudann closed 1 month ago

ciobanudann commented 1 month ago

hey guys, for some reason when I'm loading a PDF with a lot of data, it starts to fail on the build_index method,

please let me know what information should I provide so you can reproduce it on your end

../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [14,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [15,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [16,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [17,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [18,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [26,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [27,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [28,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize failed. ⠇ Building index...../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize failed. ../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [523,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize failed. ⠋ Building index... INFO: 127.0.0.1:42150 - "POST /api/v1/sparrow-llm/ingest HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi result = await app( # type: ignore[func-returns-value] File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call return await self.app(scope, receive, send) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call await super().call(scope, receive, send) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/applications.py", line 123, in call await self.middleware_stack(scope, receive, send) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in call raise exc File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in call await self.app(scope, receive, send) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/routing.py", line 758, in call await self.middleware_stack(scope, receive, send) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/routing.py", line 778, in app await route.handle(scope, receive, send) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/routing.py", line 299, in handle await self.app(scope, receive, send) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/routing.py", line 79, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/starlette/routing.py", line 74, in app response = await func(request) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(values) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/api.py", line 116, in ingest answer = await run_from_api_ingest(agent, index_name, file, False) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/ingest.py", line 34, in run_from_api_ingest ingest.run_ingest(user_selected_agent, temp_file_path, index_name) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/embeddings/agents/llamaindex.py", line 40, in run_ingest index = self.invoke_pipeline_step(lambda: self.build_index(client, embeddings, documents, index_name, File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/embeddings/agents/llamaindex.py", line 84, in invoke_pipeline_step ret = task_call() File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/embeddings/agents/llamaindex.py", line 40, in index = self.invoke_pipeline_step(lambda: self.build_index(client, embeddings, documents, index_name, File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/embeddings/agents/llamaindex.py", line 70, in build_index index = VectorStoreIndex.from_documents( File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/indices/base.py", line 145, in from_documents return cls( File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/indices/vector_store/base.py", line 75, in init super().init( File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/indices/base.py", line 94, in init index_struct = self.build_index_from_nodes( File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/indices/vector_store/base.py", line 308, in build_index_from_nodes return self._build_index_from_nodes(nodes, insert_kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/indices/vector_store/base.py", line 280, in _build_index_from_nodes self._add_nodes_to_index( File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/indices/vector_store/base.py", line 233, in _add_nodes_to_index nodes_batch = self._get_node_with_embedding(nodes_batch, show_progress) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/indices/vector_store/base.py", line 141, in _get_node_with_embedding id_to_embed_map = embed_nodes( File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/indices/utils.py", line 138, in embed_nodes new_embeddings = embed_model.get_text_embedding_batch( File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py", line 102, in wrapper self.span_drop(*args, id=id, err=e, kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py", line 77, in span_drop h.span_drop(*args, id=id, err=err, *kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/instrumentation/span_handlers/base.py", line 47, in span_drop self.prepare_to_drop_span(args, id=id, err=err, kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/instrumentation/span_handlers/null.py", line 35, in prepare_to_drop_span raise err File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py", line 100, in wrapper result = func(args, kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/core/base/embeddings/base.py", line 280, in get_text_embedding_batch embeddings = self._get_text_embeddings(cur_batch) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/embeddings/huggingface/base.py", line 204, in _get_text_embeddings return self._embed(texts) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/llama_index/embeddings/huggingface/base.py", line 161, in _embed model_output = self._model(encoded_input) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, *kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/transformers/models/mpnet/modeling_mpnet.py", line 543, in forward embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, inputs_embeds=inputs_embeds) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/transformers/models/mpnet/modeling_mpnet.py", line 102, in forward position_embeddings = self.position_embeddings(position_ids) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(args, kwargs) File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 190, in forward return F.embedding( File "/home/ciobanudann/ai/sparrow/sparrow-ml/llm/.env_llamaindex/lib/python3.10/site-packages/torch/nn/functional.py", line 2551, in embedding return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) RuntimeError: CUDA error: device-side assert triggered Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

abaranovskis-redsamurai commented 1 month ago

Hi, I haven't encountered any similar issue, I'm mainly focused on Sparrow Parse agent currently.