Closed pauldintel closed 9 months ago
I'm facing the same issue here. My system environment is CentOS Linux release 7.4.1708 (Core), and I'm using Python 3.8.1.
Have you found any solution or workaround for this problem?
I'm facing the same issue here. My system environment is CentOS Linux release 7.4.1708 (Core), and I'm using Python 3.8.1.
Have you found any solution or workaround for this problem?
I am using CentOS 8 stream and Python 3.10 ... no solution yet .waiting for the repo owners to comment .
I'm facing the same issue here. My system environment is CentOS Linux release 7.4.1708 (Core), and I'm using Python 3.8.1. Have you found any solution or workaround for this problem?
I am using CentOS 8 stream and Python 3.10 ... no solution yet .waiting for the repo owners to comment .
Hello, I referred to the testing steps in this repository (https://github.com/myscale/vector-db-benchmark/blob/master/docs/step-by-step-guide-for-benchmark.md#qdrant) and made some minor modifications to the QdrantUploader.init_client method in qdrant/engine/clients/qdrant/upload.py. I understand that the program is now able to run, but it appears to be hanging during runtime and there are anomalies detected in the collection status. ` curl 'http://{ip}:{port}/collections/benchmark'
{"result":{"status":"red","optimizer_status":{"error":"Service internal error: IO Error: Function not implemented (os error 38)"},"vectors_count":12ctors_count":0,"points_count":1240620,"segments_count":24,"config":{"params":{"vectors":{"size":100,"distance":"Cosine"},"shard_number":3,"replicatite_consistency_factor":1,"on_disk_payload":true},"hnsw_config":{"m":16,"ef_construct":100,"full_scan_threshold":10000,"max_indexing_threads":0,"on_dizer_config":{"deleted_threshold":0.2,"vacuum_min_vector_number":1000,"default_segment_number":0,"max_segment_size":null,"memmap_threshold":10000,"i":20000,"flush_interval_sec":5,"max_optimization_threads":1},"wal_config":{"wal_capacity_mb":32,"wal_segments_ahead":0},"quantization_config":null},{}},"status":"ok","time":0.001639665} `
You forgot to add -p 6334:6334
This was fixed in https://github.com/qdrant/vector-db-benchmark/commit/9b22a842b9a3a42ada68c28938447cebed61f696
Feel free to let us know if you continue to face the error.
@KShivendu still facing this when running the qdrant tests, is a specific version of Python needed?
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/opt/vector-db-benchmark/run.py", line 91, in <module>
app()
File "/opt/vector-db-benchmark/run.py", line 86, in run
raise e
File "/opt/vector-db-benchmark/run.py", line 59, in run
client.run_experiment(
File "/opt/vector-db-benchmark/engine/base_client/client.py", line 115, in run_experiment
upload_stats = self.uploader.upload(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload
latencies = list(
^^^^^
File "/home/ubuntu/.pyenv/versions/3.11.10/lib/python3.11/multiprocessing/pool.py", line 873, in next
raise value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x710345e4a950>'. Reason: 'TypeError("cannot pickle '_thread.RLock' object")
@KShivendu still facing this when running the qdrant tests, is a specific version of Python needed?
Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/opt/vector-db-benchmark/run.py", line 91, in <module> app() File "/opt/vector-db-benchmark/run.py", line 86, in run raise e File "/opt/vector-db-benchmark/run.py", line 59, in run client.run_experiment( File "/opt/vector-db-benchmark/engine/base_client/client.py", line 115, in run_experiment upload_stats = self.uploader.upload( ^^^^^^^^^^^^^^^^^^^^^ File "/opt/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload latencies = list( ^^^^^ File "/home/ubuntu/.pyenv/versions/3.11.10/lib/python3.11/multiprocessing/pool.py", line 873, in next raise value multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x710345e4a950>'. Reason: 'TypeError("cannot pickle '_thread.RLock' object")
Same issue
In my case, the unpicklable is not in the passed-in parameters but seems the result, because I use wrong port.
I have launched the server using the docker image and vector DB launched fine : Same machine in other session I am using the vector benchmark run . I am using python and pip3.10 .
[root@9049fa05600b ~]# docker run -p 6333:6333 qdrant/qdrant _ _| | | | /
|/ _
| '/ ` | ' | | | (| | (| | | | (| | | | | | _, |_,|| _,|| ||_| ||Access web UI at http://localhost:6333/dashboard
[2023-07-18T22:34:56.204Z INFO storage::content_manager::consensus::persistent] Initializing new raft state at ./storage/raft_state [2023-07-18T22:34:56.236Z INFO qdrant] Distributed mode disabled [2023-07-18T22:34:56.236Z INFO qdrant] Telemetry reporting enabled, id: a96d4b56-dba5-4f4c-9332-04ab9c9033eb [2023-07-18T22:34:56.238Z INFO qdrant::tonic] Qdrant gRPC listening on 6334 [2023-07-18T22:34:56.238Z INFO qdrant::tonic] TLS disabled for gRPC API [2023-07-18T22:34:56.251Z INFO qdrant::actix] TLS disabled for REST API [2023-07-18T22:34:56.251Z INFO qdrant::actix] Qdrant HTTP listening on 6333 [2023-07-18T22:34:56.251Z INFO actix_server::builder] Starting 7 workers [2023-07-18T22:34:56.251Z INFO actix_server::server] Actix runtime found; starting in Actix runtime [2023-07-18T22:35:04.526Z INFO actix_web::middleware::logger] 172.17.0.1 "DELETE /collections/benchmark HTTP/1.1" 200 72 "-" "python-httpx/0.24.1" 0.000457 [2023-07-18T22:35:04.530Z INFO actix_web::middleware::logger] 172.17.0.1 "DELETE /collections/benchmark HTTP/1.1" 200 69 "-" "python-httpx/0.24.1" 0.000119 [2023-07-18T22:35:05.562Z INFO actix_web::middleware::logger] 172.17.0.1 "PUT /collections/benchmark HTTP/1.1" 200 71 "-" "python-httpx/0.24.1" 1.031083
When I am running benchmark test getting below error :
[root@9049fa05600b vector-db-benchmark]# python3.10 -m run --engines -m-16- --datasets glove-* Running experiment: qdrant-mmap-m-16-ef-128 - glove-25-angular Downloading http://ann-benchmarks.com/glove-25-angular.hdf5... Moving: /tmp/tmp6q2343mx -> /root/vectordb/vector-db-benchmark/datasets/glove-25-angular/glove-25-angular.hdf5 Experiment stage: Configure Experiment stage: Upload 1343it [00:00, 18694.49it/s] Experiment qdrant-mmap-m-16-ef-128 - glove-25-angular interrupted Traceback (most recent call last): File "/root/vectordb/vector-db-benchmark/run.py", line 52, in run client.run_experiment(dataset, skip_upload, skip_search) File "/root/vectordb/vector-db-benchmark/engine/base_client/client.py", line 70, in run_experiment upload_stats = self.uploader.upload( File "/root/vectordb/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload latencies = list( File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 873, in next raise value multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f5fc488fe20>'. Reason: 'TypeError("cannot pickle '_thread.RLock' object")' Traceback (most recent call last):
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals)
File "/root/vectordb/vector-db-benchmark/run.py", line 79, in
app()
File "/root/vectordb/vector-db-benchmark/run.py", line 74, in run raise e
File "/root/vectordb/vector-db-benchmark/run.py", line 52, in run client.run_experiment(dataset, skip_upload, skip_search)
File "/root/vectordb/vector-db-benchmark/engine/base_client/client.py", line 70, in run_experiment upload_stats = self.uploader.upload(
File "/root/vectordb/vector-db-benchmark/engine/base_client/upload.py", line 56, in upload latencies = list(
File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 873, in next raise value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f5fc488fe20>'. Reason: 'TypeError("cannot pickle '_thread.RLock' object")'
Also there is no content generated inside results dir. Any pointer how to resolve this issue ? thanks