Open Matthieu-Tinycoaching opened 1 year ago
Hi @Matthieu-Tinycoaching – I think this is because the memory on the GPU is full. You can lower the # of models loaded onto the GPU to make this work.
Hi @bobvanluijt thanks for your feedback!
Even if I reduce the number of models loaded onto the GPU to 2, I got both errors regularly when importing:
2022-10-17 14:03:59.222 | DEBUG | __main__:handle_results:169 - fail with status 500: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-10-17 14:03:59.222 | DEBUG | __main__:handle_results:169 - send POST request: Post "http://loadbalancer:8080/vectors": read tcp 192.168.16.5:39354->192.168.16.4:8080: read: connection reset by peer
Lastly, I even removed nginx
load-balancer and still got regularly those issues with the following docker-compose.yml file:
---
version: '3.4'
services:
weaviate:
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: semitechnologies/weaviate:1.13.2
ports:
- 8080:8080
restart: on-failure:0
volumes:
- /media/matthieu/HDD_4T01/weaviate:/var/lib/weaviate
depends_on:
- t2v-transformers-01-001
environment:
TRANSFORMERS_INFERENCE_API: 'http://t2v-transformers-01-001:8080'
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'text2vec-transformers'
ENABLE_MODULES: 'text2vec-transformers'
CLUSTER_HOSTNAME: 'node1'
t2v-transformers-01-001:
image: semitechnologies/transformers-inference:sentence-transformers-multi-qa-MiniLM-L6-cos-v1
restart: always
environment:
ENABLE_CUDA: '1'
NVIDIA_VISIBLE_DEVICES: '0' # <== set the GPU to use. 0 = 1st GPU, 1 = 2nd, etc
deploy:
resources:
reservations:
devices:
- capabilities:
- 'gpu'
...
Hmm - interesting. @antas-marcin does this issue ring a bell?
Hi,
I try to reproduce step 2 of the semantic search through wikipedia on my local computer with RTX 3090, and while importing data with the
nohup python3 -u import.py &
command I got the following error message:I follow the guidelines provided at https://weaviate.io/developers/weaviate/current/tutorials/semantic-search-through-wikipedia.html#step-2-import-the-dataset-and-vectorize-the-content and set the following files to work on local computer:
1 /
docker-compose.yml
:2 /
nginx.conf
:My GPU card is well configured with
CUDA 11.6
andcuDNN
and the following commandnvidia-smi
returns: