weaviate / semantic-search-through-wikipedia-with-weaviate

Semantic search through a vectorized Wikipedia (SentenceBERT) with the Weaviate vector search engine
MIT License
241 stars 21 forks source link

Import slow down #5

Closed murak038 closed 2 years ago

murak038 commented 2 years ago

Hi,

I was trying to run the demo on an AWS instance and was running into errors/slow down after I imported 12,000,000 paragraphs.

Instance Details: g4dn.12xlarge 4 T4 GPUs 48 vCPU 192 GB RAM Ubuntu 18.04 OS

After I import 12 million paragraphs, the machine performance considerably slows down. I start getting a batch creation timeout message for a batch size of 512, and I continue to get the message when I reduce the batch size to 50. Any ideas why the process slows down so much? The "vectorCacheMaxObjects": 150000000000 so I know that is not the issue.

bobvanluijt commented 2 years ago

Hi @murak038 – apologies for the slow response.

Would you mind bringing this question to our Slack channel? More people might benefit from the solution there.