After I import 12 million paragraphs, the machine performance considerably slows down. I start getting a batch creation timeout message for a batch size of 512, and I continue to get the message when I reduce the batch size to 50. Any ideas why the process slows down so much? The "vectorCacheMaxObjects": 150000000000 so I know that is not the issue.
Hi,
I was trying to run the demo on an AWS instance and was running into errors/slow down after I imported 12,000,000 paragraphs.
Instance Details: g4dn.12xlarge 4 T4 GPUs 48 vCPU 192 GB RAM Ubuntu 18.04 OS
After I import 12 million paragraphs, the machine performance considerably slows down. I start getting a batch creation timeout message for a batch size of 512, and I continue to get the message when I reduce the batch size to 50. Any ideas why the process slows down so much? The "vectorCacheMaxObjects": 150000000000 so I know that is not the issue.