Closed nehajatav closed 2 years ago
You have reached a bottleneck, attempting to push more load increases latency due to queueing. Most likely related to IO.
What type of IO are these instances configured with? Indexing is also write-heavy and all document operations are written to the transaction log and each batch synched for durability. Synch is known to have a high cost on network attached storage.. This can be toggled by sync-transactionlog
@jobergum this particular one was on Ceph implementation of PVC
Yes, Not familiar with Ceph, but smells like slow high latency remote storage. See my comment above on turning off synching for remote storage.
We tried turning off sync-transactionlog on Ceph, we didn't get any improvement in feeding On NAS, we saw a minor improvement
Describe the bug We are running feed client and tuning the number of instances of feeder client. When increasing the instances of feeder, we notice that the host running the feeder has no resource contention (~10% cpu and memory used). We have configured container allocatedMemory to 90% of 64G - this is nearly constant as is cpu ~4/8 upon increasing feeders. The feeding concurrency of content is 0.8 (8 cpu available) - resource utilization on this is also nearly constant 3/8 cores used and 46/64 (feed block at ~60). When we keep only 1 feeder instance, we get for every 2k ingestions, ~500ms of avg latency. However, when we bump it up to 3, the average latency goes to ~2000ms, with resource utilization constant (container memory and cpu bumps up but still well within limits for a small duration but it again settles down). The overall throughput remains constant while the resources remain unutilized. Is this expected?
To Reproduce Steps to reproduce the behavior:
Expected behavior The latency should NOT have gone up and the overall throughput should have increased
Environment (please complete the following information): OS: Docker image vespa:7.559.12 Infrastructure: Kubernetes Versions Major:"1", Minor:"21", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"
Vespa version 7.559.12 compiled with go1.16.13 on linux/amd64