Closed kdevan closed 3 weeks ago
After about an hour it will continue.
Are you saying that the write proceeds after some time automatically? Can you tell me if your collection has a large number of nested unique fields?
Are you saying that the write proceeds after some time automatically?
Yeah I was surprised to see that.
Can you tell me if your collection has a large number of nested unique fields?
Flat collection of 27 fields, mostly strings with a few int.
It starts out with a little over 21,000 queued writes. It's taking about four hours to process these (that would be assuming there's no point that it halts. The halt would add an extra hour on top of that).
EDIT: The pipeline was actually running just now and it actually halts at exactly the same number of queued writes. It's right now halted at 8375.
This may have been an exception for some document being imported at that point. After today I'll know and if so I'll go ahead and close this issue.
Second update: I do believe this happens because of exceptional documents that are being imported at the time! For anyone else who sees something like this, just know that even if the log seems repetitive and that writes are halted or frozen, Typesense is doing its thing!
I'm curious to hear what was problematic/exceptional about these particular documents. Were they super large?
Yeah I'm curious about that too. It could effect some other parts of the system as well! I'm going to do some digging and see if I can figure out which ones were part of that batch. I'll update here if I can figure anything out.
Description
I'm running into an issue where at a certain point of a bulk import that happens daily, the queued writes seem to just stop. This is just a small sample of the logs, it goes way back just sitting at 8375. The number it stops at is different over the last few days. This causes the health check to return false and no other operations are able to run. After about an hour it will continue.
Resource wise there are plenty of resources:
40GB free memory 0 swap used 30GB free HD space 16 cores
Stats:
Any idea what this might be? Any help or direction for troubleshooting this is much appreciated.
Steps to reproduce
This happens after using
/documents/import?action=upsert
to upsert many jsonl documents.Expected Behavior
Expect the queued writes to continue.
Actual Behavior
Queued writes stop writing.
Metadata
Typesense Version: 27.1
OS: Debian bookworm