Currently, handling errors coming from ingesting documents within an archive can get pretty nasty.
Also, the way we do it right now cannot be parallelized for speedup and first collects all documents/chunks before doing the embedding. Missed opportunity for concurrency here and highly error-prone.
Currently, handling errors coming from ingesting documents within an archive can get pretty nasty. Also, the way we do it right now cannot be parallelized for speedup and first collects all documents/chunks before doing the embedding. Missed opportunity for concurrency here and highly error-prone.