right now we have limited the execution of the ingestion function to 1. LanceDB has a native lock system based on DynamoDB locks currently in beta. We should experiment with this and remove the artificial limitation of 1 max concurrent execution for the writer.
Once this is implemented, we should run load tests to understand impact on retrieval performance against the same (user+knowledge-base).
right now we have limited the execution of the ingestion function to 1. LanceDB has a native lock system based on DynamoDB locks currently in beta. We should experiment with this and remove the artificial limitation of 1 max concurrent execution for the writer.
Once this is implemented, we should run load tests to understand impact on retrieval performance against the same (user+knowledge-base).