Process files in parallel

prrao87 / db-hub-fastapi

Async bulk data ingestion and querying in various document, graph and vector databases via their Python clients

MIT License

33 stars 3 forks source link

Process files in parallel #15

Closed sanders41 closed 1 year ago

sanders41 commented 1 year ago

Yesterday I wasn't happy with the processing times and thought it could be better. I messed with it a bit more and got the bulk indexing time down from ~9.3s to ~3.7s by processing the files in parallel.

prrao87 commented 1 year ago

I'm running this on an M2 mac and it's running in about 2.4 sec! 😅