search

rhsimplex / image-match

🎇 Quickly search over billions of images
2.94k stars 405 forks source link

Fast Insert #151

Open Kencho5 opened 2 years ago

Kencho5 commented 2 years ago

as you mentioned in the readme, it states that it can easily scale up to billions of images, but i'm having a hard time inserting a lot of images in elasticsearch. i've tried to do it with multiprocessing, threading and even async too, but given that i have around 160 million images it's still slow. any suggestions on how i can insert those images a bit faster?

trashmit commented 2 years ago

did you come up with a solution?

Kencho5 commented 2 years ago

yes. made bulk insert script, but don't have it anymore. you have to split functions and use multiprocessing