Add resource requirements to the retrieve from faiss component

ml6team / fondant

Production-ready data processing made easy and shareable

https://fondant.ai/en/stable/

Apache License 2.0

339 stars 26 forks source link

Add resource requirements to the retrieve from faiss component #905

Closed mrchtr closed 6 months ago

mrchtr commented 6 months ago

When using the retrieve from faiss component with the LocalCluster it is a bit tricky to find the right machine configuration. Changed the initialisation of the LocalCluster. Assign each worker 8Gb of RAM and limit the number of cores accordingly.