search

rom1504 / clip-retrieval

Easily compute clip embeddings and build a clip retrieval system with them
https://rom1504.github.io/clip-retrieval/
MIT License
2.41k stars 209 forks source link

distributed clip inference #114

Closed rom1504 closed 2 years ago

rom1504 commented 2 years ago

follow up on https://github.com/rom1504/clip-retrieval/pull/107

missing:

rom1504 commented 2 years ago

https://spark.apache.org/docs/3.2.0/spark-standalone.html#resource-allocation-and-configuration-overview https://www.nvidia.com/en-us/ai-data-science/spark-ebook/getting-started-spark-3/

trying to use resource concept of spark 3 to handle the gpu resource

rom1504 commented 2 years ago

https://stackoverflow.com/questions/69551655/how-spark-resource-gpu-manager-is-linked-to-physical-resource-availability

rom1504 commented 2 years ago

https://github.com/NVIDIA/spark-rapids/issues/1674#issuecomment-774050921

rom1504 commented 2 years ago

https://developer.nvidia.com/blog/accelerating-deep-learing-with-apache-spark-and-gpus-on-aws/

val context = TaskContext.get()
val gpu = context.resources()("gpu").addresses(0)
rom1504 commented 2 years ago

done