Project-HAMi / HAMi

Heterogeneous AI Computing Virtualization Middleware
http://project-hami.io/
Apache License 2.0
956 stars 197 forks source link

gunicorn多worker情况下,使用vgpu出现死锁 #588

Open xiaoyu1095 opened 2 weeks ago

xiaoyu1095 commented 2 weeks ago

稳定复现: 使用gunicorn启动SentenceTransformer拉起embeeding模型,(会根据worker数量,同时启动多个相同服务进程) 会卡住在SentenceTransformer(path)实例化处,后pod因为探针原因会被kill掉,重启的实例会输出unified_lock locked, waiting 1 second...

302277e0b48a3c85505e1fe323e629ea

wawa0210 commented 2 weeks ago

Can you describe your usage scenario in detail? Provide the corresponding version, node resource status, application yaml and other information