search

UKPLab / sentence-transformers

State-of-the-Art Text Embeddings
https://www.sbert.net
Apache License 2.0
14.85k stars 2.44k forks source link

Cannot close SpawnProcess for multiprocessing model.stop_multi_process() #1660

Open Muennighoff opened 2 years ago

Muennighoff commented 2 years ago

Upon running model.stop_multi_process()

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-17-8a9d42a11ec1> in <module>
      4 for p in pool['processes']:
      5     p.join()
----> 6     p.close()
      7 
      8 pool['input'].close()

AttributeError: 'SpawnProcess' object has no attribute 'close'
nreimers commented 2 years ago

Maybe it just exist when you use fork as method to start new processes?

Muennighoff commented 2 years ago

Yeah I think so. I fixed it by just removing the close() line and all works fine. I'm not sure why exactly close doesn't exist for spawn, as the docs don't mention it.

ajdaling commented 2 years ago

I just ran into the same issue. Not sure what the issue is but I removed p.close() on line 237 in SentenceTransformer.py, as suggested, and that worked for me as well. The child processes still appear close as normal and GPU memory is freed.