search

ronny-rentner / UltraDict

Sychronized, streaming Python dictionary that uses shared memory as a backend
Apache License 2.0
272 stars 25 forks source link

Unable to access Ultradict after a certain loop Limit, Issue occurs Only on Linux............. #9

Closed hemakumar01 closed 2 years ago

hemakumar01 commented 2 years ago

from UltraDict import UltraDict

ultra = UltraDict({ 'init': 'some initial data' }, name='myname1')

for i in range(1,5000): print(UltraDict(name='myname1'))

############### ERROR ################# File "/home/merit/miniconda3/lib/python3.9/site-packages/UltraDict/UltraDict.py", line 659, in unlink self.control.unlink() File "/home/merit/miniconda3/lib/python3.9/multiprocessing/shared_memory.py", line 241, in unlink _posixshmem.shm_unlink(self._name) FileNotFoundError: [Errno 2] No such file or directory: '/myname1'

ronny-rentner commented 2 years ago

Hi @hemakumar01

Thanks for your issue report. Can you please try the dev branch and see if this fixes your issue?

ronny-rentner commented 2 years ago

PS: To add some explanation: With your code, you're creating and connecting 5000 new instances of an UltraDict to shared memory. There is a bug in the main branch that prevents proper cleanup of all file handles and at some point some of the OS resources are gone and you will start seeing random exceptions. Though, of course using UltraDict like this is not really an efficient way. You should keep around your instances (unless you also spawn 5000 new processes in the same time).

hemakumar01 commented 2 years ago

I tried the dev branch and the issue is now fixed, Thanks for the reply, out of curiosity, i didnt look into the dev code much, can you explain what was done to fix the issue.

ronny-rentner commented 2 years ago

Sure, I've moved the cleanup code from an atexit handler to a finalizer and also implemented the __del__() method to correctly close the file handles again. The latter should help in your loop to close the file handles after each cycle.