search

datalad / datalad-fuse

DataLad extension to provide FUSE file system access
Other
1 stars 4 forks source link

"address" warnings about possible memory leaks due to lru_cache #73

Closed yarikoptic closed 2 years ago

yarikoptic commented 2 years ago

I have mentioned while looking at https://github.com/datalad/datalad-fuse/runs/6672682413?check_suite_focus=true of #72 that linting gives us

lint run-test: commands[0] | flake8 --config=tox.ini datalad_fuse
datalad_fuse/fsspec.py:53:6: B019 Use of `functools.lru_cache` or `functools.cache` on methods can lead to memory leaks. The cache may retain instance references, preventing garbage collection.
datalad_fuse/fsspec.py:206:6: B019 Use of `functools.lru_cache` or `functools.cache` on methods can lead to memory leaks. The cache may retain instance references, preventing garbage collection.
datalad_fuse/fuse_.py:110:6: B019 Use of `functools.lru_cache` or `functools.cache` on methods can lead to memory leaks. The cache may retain instance references, preventing garbage collection.

Needs analysis on either it is something to just ignore or we indeed would retain too many instances.

jwodder commented 2 years ago

@yarikoptic Normally, datalad fusefs is meant to be run as a command, in which case all the instances of classes that use @lru_cache are deleted when the program exits anyway. I can only see this being a problem if someone were to invoke datalad.api.fusefs() multiple times in a single process.

Either way, the best option for replacing @lru_cache seems to be the methodtools package.

yarikoptic commented 2 years ago

I can only see this being a problem if someone were to invoke datalad.api.fusefs() multiple times in a single process.

and even then I am not 100% certain it wouldn't be a "feature" -- don't we want to retain those instances as long as their methods are in the cache? I think we want, so let's just keep them and ignore those warnings for now.

Or am I missing some aspect?

jwodder commented 2 years ago

@yarikoptic If datalad.api.fusefs() is called multiple times in the same process, even if it's passed the same arguments, new instances of the datalad-fusefs classes will be constructed, and they will not have access to cached results from previous instances.

yarikoptic commented 2 years ago

ah -- gotcha. Let's then switch to methodtools