Library cache can exhaust file descriptors with LMDB - Githubissues

man-group / ArcticDB

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.

http://arcticdb.io

Other

1.52k stars 93 forks source link

Library cache can exhaust file descriptors with LMDB #1913

Closed poodlewars closed 1 month ago

poodlewars commented 1 month ago

Describe the bug

No response

Steps/Code to Reproduce

Create an Arctic instance with LMDB and hold it in memory
Create many libraries and call get_library on them. Let the library object fall out of scope.
On Windows, eventually get,

OSError: [Errno 24] Too many open files

A similar thing would probably happen on Linux.

More details:

https://chat-man.slack.com/archives/C02NEG3V38X/p1728482885941529

We should probably just remove the library cache?

Expected Results

All libraries accessed, no errors

OS, Python Version and ArcticDB Version

-

Backend storage used

LMDB

Additional Context

No response

poodlewars commented 1 month ago

Dupe of #1904