Errno 24 - Too many open files - loading 200+ image folder

[CostaConcord]

On M2 MacOS 13.1, I suspect it may be a mac specific error. When loading a folder that has more than ~200 photos, it crashes. I haven't pinned down the exact number, I think its more than 230 and less than 235. (I got tired of relaunching and moving files around, crashing, relaunching, etc.)

Traceback (most recent call last): File "/lib/python3.9/site-packages/gradio/routes.py", line 393, in run_predict File "/lib/python3.9/site-packages/gradio/blocks.py", line 1069, in process_api File "/lib/python3.9/site-packages/gradio/blocks.py", line 878, in call_function File "/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync File "/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread File "/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run File "/dataset-tag-editor-standalone/scripts/ui/block_load_dataset.py", line 135, in load_files_from_dir File "/dataset-tag-editor-standalone/scripts/dataset_tag_editor/dte_logic.py", line 750, in load_dataset File "/dataset-tag-editor-standalone/scripts/dataset_tag_editor/dte_logic.py", line 694, in load_images File "/lib/python3.9/pathlib.py", line 1205, in absolute OSError: [Errno 24] Too many open files

Any help is much appreciated. Thanks.

[M1kep]

macOS has a default ulimit of 256. This can be seen via:

❯ ulimit -n
256

You can update this by running ulimit -Sn:

# Check current limit
❯ ulimit -n
256
# Set to 1000
❯ ulimit -Sn 1000
# Confirm it's been set
❯ ulimit -n
1000

Generally you'll want to try and keep this at a reasonable level, but values in the thousands wont cause you any issues. Do note, I don't believe changing the value via ulimit will persist across reboots.

I'm not sure it's an issue with Pillow or the way this tool is opening files, but this should work as a workaround.