Too many open files during preprocessing

[AI-Guru]

Hi!

I just found your repository. I was intrigued from the beggining and after a close look I am now training on my AI station 🤗 Great work!

I tried to preprocess a dataset with a lot of audio files. The audio to image preprocessor dies with an exception, telling me that there are to many open files. Does the preprocessor close all opened files?

Looking forward to working more with your repo!

Best, Tristan

[teticio]

It should be closing files properly, but it is using other packages for I/O and I have run into this issue in the past in a different context (with pydub). Presumably you are on MacOS / Unix. Can you run

ulimit –Sn

and

ulimit -Hn

It may be that you can up the limit (as a workaround). In Python, you can do this with

import resource
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (hard, hard))

which will set the soft limit to the hard limit for this session.

[AI-Guru]

Thanks a lot! Worked like a charm!