search

xiph / awcy

http://arewecompressedyet.com/
MIT License
72 stars 46 forks source link

IOBottleneck on full CTC sets in AWCY Machines #319

Open vibhoothi opened 2 years ago

vibhoothi commented 2 years ago

Hi,

https://beta.arewecompressedyet.com/?job=avm_crosscheck_base_ra_mandatory_17f&job=avm_crosscheck_target_ra_mandatory_17f%402022-10-14T22%3A59%3A34.034Z

2022-10-14 19:12:53.603 Exception while running WalkingInStreet_1920x1080_30fps.y4m with quality 135 of A2 for run avm_crosscheck_target_ra_mandatory_17f@2022-10-14T22:59:34.034Z [Errno 24] Too many open files

vibhoothi commented 2 years ago

@tdaede Hi, This seems to be problematic, we need to think of some ways to overcome this, Maybe increase the ULIMIT further from 8GB? I am not sure, or is it because we accessing same file multiple times on same machine?

tdaede commented 2 years ago

This is actually an issue with the # of file descriptors given to rd_tool. Changing ulimit isn't enough. We need to change the maximum number of file descriptors in /etc/security/limits.conf.

We might also possibly have a fd leak.