OSError: [Errno 24] Too many open files

[ohyeah521]

[ERROR] 2023-09-01 09:54:34 - libsast scan multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "D:\venv\lib\site-packages\libsast\core_matcher\pattern_matcher.py", line 90, in pattern_matcher data = file_path.read_text('utf-8', 'ignore') File "C:\Python3\lib\pathlib.py", line 1266, in read_text with self.open(mode='r', encoding=encoding, errors=errors) as f: File "C:\Python3\lib\pathlib.py", line 1252, in open return io.open(self, mode, buffering, encoding, errors, newline, File "C:\Python3\lib\pathlib.py", line 1120, in _opener return self._accessor.open(self, flags, mode) OSError: [Errno 24] Too many open files: 'M:\uploads\b1436c59b24ebc3ab8632f24aaa16cb4\java_source\com\taobao\login4android\membercenter\security\SecurityEntranceActivity.java'

res = scanner.scan()

File "D:\venv\lib\site-packages\libsast\scanner.py", line 58, in scan results['pattern_matcher'] = PatternMatcher( File "D:\venv\lib\site-packages\libsast\core_matcher\pattern_matcher.py", line 58, in scan results = pool.map( File "C:\Python3\lib\multiprocessing\pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "C:\Python3\lib\multiprocessing\pool.py", line 771, in get raise self._value libsast.exceptions.RuleProcessingError: Rule processing error.

[ajinabraham]

This looks like a resource limiter exception from OSError: [Errno 24] Too many open files: Might be Windows specific. I will cut a new release were you can reduce the number of multiprocessing pool via environment variable.

[ajinabraham]

Can you try out libsast 2.0.2 ?

In your Windows/Thread ripper setup, the workers are defaulted to minimum of 61. Set an environment variable LIBSAST_WORKERS=10 and perform a scan. If it works, you can increase the worker count until you see the OSError. The higher the worker count, the better the performance/speed of the scan. This environment variable allows you to use a worker count that gives you best performance without exceptions on windows.

Other wise, use something like WSL for Windows and set a higher ulimit for libsast scans.

[ohyeah521]

I tested it, but it still doesn't work properly

[ajinabraham]

Have you tired with different values for LIBSAST_WORKERS ?

[ohyeah521]

Sorry, I wrote the wrong environment variable when testing libsast 2.0.2 yesterday. After I corrected it, it was able to work normally. libsast 2.0.3 work properly.