Master test open too many files (hashtag2894)

[vikramsubramanian]

My local machine has ulimit 256 and when I run make test it starts complaining too many opened files. I assume this partially because we open 64 files in order to parallel hash index. But still 256 should be sufficient to run tests.

P.S. I'm labelling this as a bug for now. Feel free to change if this is expected. )

[mayil-ai[bot]]

Summary: Test process opens too many files, causing "too many opened files" error.

Possible Solution

• Increase the ulimit for open files in the system or for the user running the make test command. This can be done temporarily for the session with ulimit -n <new_limit> or permanently by editing /etc/security/limits.conf or a similar system configuration file.
• Modify the test suite setup to ensure that file descriptors are properly managed. Implement or ensure that teardown_test_environment() is closing all open files after each test case or test suite.
• Review and possibly refactor the parallel_hash_index() function to limit the number of simultaneously open files, or to ensure that files are closed as soon as they are no longer needed.
• If the tests are running in parallel, consider reducing the level of parallelism to decrease the number of concurrently open files.
• Check the Makefile and any associated test scripts to see if there is a way to configure the number of threads or processes used during testing, which may indirectly control the number of open files.
• If the issue persists and is related to the inherent design of the tests or the database, consider implementing a file descriptor pool or manager that can throttle the number of open files at any given time to stay within system limits.

Code snippets to check

• test/main/config_test.cpp

  The code snippet includes a test that sets the maximum number of threads for execution, which could be related to the issue of too many files being opened during tests.