search

ornladios / ADIOS2

Next generation of ADIOS developed in the Exascale Computing Program
https://adios2.readthedocs.io/en/latest/index.html
Apache License 2.0
268 stars 125 forks source link

Ulimit: Programatically increase softlimit to hard-limit #3372

Open ax3l opened 1 year ago

ax3l commented 1 year ago

In post-processing, we often have less resources as in writing. We often reach the ulimit -n for open files on Jupyter or ParaView or VisIt or scripts for sub-files.

This can often be work-arounded by increasing the ulimit -n ... soft-limit closer to the hard-limit of a machine.

Idea discussed today with @pnorbert @franzpoeschel: Let's programmatically increase the soft-limit of a ADIOS2-enabled process to like 90% or 100% of the hard-limit for open files.

X-ref:

pnorbert commented 1 year ago

This has been added to master branch with #3376. BP3, BP4 and BP5 now will raise the softlimit to the hardlimit automatically. Can you please test this out on your datasets. What about Windows? I made a blind call for windows to raise the limit but I don't know if it works.