Raising fdlimit to maximum on startup

[vdudouyt]

One of the common problems I hit while running my own server software in high-load production is a shortage of fdlimit resources. A common approach to mitigate this problem is to automatically set the fdlimit before tokio::main() to a maximum value possible accordingly to sysctl settings: https://github.com/vdudouyt/std-fpm/blob/master/src/fdlimit.rs

Alternatively you can use the fdlimit crate, although I don't like it's thiserror dependency causing some proc-macro2 stuff to appear in cargo tree.

[evgenyigumnov]

I agree with ChatGPT:

Limits on the number of file descriptors (fdlimit) in operating systems like Linux are often set to relatively low values by default for several reasons:

1. Security and Stability: Restricting the number of file descriptors helps prevent a single application or process from exhausting all available system resources, which could negatively impact the performance or stability of the entire system.
2. Legacy and Compatibility: Older Unix-like systems had more limited resources, and some applications might not have functioned correctly with higher limit values. Consequently, conservative values remain inherited as default settings.
3. Manageability: System administrators prefer to control the resources allocated to processes to ensure predictable system behavior.

Is It a Good Idea to Change Limits from a Web Server?

Modifying the fdlimit from within a web server at runtime is a controversial practice. Here are the pros and cons:

Pros:

• Automation: Setting limits in code ensures that the application will have access to the necessary resources, which is crucial for high-load scenarios.
• Convenience: Developers don’t need to rely on administrators to manually configure limits on each system.

Cons:

• Lack of Transparency: If the web server modifies system settings "silently," it can complicate debugging and maintenance. Administrators may be unaware of the changes, making it harder to diagnose resource-related issues.
• Security Risks: Raising file descriptor limits can make the system vulnerable to resource exhaustion attacks, where an attacker attempts to use up all available descriptors.
• Loss of Control: System administrators might prefer to manage these limits manually based on the system's resource requirements and policies.

Recommendations:

1. Coordination Is Better: Changing limits should ideally be part of system configuration and agreed upon with the administrator. This can be done via configuration files like /etc/security/limits.conf or through server settings (e.g., systemd).
2. Logging and Documentation: If limits are changed programmatically, it should be well-documented and logged, so administrators can track the changes.
3. Configurability: A good practice is to make this behavior configurable, allowing the administrator to decide whether to increase the limits automatically.

Overall, adjusting limits is a powerful tool but should be used carefully and thoughtfully.