search

nginx / unit

NGINX Unit - universal web app server - a lightweight and versatile open source server that simplifies the application stack by natively executing application code across eight different programming language runtimes.
https://unit.nginx.org
Apache License 2.0
5.29k stars 325 forks source link

Frequent OOM-Killer in unit #940

Closed MrSipping closed 9 months ago

MrSipping commented 10 months ago

Hi, I continue to encounter almost weekly, oom-killers on two servers working in balancer. I think I have configured the processes in the best way and what is needed, could you give me a hand?

This is the error I see:

unit.service: A process of this unit has been killed by the OOM killer. Sep 04 17:16:41 unit systemd[1]: unit.service: Failed with result 'oom-kill'. Sep 04 17:16:41 unit systemd[1]: unit.service: Consumed 2h 31min 11.213s CPU time.

ac000 commented 10 months ago

The obvious question is, are your applications leaking memory?

tippexs commented 10 months ago

Please share the Unit Configuration and what language module in what version you are using.

MrSipping commented 10 months ago

Please share the Unit Configuration and what language module in what version you are using.

I had to remove the directories for privacy and because they were too long but my configuration is like this

` { "settings": { "http": { "max_body_size": 536870912, "discard_unsafe_fields": false } },

"access_log": "/var/log/unit.access.log",
"listeners": {
    "*:80": {
        "pass": "routes/main"
    },

    "*:443": {
        "pass": "routes/main",
        "tls": {
            "certificate": "bundle"
        }
    }
},

"routes": {
    "main": [

    ],

    "online": [

    ],

    "online-sm": [

    ],

    "online-v1": [

    ]
},

"applications": {
    "online-batch": {
        "type": "php",
        "processes": 8,
        "user": "www-data",
        "group": "www-data",
        "working_directory": " ",
        "targets": {

        },

        "limits": {
            "timeout": 3600,
            "requests": 1024
        },

        "environment": {
            "_MY_ID": "##",
            "_MY_ENVIRONMENT": "prod",
            "_MY_RESOURCE_ROOT": "#######",
            "_MY_ENGINE_ROOT": "#########",
            "_MY_AUTOLOADER": "########",
            "_MY_BOOTSRAP_VERSION": "###"
        },

        "options": {
            "admin": {
                "memory_limit": "1248M",
                "variables_order": "EGPCS",
                "expose_php": "0",
                "post_max_size": "256M",
                "upload_max_filesize": "256M",
                "max_execution_time": "3600",
                "max_input_time": "1800",
                "default_socket_timeout": "3600",
                "timezone": "Europe/Rome"
            },

            "user": {
                "display_errors": "0"
            }
        }
    },

    "online": {
        "type": "php",
        "processes": 512,
        "user": "www-data",
        "group": "www-data",
        "working_directory": "########",
        "targets": {

        },

        "limits": {
            "timeout": 360,
            "requests": 1024
        },

        "environment": {
            "_MY_ID": "##",
            "_MY_ENVIRONMENT": "prod",
            "_MY_RESOURCE_ROOT": "#########",
            "_MY_ENGINE_ROOT": "#########",
            "_MY_AUTOLOADER": "########",
            "_MY_BOOTSRAP_VERSION": "###"
        },

        "options": {
            "admin": {
                "default_socket_timeout": "480",
                "variables_order": "EGPCS",
                "zlib.output_compression_level": "-1",
                "upload_max_filesize": "24M",
                "max_input_time": "180",
                "expose_php": "0",
                "max_execution_time": "360",
                "zlib.output_compression": "0",
                "post_max_size": "128M",
                "memory_limit": "512M",
                "timezone": "Europe/Rome"
            },

            "user": {
                "display_errors": "0"
            }
        }
    },

    "online-sm": {
        "type": "php",
        "processes": 32,
        "user": "www-data",
        "group": "www-data",
        "working_directory": "########",
        "targets": {

        },

        "limits": {
            "timeout": 360,
            "requests": 1024
        },

        "environment": {
            "_MY_ID": "##",
            "_MY_ENVIRONMENT": "prod",
            "_MY_RESOURCE_ROOT": "########",
            "_MY_ENGINE_ROOT": "##########",
            "_MY_AUTOLOADER": "#############",
            "_MY_BOOTSRAP_VERSION": "########"
        },

        "options": {
            "admin": {
                "default_socket_timeout": "360",
                "variables_order": "EGPCS",
                "zlib.output_compression_level": "-1",
                "upload_max_filesize": "24M",
                "max_input_time": "240",
                "expose_php": "0",
                "max_execution_time": "480",
                "zlib.output_compression": "0",
                "post_max_size": "128M",
                "memory_limit": "512M",
                "timezone": "Europe/Rome"
            },

            "user": {
                "display_errors": "0"
            }
        }
    },

    "online-batch": {
        "type": "php",
        "processes": 2,
        "user": "www-data",
        "group": "www-data",
        "working_directory": "#######",
        "targets": {

        },

        "limits": {
            "timeout": 3600,
            "requests": 1024
        },

        "environment": {
            "_MY_ID": "##",
            "_MY_ENVIRONMENT": "prod",
            "_MY_RESOURCE_ROOT": "#######",
            "_MY_ENGINE_ROOT": "####################",
            "_MY_AUTOLOADER": "############",
            "_MY_BOOTSRAP_VERSION": "####"
        },

        "options": {
            "admin": {
                "memory_limit": "1024M",
                "variables_order": "EGPCS",
                "expose_php": "0",
                "post_max_size": "256M",
                "upload_max_filesize": "256M",
                "max_execution_time": "3600",
                "max_input_time": "1800",
                "default_socket_timeout": "3600"
            },

            "user": {
                "display_errors": "0"
            }
        }
    },

    "online-v1": {
        "type": "php",
        "processes": 128,
        "user": "www-data",
        "group": "www-data",
        "working_directory": "#########",
        "targets": {

        },

        "limits": {
            "timeout": 360,
            "requests": 1024
        },

        "environment": {
            "_MY_ID": "##",
            "_MY_ENVIRONMENT": "prod",
            "_MY_RESOURCE_ROOT": "#########",
            "_MY_ENGINE_ROOT": "###########",
            "_MY_AUTOLOADER": "###################",
            "_MY_BOOTSRAP_VERSION": "###"
        },

        "options": {
            "admin": {
                "memory_limit": "512M",
                "variables_order": "EGPCS",
                "expose_php": "0",
                "zlib.output_compression": "0",
                "zlib.output_compression_level": "-1",
                "post_max_size": "128M",
                "upload_max_filesize": "24M",
                "max_execution_time": "480",
                "max_input_time": "240",
                "default_socket_timeout": "360"
            },

            "user": {
                "display_errors": "0"
            }
        }
    }
}

} `

tippexs commented 10 months ago

Thanks for the Configuration. What is the sizing of your Server RAM and CPU?

From the configuratoin I can see you are using static process configuration. In this case your are keeping a constantly high number of PHP worker processes ready to serve requests. Did you try to use dynamic process configuration? http://unit.nginx.org/configuration/#configuration-proc-mgmt-prcs

MrSipping commented 10 months ago

Thanks for the Configuration. What is the sizing of your Server RAM and CPU?

From the configuratoin I can see you are using static process configuration. In this case your are keeping a constantly high number of PHP worker processes ready to serve requests. Did you try to use dynamic process configuration? http://unit.nginx.org/configuration/#configuration-proc-mgmt-prcs

Server is: 4 vcpu ARM 8gb RAM with 2gb swap

tippexs commented 10 months ago

Thanks for sharing. Doing some simple math around the configuratoin you just shared shows that thei server does not have enough memory to serve all the PHP worker.

An example:

{
  "online-v1": {
    "type": "php",
    "processes": 128,
    "user": "www-data",
    "group": "www-data",
    "working_directory": "#########",
    "targets": {},
    "limits": {
      "timeout": 360,
      "requests": 1024
    },
    "environment": {
      "_MY_ID": "##",
      "_MY_ENVIRONMENT": "prod",
      "_MY_RESOURCE_ROOT": "#########",
      "_MY_ENGINE_ROOT": "###########",
      "_MY_AUTOLOADER": "###################",
      "_MY_BOOTSRAP_VERSION": "###"
    },
    "options": {
      "admin": {
        "memory_limit": "512M",
        "variables_order": "EGPCS",
        "expose_php": "0",
        "zlib.output_compression": "0",
        "zlib.output_compression_level": "-1",
        "post_max_size": "128M",
        "upload_max_filesize": "24M",
        "max_execution_time": "480",
        "max_input_time": "240",
        "default_socket_timeout": "360"
      },
      "user": {
        "display_errors": "0"
      }
    }
  }
}

See: https://www.php.net/manual/en/ini.core.php#ini.memory-limit

In this example the max memory a single script on a single PHP worker is allowed to consume is 512M. In this case you are using static process configuration and initally create 128 worker processe. EACH one is allowed to max. utilize 512M. 128 processes * 512 MB/max = 65536 MB of RAM just for this configuration.

This will not play out well and this is just one configuration. The other applications come on top of it. 8GB with 2GB Swap is not enough memory for this configuration. You can use dynamic process configuraiton and decrease the memory_limit per PHP application.

Another question is what application is making use of the maximum memory configured and is there a way to tune it.

Let me know if you have any more questions.

tippexs commented 9 months ago

Closing due to inactivity