More info about error "Too many handler requests outstanding, your handler isn't running"

[MauDagos]

Hello Pushpin team,

We recently suffered an outage where we noticed that Pushpin had dropped all connection. From the Pushpin error logs, this log message seemed like the most important one:

Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/superpoll.c:140: errno: Resource temporarily unavailable) Too many handler requests outstanding, your handler isn't running: 256 is greater than hot 256 max.

I was wondering if we could get more information about what this log message means, how to avoid it and how to act upon it. Eventually we just restarted Pushpin.

Here's a greater snapshot of the error logs during the time of the outage, in case it helps:

Wed, 02 Oct 2024 15:36:55 GMT [ERROR] (src/task/fd.c:231: errno: None) Error adding fd: -1 or socket: 0x239c4e0 to task wait list.
Wed, 02 Oct 2024 15:36:55 GMT [ERROR] (src/handler.c:329: errno: None) Receive on handler socket failed.
Wed, 02 Oct 2024 15:36:55 GMT [WARN] (src/handler.c:374: errno: None) Skipped invalid message from handler: ipc:///var/run/pushpin/pushpin-m2-out-443
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/superpoll.c:140: errno: Resource temporarily unavailable) Too many handler requests outstanding, your handler isn't running: 256 is greater than hot 256 max.
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/task/fd.c:231: errno: None) Error adding fd: -1 or socket: 0x239c4e0 to task wait list.
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/handler.c:329: errno: None) Receive on handler socket failed.
Wed, 02 Oct 2024 15:36:56 GMT [WARN] (src/handler.c:374: errno: None) Skipped invalid message from handler: ipc:///var/run/pushpin/pushpin-m2-out-443
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/io.c:150: errno: Resource temporarily unavailable) Handshake failed with error code: -29312
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/io.c:195: errno: None) SSL handshake failed: -1
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/superpoll.c:140: errno: Resource temporarily unavailable) Too many handler requests outstanding, your handler isn't running: 256 is greater than hot 256 max.
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/task/fd.c:231: errno: None) Error adding fd: -1 or socket: 0x239c4e0 to task wait list.
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/handler.c:329: errno: None) Receive on handler socket failed.
Wed, 02 Oct 2024 15:36:56 GMT [WARN] (src/handler.c:374: errno: None) Skipped invalid message from handler: ipc:///var/run/pushpin/pushpin-m2-out-443
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/task/fd.c:231: errno: File exists) Error adding fd: 338 or socket: (nil) to task wait list.
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/io.c:173: errno: None) Error sending SSL data.
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/io.c:777: errno: None) Write error when sending all.
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/connection.c:1324: errno: None) Error delivering to MSG listener on FD 338, closing them.
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/task/fd.c:231: errno: File exists) Error adding fd: 338 or socket: (nil) to task wait list.
Wed, 02 Oct 2024 15:36:56 GMT [ERROR] (src/io.c:524: errno: None) ssl_close_notify failed with error code: -1

We are running Pushpin on version 1.35.0.

Thanks in advance!

[jkarneges]

Hi @MauDagos,

These look like mongrel2 logs. Pushpin's use of mongrel2 is considered deprecated for a few years now.

To avoid this issue, I suggest upgrading to the latest Pushpin and using its built-in connection manager instead. When upgrading, you'll want to update the services= setting in your pushpin.conf, by removing mongrel2 and m2adapter from the comma-separated list, and adding connmgr.

If you want to stay on 1.35.0 for some reason, you can still use the newer connection manager but it will have a different name in that case ("condure") and it will need to be installed separately. There are apt packages for it. As of 1.38.0 it comes built-in.

History:

• 1.30.0 (2020-07-29) - Condure introduced as optional replacement for Mongrel2.
• 1.31.0 (2020-11-06) - Use Condure instead of Mongrel2, by default.
• 1.38.0 (2024-01-08) - Move Condure into Pushpin and name the program pushpin-condure.
• 1.40.0 (2024-07-17) - Rename Condure to connmgr.

[MauDagos]

Ok, we will look into updating Pushpin to a newer version. Thanks!