Socket Leak Issues

[joaomirandas]

Hi guys,

I already made my extensive searches over the web and isn't found any tip about that. Sounds like be too specific issues. I have some instances of TotalJS running on my servers. Around 33 simultaneous instances over docker - and inside of each one these i has a lot of routes for API's.Some days ago i started to face 5** errors on my nginx - when i go to check status i saw aborted execution.

Searching over there i discover it's related to "Socket Leak" based on this section

Did you ever see something similar to this? Have any type of tip?

My method to get back online is restart nginx but impossible to be checking this every minute.. I would like to avoid these leaks

[petersirka]

Hi @joaomirandasa, I see it for the first time. Are you sure that the problem can be in Total.js? Can you check the logs of Total.js app if they contain something special? Do you use latest version of Total.js framework from NPM? (I fixed a bug in WebSocket parser previous month). Was the application frozen?

Thank you

[joaomirandas]

It's not specific for TotalJS application, but related to how much traffic i have on API endpoints and WebSocket connections... I freezed previous version of Total because i couldn't update my codes with high frequency - it's too large and complex to that.

My primary question it's to discover if you already faced that before. I will try some improvements on my nginx.conf to try avoid and let you know if i can fix. Thank you!

[petersirka]

Let me know if you will need a help. I wonder where is the problem and of course I'm open for solving of all problems with Total.js framework.

I have improved WebSocket in Total.js v4, there is really powerful - but this hidden problem can be a bug, so I'll watch info from your side.

[joaomirandas]

Solved!

I couldn't find what exactly was wrong so start my moves to migrate application between servers. Everything is dockerized so it's pretty easy to do. Before migrate i tried everything you can think, reinstall nginx, check my routes, load-tests on any ROUTE({...}) - nothing works to avoid issues, during production nginx keeping failing and desarming. I found an script on internet to auto-reboot nginx in case of failure. It works but still be an ugly solution and not professional for production env.

# !/bin/sh
service=$@
/bin/systemctl -q is-active "$service.service"
status=$?
if [ "$status" == 0 ]; then
    echo "*Nginx heartbeat okay"
else
    echo "*Nginx failure on check"
    /bin/systemctl restart "$service.service"
fi

This is an .sh file used with an cron-task every 30s to check status of nginx process and restart if necessary.

I get in contact with nginx support for more information, they can't help me, i available to paid for support but they don't have specialist in this cases.

When i migrate to a fresh new server, dosn't have more issues with socket leaks. I promote an Grafana + InfluxDB to track my daily performance, these last 2 days, normally like a charm. (Ignore start of second graph, my bad on generation)

Again thank you so much for your attention and support. This amazing Total.JS makes everything i can think possible, really powerful tool for creativity. BRegards.

[petersirka]

Thank you for feedback!