search

OpenHausIO / connector

Connector between backend & local network
MIT License
1 stars 2 forks source link

Worker does not exit on `EHOSTUNREACH` error? #62

Open mStirner opened 4 weeks ago

mStirner commented 4 weeks ago

Trying to connect to a invalid/non exisiting/offline tcp socket/server, causes to spwan worker, but do not exit. This creates a memory leak.

Related/debugging information: #61 A watch on watch -n 10 "systemctl status connector" and trying to connect to a offline raspberry increases the Tasks number continusly.

● connector.service - OpenHaus SmartHome/IoT connector
     Loaded: loaded (/usr/lib/systemd/system/connector.service; enabled; preset: disabled)
     Active: active (running) since Sat 2024-10-26 12:46:59 CEST; 2 days ago
       Docs: https://open-haus.io
   Main PID: 51700 (node)
      Tasks: 87 (limit: 11107)
     Memory: 645.1M
        CPU: 17min 31.599s
     CGroup: /system.slice/connector.service
             └─51700 /usr/bin/node index.js

Oct 28 16:47:48 open-haus.lan node[51700]: Worker spawend for url http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:47:51 open-haus.lan node[51700]: [2024.10.28 - 16:47.51.943][error][system] [error] tcp://192.168.2.178:8080 Error: connect EHOSTUNREACH 192.168.2.178:8080
Oct 28 16:47:51 open-haus.lan node[51700]:     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
Oct 28 16:47:51 open-haus.lan node[51700]:   errno: -113,
Oct 28 16:47:51 open-haus.lan node[51700]:   code: 'EHOSTUNREACH',
Oct 28 16:47:51 open-haus.lan node[51700]:   syscall: 'connect',
Oct 28 16:47:51 open-haus.lan node[51700]:   address: '192.168.2.178',
Oct 28 16:47:51 open-haus.lan node[51700]:   port: 8080
Oct 28 16:47:51 open-haus.lan node[51700]: }
Oct 28 16:47:51 open-haus.lan node[51700]: [2024.10.28 - 16:47.51.945][debug][system] [closed] tcp://192.168.2.178:8080
● connector.service - OpenHaus SmartHome/IoT connector
     Loaded: loaded (/usr/lib/systemd/system/connector.service; enabled; preset: disabled)
     Active: active (running) since Sat 2024-10-26 12:46:59 CEST; 2 days ago
       Docs: https://open-haus.io
   Main PID: 51700 (node)
      Tasks: 93 (limit: 11107)
     Memory: 687.9M
        CPU: 17min 34.622s
     CGroup: /system.slice/connector.service
             └─51700 /usr/bin/node index.js

Oct 28 16:50:44 open-haus.lan node[51700]: Worker spawend for url http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:50:47 open-haus.lan node[51700]: [2024.10.28 - 16:50.47.495][error][system] [error] tcp://192.168.2.178:8080 Error: connect EHOSTUNREACH 192.168.2.178:8080
Oct 28 16:50:47 open-haus.lan node[51700]:     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
Oct 28 16:50:47 open-haus.lan node[51700]:   errno: -113,
Oct 28 16:50:47 open-haus.lan node[51700]:   code: 'EHOSTUNREACH',
Oct 28 16:50:47 open-haus.lan node[51700]:   syscall: 'connect',
Oct 28 16:50:47 open-haus.lan node[51700]:   address: '192.168.2.178',
Oct 28 16:50:47 open-haus.lan node[51700]:   port: 8080
Oct 28 16:50:47 open-haus.lan node[51700]: }
Oct 28 16:50:47 open-haus.lan node[51700]: [2024.10.28 - 16:50.47.497][debug][system] [closed] tcp://192.168.2.178:8080
● connector.service - OpenHaus SmartHome/IoT connector
     Loaded: loaded (/usr/lib/systemd/system/connector.service; enabled; preset: disabled)
     Active: active (running) since Sat 2024-10-26 12:46:59 CEST; 2 days ago
       Docs: https://open-haus.io
   Main PID: 51700 (node)
      Tasks: 102 (limit: 11107)
     Memory: 768.2M
        CPU: 17min 40.018s
     CGroup: /system.slice/connector.service
             └─51700 /usr/bin/node index.js

Oct 28 16:56:35 open-haus.lan node[51700]: Worker spawend for url http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:56:38 open-haus.lan node[51700]: [2024.10.28 - 16:56.38.535][error][system] [error] tcp://192.168.2.178:8080 Error: connect EHOSTUNREACH 192.168.2.178:8080
Oct 28 16:56:38 open-haus.lan node[51700]:     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
Oct 28 16:56:38 open-haus.lan node[51700]:   errno: -113,
Oct 28 16:56:38 open-haus.lan node[51700]:   code: 'EHOSTUNREACH',
Oct 28 16:56:38 open-haus.lan node[51700]:   syscall: 'connect',
Oct 28 16:56:38 open-haus.lan node[51700]:   address: '192.168.2.178',
Oct 28 16:56:38 open-haus.lan node[51700]:   port: 8080
Oct 28 16:56:38 open-haus.lan node[51700]: }
Oct 28 16:56:38 open-haus.lan node[51700]: [2024.10.28 - 16:56.38.537][debug][system] [closed] tcp://192.168.2.178:8080

start to watch:
16:23 = 49 16:25 = 51 16:28 = 54 16:33 = 63 16:37 = 69

mStirner commented 4 weeks ago

After restarting the backend, tasks down to 34 and the journal is flooded with disconnects:

Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e567f84469378713d4ad1f/interfaces/66e567f84469378713d4ad21
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: [2024.10.28 - 16:57.55.640][debug][forwarder] Woker "ssdp.js" exited: 1006
Oct 28 16:57:55 open-haus.lan node[51700]: [2024.10.28 - 16:57.55.640][warn][forwarder] Worker exited with error, respawn
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a14572e58e7085f2a2e3/interfaces/66e4a14572e58e7085f2a2e4
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: [2024.10.28 - 16:57.55.644][debug][forwarder] Woker "mdns.js" exited: 1006
Oct 28 16:57:55 open-haus.lan node[51700]: [2024.10.28 - 16:57.55.644][warn][forwarder] Worker exited with error, respawn
mStirner commented 4 weeks ago

A quick fix would be a process.exit(1) in the error listener: sockets/tcp.js

const net = require("net");

module.exports = ({ host, port }) => {

    const logger = require("../system/logger.js");

    let socket = new net.Socket();

    // keep socket alive
    socket.setKeepAlive(true, 5000);
    //socket.unref(); // test for #62, did nothing!

    socket.on("error", (err) => {
        logger.error(`[error] tcp://${host}:${port}`, err);
        // why not cal process.exit() here? 
        // this file/function should only be called in a worker thread
        // so it dosnt matter if the process exits or not...
        // if the socket.unref() approach fails, try the exit method
        process.exit(1);
    });

    socket.on("close", () => {
        logger.debug(`[closed] tcp://${host}:${port}`);
    });

    socket.on("connect", () => {
        logger.info(`[connected] tcp://${host}:${port}`);
    });

    socket.connect(port, host);

    return socket;

};