Pretzel server exits after Cannot set headers after they are sent to the client

Error in sending API response causes Pretzel server to exit

When a Pretzel server is restarted and one of the Pretzel clients of that server sends an API request without refreshing the tab after the server restart, it sometimes happens that the server gets the following error and exits.

This only seems to happen shortly after a server restart, so while it is annoying it is easy to manage by checking the server and restarting again if needed. As such it is not a high priority.

Error details

Error: Cannot set headers after they are sent to the client
    at new NodeError (node:internal/errors:371:5)
    at ServerResponse.setHeader (node:_http_outgoing:576:11)
    at SendStream.redirect (/mnt/mongodb/home/pretzel/lb4app/node_modules/serve-static/index.js:203:9)
    at SendStream.emit (node:events:520:28)
    at SendStream.redirect (/mnt/mongodb/home/pretzel/lb4app/node_modules/send/index.js:479:10)
    at onstat (/mnt/mongodb/home/pretzel/lb4app/node_modules/send/index.js:727:41)
    at FSReqCallback.oncomplete (node:fs:199:5)

(first noted 2022Mar21)

This has only been seen shortly after a server restart.

These observations have been made while trying to characterise the error so far :

it doesn't always happen; it may be when an existing browser client tab (re)connects to a server which has just restarted; once the server has had client interactions it doesn't seem to occur;
this scenario is not definite - other cases seemed to be different

Debugging will be hampered by the fact that this is intermittent and difficult to reproduce, and may only occur in the production build / environment. Adding some auditing / verification / trace functionality on when reply callbacks are called may provide some information. The error occurs in Node.js framework functions - it may be a Node.js framework issue, which will be checked by updating that, or it may be the Pretzel server endpoint function is not following the required usage of the framework functions, i.e. NodeJs / Express / Loopback.

This looks like one of those issues which are difficult to find, but for which the fix is likely to be a few lines of code.

[x] update Loopback packages and update server Node.js from 16 to 20. 54d75263 upgrade loopback packages using lb4 update
- [ ] test this on the internal test server to see if it has addressed the 'Cannot set headers' issue.

Server exit after failures in getting ontology - improve handling of these errors

In local testing the server result cache was effectively cleared, and as expected the ontology tree was requested. These requests got failures and the failures caused the server to exit. It will be changed to handle these internet connection failures, and not exit.

Detail

On my local I renamed node_modules to rebuild it. The default results cache is in node_modules, so it was effectively cleared.

The server requested the ontology tree from cropOntology.org, and the following errors occurred, and the server exited. curl from cropOntology.org works OK - the cause of the connection errors has not been determined.

AggregateError: 
...    at processTimers (node:internal/timers:514:7) 'Error: connect ETIMEDOUT 104.155.80.76:443\nError: connect ENETUNREACH 2405:dc00:0:3::689b:504c:443 - Local (:::0)'

queueAppend starting:requestName queue 1 0
ontologyGetTree CO_338
ontologyGetNode CO_338 undefined
ontologyGetNode cropOntology.org /tree CO_338 undefined /tree/CO_338:ROOT
ontologyGetNode /tree undefined undefined AggregateError: 
    at internalConnectMultiple (node:net:1117:18)
    at internalConnectMultiple (node:net:1185:5)
    at Timeout.internalConnectMultipleTimeout (node:net:1711:5)
    at listOnTimeout (node:internal/timers:575:11)
    at processTimers (node:internal/timers:514:7) {
  code: 'ETIMEDOUT',
  [errors]: [
    Error: connect ETIMEDOUT 104.155.80.76:443
        at createConnectionError (node:net:1647:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1706:38)
        at listOnTimeout (node:internal/timers:575:11)
        at processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '104.155.80.76',
      port: 443
    },
    Error: connect ENETUNREACH 2405:dc00:0:3::689b:504c:443 - Local (:::0)
        at internalConnectMultiple (node:net:1181:16)
        at Timeout.internalConnectMultipleTimeout (node:net:1711:5)
        at listOnTimeout (node:internal/timers:575:11)
        at processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2405:dc00:0:3::689b:504c',
      port: 443
    }
  ]
}
ontologyGetNode undefined
queueAppend complete:requestName queue 1 1
queueAppend starting:requestName queue 1 0
ontologyGetTree CO_321
ontologyGetNode CO_321 undefined
ontologyGetNode cropOntology.org /tree CO_321 undefined /tree/CO_321:ROOT
Request GET /api/Ontologies/getTree?rootId=CO_338 failed with status code 500. TypeError: Cannot read properties of undefined (reading 'id')
    at Object.ontologyGetChildren (pretzel/lb4app/lb3app/common/utilities/get-ontology.js:43:33)
    at pretzel/lb4app/lb3app/common/utilities/get-ontology.js:31:26
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

Changes implemented

e7f18309 change form of github path of interval-bins in package.json 15602271 handle connection failures when requesting ontology tree 3a14c51b direct dependency on serve-favicon is not required

plantinformatics / pretzel