[Cluster] ECONNRESET error support

orgrimarr commented 3 years ago

Environment

ArangoJS: 7.2.0
ArangoDB 3.7.5 Cluster
- Tested on docker
- Tested on windows

Description

The arangojs lib in ROUND_ROBIN mode can failover to another arango in case of ECONNREFUSED error.

When an arango server shutting down or du to network issue, an ECONNRESET error may occure.

This error is not handled by arangojs and throw an error instead of retrying

Steps to reproduce

Coordinator run on ports 9001, 9002, 9003

Script
const arangojs = require('./build')

const test = async function(){ const db = new arangojs.Database({ url: ['https://127.0.0.1:9001', 'https://127.0.0.1:9002', 'https://127.0.0.1:9003'], maxRetries: 3, databaseName: '_system', loadBalancingStrategy: "ROUND_ROBIN", agentOptions: { rejectUnauthorized: false } })

db.useBasicAuth('root', '')

while(true){
    console.time('test')
    const cursor = await db.query({
        query: `
            FOR element IN @@collection
            RETURN element
        `,
        bindVars: {
            "@collection": '_users'
        }
    })
    const results = []
    for await (const result of cursor) {
        results.push(result)
    }
    console.timeEnd('test')
    console.log('OK', results.length)
}

}

test() .catch(console.error)


### Cluster
- node1: /usr/bin/arangodb --ssl.auto-key
- node2: /usr/bin/arangodb --ssl.auto-key --starter.join=db1
- node3: /usr/bin/arangodb --ssl.auto-key --starter.join=db1
> docker-compose.yaml
```yaml
version: '3.7'
services:
  db1:
    image: arangodb:3.7.5
    container_name: db1
    command: /usr/bin/arangodb --ssl.auto-key
    environment:
      ARANGO_ROOT_PASSWORD:
    ports:
      - 9001:8529
    volumes:
      - ./db1:/data
  db2:
    image: arangodb:3.7.5
    container_name: db2
    command: /usr/bin/arangodb --ssl.auto-key --starter.join=db1
    environment:
      ARANGO_ROOT_PASSWORD:
    ports:
      - 9002:8529
    volumes:
      - ./db2:/data
  db3:
    image: arangodb:3.7.5
    container_name: db3
    command: /usr/bin/arangodb --ssl.auto-key --starter.join=db1
    environment:
      ARANGO_ROOT_PASSWORD:
    ports:
      - 9003:8529
    volumes:
      - ./db3:/data

Steps

Run the cluster
Run the script
Kill an arango instance

Error

OK 1
test: 3.292ms
OK 1
Error: socket hang up
    at connResetException (internal/errors.js:609:14)
    at TLSSocket.socketOnEnd (_http_client.js:459:23)
    at TLSSocket.emit (events.js:326:22)
    at endReadableNT (_stream_readable.js:1223:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ECONNRESET',
  request: ClientRequest {
    _events: [Object: null prototype] {
      response: [Function],
      timeout: [Function],
      error: [Function]
    },
    _eventsCount: 3,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    _last: false,
    chunkedEncoding: false,
    shouldKeepAlive: true,
    useChunkedEncodingByDefault: true,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    _contentLength: null,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    socket: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: true,
      _SNICallback: null,
      servername: false,
      alpnProtocol: false,
      authorized: false,
      authorizationError: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE',

Proposition

Allow failover for ECONNRESET errors
connection.ts, line 60X
- Delete the syscall test
- Add a case for ECONNRESET

error.ts. This is kind a dirty fix

Example:

if (
!task.host &&
this._shouldRetry &&
task.retries < (this._maxRetries || this._hosts.length - 1) &&
isSystemError(err) &&
(err.code === "ECONNREFUSED" || err.code === "ECONNRESET" )
)

export function isSystemError(err: any): err is SystemError {
return (
Object.getPrototypeOf(err) === Error.prototype &&
err.hasOwnProperty("code") 
);
}

pluma commented 3 years ago

This is intentional. ECONNRESET can occur after a requests has started processing. Retrying the request could result in data loss or corruption. ECONNREFUSED can only occur before the request has been started processing, so it can be retried safely.

pluma commented 3 years ago

FYI arangojs 7.3.0 now sets the default agentOptions socket reuse to lifo which should reduce the likelihood of encountering connection timeouts when making infrequent requests. This will not prevent this from happening but it should be less frequent on versions of Node where this is not the default.

arangodb / arangojs