Websocket closes and dies

[tuloski]

I have a rippled with the following modif on the rippled.cfg:

[server] port_ws_public

[port_ws_public] port = 8888 ip = 0.0.0.0 protocol = ws

I have multiple peers (locally) that connect to that ws.

If one host requests lots of data via the ws (for example lots of books), the ws closes with code 1000, but:

• It's not technically a code 1000.
• I'm not able to reconnect to the ws until I restart rippled.
• All other peers that were using that ws are disconnected and can't reconnect until restart of rippled.

The following code (with RippleAPI) can be used to trigger the error:

const RippleAPI = require('ripple-lib').RippleAPI;

var util = require('util');
var fs = require('fs');

var CURRENCIES = [
    { 
        currency : 'USD',
          issuer : 'rvYAfWj5gh67oV6fW32ZzP3Aw4Eubs59B'
    },
    { 
        currency : 'BTC',
          issuer : 'rvYAfWj5gh67oV6fW32ZzP3Aw4Eubs59B'
    },
    { 
        currency : 'USD',
          issuer : 'rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq'
    },
    { 
        currency : 'EUR',
          issuer : 'rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq'
    },
    { 
        currency : 'BTC',
          issuer : 'rchGBxcD1A1C2tdxF6papQYZ8kjRKMYcL'
    },
    { 
        currency : 'BCH',
          issuer : 'rchGBxcD1A1C2tdxF6papQYZ8kjRKMYcL'
    },
    { 
        currency : 'ETH',
          issuer : 'rchGBxcD1A1C2tdxF6papQYZ8kjRKMYcL'
    },
    { 
        currency : 'ETC',
          issuer : 'rchGBxcD1A1C2tdxF6papQYZ8kjRKMYcL'
    },
    { 
        currency : 'REP',
          issuer : 'rchGBxcD1A1C2tdxF6papQYZ8kjRKMYcL'
    },
];

api = new RippleAPI({
    //server: 'wss://s-west.ripple.com:443'
    //server: 'wss://s1.ripple.com:443'
    server: 'ws://127.0.0.1:8888',
    maxFeeXRP: '0.1'
});

//----- RIPPLE API EVENTS ----------------
api.on('error', (errorCode, errorMessage) => {
    console.log('Ripple API error:' + errorCode + ': ' + errorMessage);
});

api.on('connected', () => {
    console.log('Ripple API connected');
});

api.on('disconnected', (code) => {
    console.log('Ripple API disconnected, code:', code);
});

api.on('ledger', ledger => {
    console.log('Ripple API Ledger:' + ledger.ledgerVersion);
    let n_books = 0;
    var promises_book = []; //Collect promises from all books in parallel
    let t0 = process.hrtime();
    for (var i in CURRENCIES){
        for (var j in CURRENCIES){
            if (i != j){
                let book_requested = {
                    base : {
                        currency : CURRENCIES[i].currency,
                        counterparty : CURRENCIES[i].issuer,
                    },
                    counter : {
                        currency : CURRENCIES[j].currency,
                        counterparty : CURRENCIES[j].issuer,
                    }
                };          
                promises_book.push(api.getOrderbook('rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq',book_requested,{limit : 100}));            

                n_books++;
            }                   
        }               
    }
    Promise.all(promises_book).then(function(results) {
        let t = process.hrtime();
        console.log('Time for ' + n_books + ' books ms: ' + ((t[0] - t0[0])*1000 + (t[1] - t0[1])/1000000));
    });         
});

api.connect();

[tuloski]

https://www.xrpchat.com/topic/30893-rippled-local-ws-disconnects-and-dies/

[miguelportilla]

Hello Tuloski,

I reproduced the issue using your script. The connections you are opening to the server are non-admin privileged and therefore are rate limited. This is a server mechanism meant to protect against client connections demanding too much in a short period of time.

A few running instances of your script consume enough drop fees fast enough to cross a server threshold that will cause the IP to be disconnected. When this happens, all WebSocket connections from that IP will be disconnected and new connections will not be allowed from the same IP for a period of time. If you set your log severity to info, you will see the warnings and disconnects documented in the log file. The rate limitation does not apply to admin connections.

I also noticed that your script (possibly ripple API?) attempts to reconnect immediately after being disconnected. This aggravates the problem as it will extend the waiting period necessary to connect again.

You have two choices, either use admin connections or slow the number of requests per WebSocket.

-M

[movitto]

See screenshot for attached script output. Note updated return code (1008 for "Policy Violation")

[yinjun1991]

Hello Tuloski,

I reproduced the issue using your script. The connections you are opening to the server are non-admin privileged and therefore are rate limited. This is a server mechanism meant to protect against client connections demanding too much in a short period of time.

A few running instances of your script consume enough drop fees fast enough to cross a server threshold that will cause the IP to be disconnected. When this happens, all WebSocket connections from that IP will be disconnected and new connections will not be allowed from the same IP for a period of time. If you set your log severity to info, you will see the warnings and disconnects documented in the log file. The rate limitation does not apply to admin connections.

I also noticed that your script (possibly ripple API?) attempts to reconnect immediately after being disconnected. This aggravates the problem as it will extend the waiting period necessary to connect again.

You have two choices, either use admin connections or slow the number of requests per WebSocket.

-M

How to use admin connections? please

[MarkusTeufelberger]

Connect on an admin-only port.

[JoelKatz]

The server will also send you specific warnings to slow down. Backing off a second each time you get that warning should help. Look for RPC error code slowDown and JSON response entry "warning" : "load".