handling errors in general

[journeytosilius]

after solving the process rotation to avoid the subscription limit on Binance, sometimes I'm getting much more frequent errors, specially with the ob

    if (ec)
    {
        std::cerr << "subscribe book error: fl=" << fl << ", ec=" << ec << ", emsg=" << emsg << std::endl;

        return false;
    }

how do I handle an error like ec=104 ? I am running this on a thread, so where should one throw the exception in order to catch it and retry / rerun ? Is it possible to read the returned bool when the error triggers somewhere or it's better to throw and exception ?

Thanks

[niXman]

hi,

myhome$> errno 102
ENETRESET 102 Network dropped connection on reset

how do I handle an error like ec=104 ?

it depends...

I am running this on a thread

why you need another thread? just for rotation of binance connection? if yes - the binapi provides a fully asynchronous API, I can't see any reason NOT use just another timer for this...

for example (in pseudocode):

int main() {
auto binance_connection = std::make_shared<...>(...);
asio::timer reconnection_timer(...);
reconnection_timer.expires_from_now(10 hrs);
reconnection_timer.async_wait(
[&binance_connection](const boost::system::error_code& error){
   if ( !error ) {
      auto new_connection = std::make_shared<...>(...);
      // perform what you need, ie connect, subscribe, sync, etc...
     ...
     binance_connection = new_connection;
   }
}
);
}

[journeytosilius]

hey ! thanks for the tip, I will study it. But this is not the problem the rotation is working just fine and I do it externally, since I am also running other stuff at the same time, so it's controlled by an external operator. The problem is that I don't know how to just re-run the asio context :

    ws.part_depth("BTCUSDT", binapi::e_levels::_5, binapi::e_freq::_100ms,
        [](const char *fl, int ec, std::string emsg, auto depths) {
            if ( ec ) {
                std::cerr << "subscribe part_depth error: fl=" << fl << ", ec=" << ec << ", emsg=" << emsg << std::endl;

                return false;
            }

Since with the OB and the singlet symbol ticker book I am getting more frequenty ec=102 and ec=104 and when this happens I just want it to run again immediately

[niXman]

The problem is that I don't know how to just re-run the asio context :

when a handler is called with any error this doesn't mean that a io_context will stop.

one more tip: as usual a trading bots use at least two connections. the first used for collecting data (orders, trades, etc), and the second - for trading. in that case there is no need to synchronize a trades algos when restarting the connection.

[journeytosilius]

ok, so reading your pseudo-code above and this : https://github.com/niXman/binapi/blob/master/examples/websockets/main.cpp I understand that you handle the connection, and on the demo, it stops after 5 seconds ( missed that part, sorry ) thing is ... how do you make it check for errors all the time instead of setting a timer ? thanks so much

[journeytosilius]

Hi ! So, following the pseudo-code provided and reading the asio documentation, I have reached to this :

    boost::asio::io_service ioctx;

    auto ws = std::make_shared<binapi::ws::websockets>(ioctx, "stream.binance.com", "9443");

    const char *marketchar = market.c_str();

    int error = 0;

    auto book_handler = ws->book(marketchar, [&](const char *fl, int ec, std::string emsg, auto book)
                                 {
                if (ec)
                {
                    std::cerr << "subscribe book error: fl=" << fl << ", ec=" << ec << ", emsg=" << emsg << std::endl;
                    error = ec;
                    return false;
                }
                std::cout << book << std::endl;
                return true; });

    boost::posix_time::seconds interval(1);
    boost::asio::deadline_timer timer0{ioctx, interval};
    timer0.async_wait(boost::bind(tick,boost::asio::placeholders::error,&timer0,&interval));

    ioctx.run();

This works and runs the handler every 1 second without blocking. But I can't figure out how and when do I have to pass the error generated by the book_handler in order to handle it ? If you can provide another hint it would be great ... thanks !

[niXman]

But I can't figure out how and when do I have to pass the error generated by the book_handler in order to handle it ?

it depends... but in this case and based on the info I have - you need to stop trading algos and try to restart WS connection...

[niXman]

but I think it's more interesting in this situation to understand why that error occurs...

[journeytosilius]

hey ! Thanks ... yes, I have been doing more testing and the problem comes because this mode, which is the "writer" ( the mode I use to write to the db ) inserts so much data that for some reason the websocket gets disconnected.

I run the writer on a small computer, and while in with the trades stream it did not happen, on the OB stream it does, so I believe it's just that the thread gets clogged since the inserts can't keep up with the rate of incoming data ( there is a visible delay on the inserts ) and there is a point where the websocket gets disconnected. The rate of incoming data is much heavier on the OB stream than on the trades stream so it kinda makes sense. Maybe this is because since there is a long queue, one is not sending back the pong message that Binance requires ?

I am now trying to implement libpq's COPY batch inserts, buffering the incoming data first on a separate thread, then writing in chunks via COPY. But still, it would be interesting to know how to handle the error because it may happen. If it happens, it will restart upon rotation is triggered, so it won't totally die, but it would be nice to be capable to handle it and restart if possible

[niXman]

The rate of incoming data is much heavier on the OB stream than on the trades stream so it kinda makes sense. Maybe this is because since there is a long queue, one is not sending back the pong message that Binance requires ?

I think yes, it is quite possible.. Binance it does not tolerate to slow reading =)

you can use asio's thread pool as queue of data for DB, I think this is the easiest way...

thread_pool pool{2};
my_db db{...};
...

// OB cb:
[&pool, &db](..., auto book) {
   asio::post(
      pool,
      [book=std::move(book)]() {
         db.write(book);
      }
   );
};

[journeytosilius]

thanks, writing works correctly now by buffering and using copy instead of inserts ... I'm going to close this for the time being