trades ws stream out of sync / time drift

[journeytosilius]

frequency gets higher during event

stream gets totally out of sync shortly after ( trades time T was out of sync with Binance for close to 1 minute here already

after a while, it seems that it starts getting past messages again, but the clock is already like 8 minutes out of sync and it doesn't recover

I simply instantiate a buffer with pre-initialized reserved size of 500 and each new message I pushback, then I delete the first when it exceeds the reserved value so to create this sequence, there is nothing complicated on the way I do it. I don't understand why this happens but it looks like it's from Binance's side or maybe from the implementation ? But I don't really know just yet. I have triple-checked everything locally and my hardware too

[journeytosilius]

I have pretty fast fiber link and my software is running on dual xeon proliant servers, so this is not a matter of power or connection quality I believe. Let me know if you have some ideas ...

[niXman]

very strange... can I look at the code?

[journeytosilius]

mainly this

class TradeBuffer
{
public:
    vector<double> tradespriceb;
    vector<double> tradesquantityb;
    vector<int> is_maker;
    vector<size_t> eventtimeb;
    int buffer_size;

    TradeBuffer(int const &buffer_size)
    {
        this->buffer_size = buffer_size;
        tradespriceb.reserve(buffer_size);
        tradesquantityb.reserve(buffer_size);
        is_maker.reserve(buffer_size);
    };

    void add_to_buffer(auto const &trades)
    {
        tradespriceb.push_back(double(trades.p));
        tradesquantityb.push_back(double(trades.q));
        eventtimeb.push_back(size_t(trades.E));
        is_maker.push_back(trades.m);
    }

    void check_buffer_size()
    {
        if (tradespriceb.size() >= 500)
        {
            tradespriceb.erase(tradespriceb.begin());
            tradesquantityb.erase(tradesquantityb.begin());
            eventtimeb.erase(eventtimeb.begin());
            is_maker.erase(is_maker.begin());
        }
    }
};

TradeBuffer buff(500);

const char *marketc = market.c_str();

boost::asio::io_context ioctx;
binapi::ws::websockets ws{ioctx, "stream.binance.com", "9443"};

ws.trade(marketc, [&](const char *fl, int ec, std::string emsg, auto trades)
         {
             if (ec)
             {
                 std::cerr << "subscribe book error: fl=" << fl << ", ec=" << ec << ", emsg=" << emsg << std::endl;

                 return false;
             }

             buff.add_to_buffer(trades);
             buff.check_buffer_size();

             return true;
         });

ioctx.run();

[journeytosilius]

just a commnent, the clock I'm printing is not trades.E but trades.T, which is the trade time. The trades.E vector is in the buffer but I'm not using it currently, so that is irrelevant to the issue.

[niXman]

hmm... I need to think about this...

but I have a suggestion: try to profile your code using valgrind+callgrind, and then provide collected data here, I'll analyze it.

[journeytosilius]

sure, I'll post you soon

[journeytosilius]

valgrind.txt callgrind.txt

Seems that I have a leak when I use the Yaml parser to parse markets from a yaml file, but the rest I don't know how to identify ... maybe you can shed some light ... thank you !

[niXman]

thanks, but I need the file located at /home/xavier/spectre/build/callgrind.out.13278 ;)

[journeytosilius]

callgrind.out.13278.log here :) thanks

[journeytosilius]

meanwhile I will run two procs, one with graphing and one without on the same market and see if that's the actual culprit, but I'm pretty sure I already tested this a week ago and it also happened. It's hard to catch because it seems to happen when frequency gets very high ( sell-off or pumps ) but I will keep on posting findings

[journeytosilius]

[journeytosilius]

The drift is def caused by the graphing app. Sorry and thank you very much