Duplicate rows watchTrades gateio

little-bit-time commented 1 month ago

Operating System

debian

Programming Languages

JavaScript

CCXT Version

4.3.67

Description

Hello, I'm receiving duplicate websocket trades.

serv_time;trade_time:side;price;vol
1721870532569;1721870532547;buy;64421;1
1721870532571;1721870532547;buy;64421.1;1
1721870532571;1721870532547;buy;64421.2;1
1721870532571;1721870532547;buy;64421.3;1
1721870532573;1721870532547;buy;64421.1;1
1721870532573;1721870532547;buy;64421.2;1
1721870532573;1721870532547;buy;64421.3;1
1721870532576;1721870532547;buy;64421.1;1
1721870532576;1721870532547;buy;64421.2;1
1721870532576;1721870532547;buy;64421.3;1
1721870532646;1721870532625;sell;64429.3;929
1721870532772;1721870532749;buy;64429.4;2
1721870532777;1721870532749;buy;64433.1;9
1721870532821;1721870532807;buy;64435.9;2
1721870532912;1721870532893;buy;64439.3;6
1721870532912;1721870532893;buy;64439.3;17
1721870532915;1721870532893;buy;64439.3;6
1721870532915;1721870532893;buy;64439.3;17
1721870532983;1721870532961;sell;64439.8;1
1721870533537;1721870533521;buy;64439.9;2
1721870533537;1721870533521;buy;64439.9;8
1721870533540;1721870533521;buy;64439.9;2
1721870533540;1721870533521;buy;64439.9;8
1721870533737;1721870533716;buy;64439.9;27
1721870533737;1721870533716;buy;64439.9;46
1721870533739;1721870533716;buy;64439.9;27
1721870533739;1721870533716;buy;64439.9;46
1721870533747;1721870533725;buy;64440.8;3
1721870533747;1721870533725;buy;64442.3;2
1721870533747;1721870533726;buy;64442.8;1
1721870533747;1721870533726;buy;64448.6;14
1721870533747;1721870533726;buy;64448.8;2
1721870533747;1721870533730;buy;64449.7;32

After reviewing the code and printing to the console, I see that the limit set with limit = trades.getLimit(tradeSymbol, limit); fluctuates. It starts at 1, 2, 3, 4, and then goes back to 1.

    async watchTradesForSymbols(symbols, since = undefined, limit = undefined, params = {}) {
        /**
         * @method
         * @name gate#watchTradesForSymbols
         * @description get the list of most recent trades for a particular symbol
         * @param {string} symbol unified symbol of the market to fetch trades for
         * @param {int} [since] timestamp in ms of the earliest trade to fetch
         * @param {int} [limit] the maximum amount of trades to fetch
         * @param {object} [params] extra parameters specific to the exchange API endpoint
         * @returns {object[]} a list of [trade structures]{@link https://docs.ccxt.com/#/?id=public-trades}
         */
        await this.loadMarkets();
        symbols = this.marketSymbols(symbols);
        const marketIds = this.marketIds(symbols);
        const market = this.market(symbols[0]);
        const messageType = this.getTypeByMarket(market);
        const channel = messageType + '.trades';
        const messageHashes = [];
        for (let i = 0; i < symbols.length; i++) {
            const symbol = symbols[i];
            messageHashes.push('trades:' + symbol);
        }
        const url = this.getUrlByMarket(market);
        const trades = await this.subscribePublicMultiple(url, messageHashes, marketIds, channel, params);
        if (this.newUpdates) {
            const first = this.safeValue(trades, 0);
            const tradeSymbol = this.safeString(first, 'symbol');
            limit = trades.getLimit(tradeSymbol, limit);
        }
        return this.filterBySinceLimit(trades, since, limit, 'timestamp', true);
    }

Could you help me understand what might be causing this issue? Thank you.

Code

const gateWS = new ccxt.pro.gate({
    apiKey: GateApi,
        secret: GateApiSecret,

        'options': {
            'defaultType': 'swap',
        }
});

while (true) {
    const watchTrades = await gateWS.watchTrades("BTC/USDT:USDT");
    for (const [keyTrades, valueTrades] of Object.entries(watchTrades)) {
        let time = Date.now();
        let log = time + ";" + valueTrades.timestamp + ";" + valueTrades.side + ";" + valueTrades.price + ";" + valueTrades.amount + "\r\n";
    }

}

pcriadoperez commented 1 month ago

Thanks for reporting @little-bit-time! I'll check this with the team on the best solution and open a PR shortly. Below are the proposed solutions:

I investigated and the following is happening:

one ws message is received from the exchange with 3 trades.
For each trade a messageHash is resolved
The first trade gets resolved and returned to the user
The rest are put in the message queue
On the second watch call the are returned from

Issues of current behavior:

watch function returns duplicated trades. The user sees:

- [ Trade 1 ] 
- [ Trade 2, Trade 3, Trade 4 ]
- [ Trade 2, Trade 3, Trade 4 ]
- [[ Trade 2, Trade 3, Trade 4 ]

Solution 1 (set useMessageQueue to false):

This has the risk that if no watch function is awaiting and a trade is resolved the user will miss the update, until the next update happens, which for low volume symbols could be an issue. What the user would see:
```
- [ Trade 1 ] 
- [ Trade 2, Trade 3, Trade 4 ]
```

Solution 2 (Edit filterBy Limit so if limit is 0 it returns an empty array):

What the user would see:

   - [ Trade 1 ] 
  - [ Trade 2, Trade 3, Trade 4 ]
  - [ ]
   - [ ]

Solution 3 (Recusively call watch function if array.getLimit returns 0):

Risks a queue of recursion
Would have to apply probably to all exchanges What the user would see:
```
- [ Trade 1 ] 
- [ Trade 2, Trade 3, Trade 4 ]
```
Solution 4 (handleTrades resolve after all trades are parsed):

What the user would see:
```
- [ Trade 1 ] 
- [ Trade 2, Trade 3, Trade 4 ]
```

pcriadoperez commented 1 month ago

I implemented solution 4 locally, however gateio sends several messages at the exact same time, so it did not solve the issue. This also raised an issue in ArrayCache, that when calling getLimit, the limit is not reset until the next time an item is pushed to the array. I'm not sure if this is expected behavior.

pcriadoperez commented 1 month ago

It is not pretty but my recommendation would be:

Implement solution 4 to reduce the client.resolve calls, Implement solution 2, so a limit of 0 returns an empty array, edit the arrayCache so getLimit resets the limit and does not wait until the next item is pushed. And in this case of gate implement also solution 3 to avoid returning empty arrays to the user. I'll open a PR with this proposal

pcriadoperez commented 1 month ago

@little-bit-time , unitl we merge this solution I would recommend you set the following option to stop seeing duplicates: exchange.options['ws']['useMessageQueue'] = false

little-bit-time commented 1 month ago

Thanks for the proposed solution

        'options': {
            'defaultType': 'swap',
            'ws': {
                'useMessageQueue': false
            }
        }

I'll test it now.

little-bit-time commented 1 month ago

I'm not sure about duplicates.

1721936533623;1721936533599;sell;64810.3;4
1721936533717;1721936533686;sell;64805.8;10
1721936533717;1721936533686;sell;64805.8;10
1721936533717;1721936533686;sell;64805.8;10
1721936533717;1721936533686;sell;64805.8;10
1721936533720;1721936533686;sell;64805.8;256
1721936534937;1721936534917;buy;64801.1;221

1721936543790;1721936543774;sell;64784.8;2
1721936545023;1721936545003;sell;64778.6;10
1721936545024;1721936545003;sell;64778.6;10
1721936545024;1721936545003;sell;64778.6;10
1721936545024;1721936545003;sell;64778.6;10
1721936545026;1721936545003;sell;64778.6;10
1721936545026;1721936545003;sell;64778.6;10
1721936545026;1721936545003;sell;64778.6;10
1721936545026;1721936545003;sell;64778.6;10
1721936545026;1721936545003;sell;64778.6;10
1721936545026;1721936545003;sell;64778.6;10
1721936545029;1721936545003;sell;64778.6;100
1721936545029;1721936545003;sell;64778.6;209
1721936545029;1721936545003;sell;64778.6;224
1721936545029;1721936545003;sell;64778.6;112
1721936545029;1721936545003;sell;64778.6;10
1721936545029;1721936545003;sell;64778.6;10
1721936545031;1721936545003;sell;64778.3;2
1721936545031;1721936545003;sell;64775.9;2129
1721936545031;1721936545004;sell;64775.8;320

But now it can return trades out of order. 1721936468035 -> 1721936468036 -> 1721936468037 -> 1721936468035 -> 1721936468037 -> 1721936468038 -> 1721936468037

1721936466791;1721936466765;buy;64802.5;35
1721936466791;1721936466765;buy;64803.9;50
1721936466791;1721936466765;buy;64804.2;2
1721936466791;1721936466765;buy;64804.2;5
1721936467790;1721936467775;sell;64805.9;104
1721936467834;1721936467818;buy;64806;309
1721936467860;1721936467834;buy;64808;1
1721936467860;1721936467834;buy;64808.4;1
1721936467860;1721936467834;buy;64808.5;1
1721936468043;1721936468018;buy;64810.3;81
1721936468042;1721936468016;buy;64809.5;60
1721936468069;1721936468035;buy;64810.7;2
1721936468073;1721936468035;buy;64811.5;10
1721936468073;1721936468036;buy;64811.5;2
1721936468073;1721936468037;buy;64811.7;6
1721936468073;1721936468035;buy;64811.5;6
1721936468073;1721936468037;buy;64811.7;3
1721936468073;1721936468037;buy;64811.7;1
1721936468073;1721936468037;buy;64811.7;3
1721936468073;1721936468037;buy;64811.7;1
1721936468074;1721936468037;buy;64811.7;1
1721936468074;1721936468038;buy;64812.8;4
1721936468074;1721936468037;buy;64811.7;1
1721936468079;1721936468043;buy;64813.6;24
1721936468085;1721936468043;buy;64814;1
1721936468085;1721936468043;buy;64816;7
1721936468098;1721936468071;buy;64817.2;2
1721936468118;1721936468090;buy;64819;1
1721936468118;1721936468090;buy;64819;2
1721936468152;1721936468128;buy;64819.2;7
1721936468331;1721936468314;buy;64820;214
1721936468630;1721936468610;sell;64819.9;5
1721936468663;1721936468642;sell;64819.9;341
1721936468663;1721936468644;sell;64819.9;168
1721936468664;1721936468644;sell;64819.9;85

csotto commented 1 month ago

I am having the same problem on BINANCE, using Python and ccxt 4.3.68. Although I can understand @pcriadoperez comments I would like to point out that I did not have this issue before. I tested ccxt 4.3.50 and it seemed OK, but ccxt 4.3.60 and 4.3.68 are both generating a lot of duplicates. Please let me know if this helps.

ccxt / ccxt