Bug with FTX open_interest

[GoldenNaim]

Hello, here a part of my config.yaml :

exchanges:
    FTX:
        channel_timeouts:
            liquidations: -1
            open_interest: -1
        retries: -1
        liquidations: [BTC-PERP]
        open_interest: [BTC-PERP]

And the bug when I launch cryptostore :

Task exception was never retrieved
future: <Task finished coro=<FTX._open_interest() done, defined at /root/.local/lib/python3.7/site-packages/cryptofeed/exchange/ftx.py:81> exception=TypeError('args must not contain None')>
Traceback (most recent call last):
  File "/root/.local/lib/python3.7/site-packages/cryptofeed/exchange/ftx.py", line 116, in _open_interest
    receipt_timestamp=time()
  File "/root/.local/lib/python3.7/site-packages/cryptofeed/feed.py", line 148, in callback
    await cb(**kwargs)
  File "/root/.local/lib/python3.7/site-packages/cryptofeed/backends/backend.py", line 60, in __call__
    await self.write(feed, pair, timestamp, receipt_timestamp, data)
  File "/root/.local/lib/python3.7/site-packages/cryptofeed/backends/redis.py", line 40, in write
    await self.redis.xadd(f"{self.key}-{feed}-{pair}", data)
  File "/root/.local/lib/python3.7/site-packages/aioredis/commands/streams.py", line 105, in xadd
    return self.execute(b'XADD', stream, *args)
  File "/root/.local/lib/python3.7/site-packages/aioredis/commands/__init__.py", line 51, in execute
    return self._pool_or_conn.execute(command, *args, **kwargs)
  File "/root/.local/lib/python3.7/site-packages/aioredis/pool.py", line 196, in execute
    fut = conn.execute(command, *args, **kw)
  File "/root/.local/lib/python3.7/site-packages/aioredis/connection.py", line 326, in execute
    raise TypeError("args must not contain None")
TypeError: args must not contain None

FTX open_interest work fine with cryptofeed. Thank you

[bmoscon]

that error is coming from redis, so it seems like you haven't set that part of the config up correctly.

[GoldenNaim]

Very strange, it works with all other :

redis:
    ip: '127.0.0.1'
    port: 6379
    socket: null
    del_after_read: true
    retention_time: null
    start_flush: true

exchanges:
    BITMEX:
        channel_timeouts:
            liquidations: -1
            open_interest: -1
            trades:  -1
            funding: -1
        retries: -1
        trades: [XBTUSD]
        funding: [XBTUSD]
        liquidations: [XBTUSD]
        open_interest: [XBTUSD]
    DERIBIT:
        channel_timeouts:
            liquidations: -1
            open_interest: -1
            trades:  -1
            funding: -1
        retries: -1
        trades: [BTC-PERPETUAL]
        funding: [BTC-PERPETUAL]
        liquidations: [BTC-PERPETUAL]
        open_interest: [BTC-PERPETUAL]
    BINANCE_FUTURES:
        channel_timeouts:
            liquidations: -1
            open_interest: -1
            trades:  -1
            funding: -1
        retries: -1
        trades: [BTC-USDT]
        funding: [BTC-USDT]
        liquidations: [BTC-USDT]
        open_interest: [BTC-USDT]
    FTX:
        channel_timeouts:
            liquidations: -1
            open_interest: -1
            trades:  -1
            funding: -1
        retries: -1
        trades: [BTC-PERP]
        funding: [BTC-PERP]
        liquidations: [BTC-PERP]
        open_interest: [BTC-PERP]
    KRAKEN_FUTURES:
        channel_timeouts:
            open_interest: -1
            trades:  -1
            funding: -1
        retries: -1
        trades: [PI_XBTUSD]
        funding: [PI_XBTUSD]
        open_interest: [PI_XBTUSD]
    OKEX:
        channel_timeouts:
            open_interest: -1
            trades:  -1
            funding: -1
        retries: -1
        trades: [BTC-USD-SWAP]
        funding: [BTC-USD-SWAP]
        open_interest: [BTC-USD-SWAP]

[yohplala]

Hi @bmoscon,

I ran into the same trouble when connecting Coingecko feed, and I implemented appropriate check in the function that retrieve the data from Coingecko to check when there is a None value to replace it with an empty string ''. I think it is a limitation of Redis (this is my understanding).

Did the one who made the FTX feed implementation the check with Redis?

I confirm on my side that I ran into the same problem with FTX, but with all other exchanges, I am fine. I did not pushed further then.

[yohplala]

But then, yes, maybe there is a way for Redis to support None or null data. I have actually not checked.

[yohplala]

Well, as pointed in this bug report on Redis Github: https://github.com/aio-libs/aioredis/issues/324 'This is currently impossible due to a check for None in the args to RedisConnection::execute which involves the creation of a set from said args. ' Indeed, line 325 of connection.py:

        if None in args:
            raise TypeError("args must not contain None")

The error seems justified.

[GoldenNaim]

Well, as pointed in this bug report on Redis Github: aio-libs/aioredis#324 'This is currently impossible due to a check for None in the args to RedisConnection::execute which involves the creation of a set from said args. ' Indeed, line 325 of connection.py:

        if None in args:
            raise TypeError("args must not contain None")

The error seems justified.

Well, how can we do to store FTX open_interest with cryptostore ? Do we need to change some lines in connection.py ?

[yohplala]

Certainly not, I will not want to touch aioredis source, ah ah (I would certainly not :)) Best thing is that Bryant confirms he gets the same error. If he does, it does be a bug.

If he does not, then there does be a bug in our redis installation :) In the second case, I have no clue where to start looking - I have already spent quite some time optimizing my Redis installation, and a quick Googling of this error does not tell much about the way to solve it. Myself, in the coingecko support I made, I needed to list expected keys of the API response, and re-build a dictionary for Redis or other callbacks to use, checking all values for each key that they are not null or None.

[GoldenNaim]

You have probably right for redis :) ! I think I have found how to fix for FTX :

Line 115 on ftx.py :

 await self.callback(OPEN_INTEREST,
                                                    feed=self.id,
                                                    pair=pair,
                                                    open_interest=oi,
                                                    **timestamp=None,**
                                                    receipt_timestamp=time()
                                                    )

Replace timestamp=None by timestamp=0 , time() or other ?

EDIT : Now it works :-)

[bmoscon]

this has been fixed on the cryptofeed side, so I'm going to close this. Thanks for raising the issue!