search

ohenrik / bitfinex

A Python client for the Bitfinex API
MIT License
58 stars 32 forks source link

Unify nonce logic between Rest and Websocket requests to prevent "nonce too small" #19

Closed ohenrik closed 6 years ago

ohenrik commented 6 years ago

Related to issue #18.

Add warning about this in the documentation + unify nonce logic between Rest and Websocket requests to prevent the "nonce too small" error.

gaorufeng commented 6 years ago

I was confused by "nonce too small" for several days. I use an api key for wss_example.py, not used on other py

Is it possible to find "nonce" and "small" keywords by message. then reconnect WSS. I tried to modify wss_example.py

 def cb_auth(self, message):

        LOGGER.info(f"cb_auth received {message}")

        if isinstance(message, list):
            msg_type = message[1]
            if msg_type in self.wssactions:
                self.wssactions[msg_type]()

        # my add
        hb = "hb"
        hb2 =  "'" + hb + "'"

        if hb in message:
            save_message_log = ""
            pass
        else: 
            nowTime=datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
            save_message_log = str(nowTime) + "---" + str(message)
            print(save_message_log)
            print(type(save_message_log))
            file_handle=open(save_message_log_file,'a',encoding='utf-8')

            file_handle.write(save_message_log + '\n')
            file_handle.close()

        my_small = "small" 
        my_nonce = "nonce"

        if my_small in save_message_log and my_nonce in save_message_log:

            print(save_message_log)
            print(type(save_message_log))
            file_handle=open(save_small_message_log_file,'a',encoding='utf-8')
            file_handle.write(save_message_log + '\n')
            file_handle.close()

            sleep(2)
            self.stopit()
            sleep(18)

This does not seem to be a good way .

My message log 2018-09-27 03:22:46---[0, 'fos', []] 2018-09-27 03:22:46---[0, 'fcs', []] 2018-09-27 03:22:46---[0, 'fls', []] 2018-09-27 04:18:14---{'event': 'info', 'version': 2, 'serverId': 'e1d85dc1-7328-4694-ac24-81645c89178d', 'platform': {'status': 1}} 2018-09-27 04:18:14---{'event': 'auth', 'status': 'FAILED', 'chanId': 0, 'code': 10114, 'msg': 'nonce: small'} 2018-09-27 04:18:35---{'event': 'info', 'version': 2, 'serverId': 'c858202b-0ed4-44a7-83d2-9c2b60f647ca', 'platform': {'status': 1}} 2018-09-27 04:18:35---{'event': 'auth', 'status': 'OK', 'chanId': 0, 'userId': 2266936, 'auth_id': '0020fb0a-c098-45d3-ad0f-fc165858d401', 'caps': {'orders': {'read': 1, 'write': 1}, 'account': {'read': 1, 'write': 0}, 'funding': {'read': 1, 'write': 1}, 'history': {'read': 1, 'write': 0}, 'wallets': {'read': 1, 'write': 1}, 'withdraw': {'read': 0, 'write': 0}, 'positions': {'read': 1, 'write': 1}}} 2018-09-27 04:18:35---[0, 'ps', []] 2018-09-27 04:18:35---[0, 'ws', [['exchange', 'USD', 229.08958891, 0, None], ['funding', 'USD', 0, 0, None], ['margin', 'USD', 193.58849932, 0, None], ['funding', 'EOS', 0.02105733, 0, None]]] 2018-09-27 04:18:35---[0, 'os', []] 2018-09-27 04:18:35---[0, 'fos', []] 2018-09-27 04:18:35---[0, 'fcs', []] 2018-09-27 04:18:35---[0, 'fls', []]

gaorufeng commented 6 years ago

I personally feel that the 'nonce: small' problem is caused by multiple threads and local time. I install the NTP client or Simple Network Time Protocol (SNTP) client software on the service, and this problem will occur in large numbers. I find that the time of the Windows vps server always gets slower. I hope my thoughts will help, thank you very much for your program.


2018-09-27 11:47:33---{'event': 'info', 'version': 2, 'serverId': 'b8b9f009-3050-4212-a661-516913218753', 'platform': {'status': 1}}
2018-09-27 11:47:33---{'event': 'auth', 'status': 'FAILED', 'chanId': 0, 'code': 10114, 'msg': 'nonce: small'}
2018-09-27 11:47:54---{'event': 'info', 'version': 2, 'serverId': 'fd124a67-2c87-4633-8cc5-d52333870f44', 'platform': {'status': 1}}
2018-09-27 11:47:55---{'event': 'auth', 'status': 'OK', 'chanId': 0, 'userId': 2266936, 'auth_id': 'ae65a339-39a9-499c-b353-e39dc99164de', 'caps': {'orders': {'read': 1, 'write': 1}, 'account': {'read': 1, 'write': 0}, 'funding': {'read': 1, 'write': 1}, 'history': {'read': 1, 'write': 0}, 'wallets': {'read': 1, 'write': 1}, 'withdraw': {'read': 0, 'write': 0}, 'positions': {'read': 1, 'write': 1}}}
2018-09-27 11:47:55---[0, 'ps', []]
2018-09-27 11:47:55---[0, 'ws', [['exchange', 'USD', 229.08958891, 0, None], ['funding', 'USD', 0, 0, None], ['margin', 'USD', 193.58849932, 0, None], ['funding', 'EOS', 0.02105733, 0, None]]]
2018-09-27 11:47:55---[0, 'os', []]
2018-09-27 11:47:55---[0, 'fos', []]
2018-09-27 11:47:55---[0, 'fcs', []]
2018-09-27 11:47:55---[0, 'fls', []]
2018-09-27 11:56:23---{'event': 'info', 'version': 2, 'serverId': '8f1a92f8-2ddc-431a-af05-953a524a5669', 'platform': {'status': 1}}
2018-09-27 11:56:23---{'event': 'auth', 'status': 'FAILED', 'chanId': 0, 'code': 10114, 'msg': 'nonce: small'}
2018-09-27 11:56:45---{'event': 'info', 'version': 2, 'serverId': 'f4470216-4432-46bf-9b05-da935b7ea69f', 'platform': {'status': 1}}
2018-09-27 11:56:46---{'event': 'auth', 'status': 'OK', 'chanId': 0, 'userId': 2266936, 'auth_id': '7376ecac-1b83-42d0-afb1-e693d7ad157d', 'caps': {'orders': {'read': 1, 'write': 1}, 'account': {'read': 1, 'write': 0}, 'funding': {'read': 1, 'write': 1}, 'history': {'read': 1, 'write': 0}, 'wallets': {'read': 1, 'write': 1}, 'withdraw': {'read': 0, 'write': 0}, 'positions': {'read': 1, 'write': 1}}}
2018-09-27 11:56:46---[0, 'ps', []]
2018-09-27 11:56:46---[0, 'ws', [['exchange', 'USD', 229.08958891, 0, None], ['funding', 'USD', 0, 0, None], ['margin', 'USD', 193.58849932, 0, None], ['funding', 'EOS', 0.02105733, 0, None]]]
2018-09-27 11:56:46---[0, 'os', []]
2018-09-27 11:56:46---[0, 'fos', []]
2018-09-27 11:56:46---[0, 'fcs', []]
2018-09-27 11:56:46---[0, 'fls', []]
2018-09-27 11:57:13---{'event': 'info', 'version': 2, 'serverId': 'e5690e92-7ec9-44b2-85fe-33986d65f4c7', 'platform': {'status': 1}}
2018-09-27 11:57:13---{'event': 'auth', 'status': 'FAILED', 'chanId': 0, 'code': 10114, 'msg': 'nonce: small'}
2018-09-27 11:57:34---{'event': 'info', 'version': 2, 'serverId': 'a63a3127-faac-4fd8-9d6b-578bef331394', 'platform': {'status': 1}}
2018-09-27 11:57:35---{'event': 'auth', 'status': 'OK', 'chanId': 0, 'userId': 2266936, 'auth_id': '143656f3-b142-4fbd-bf46-e7d62eeebae1', 'caps': {'orders': {'read': 1, 'write': 1}, 'account': {'read': 1, 'write': 0}, 'funding': {'read': 1, 'write': 1}, 'history': {'read': 1, 'write': 0}, 'wallets': {'read': 1, 'write': 1}, 'withdraw': {'read': 0, 'write': 0}, 'positions': {'read': 1, 'write': 1}}}
2018-09-27 11:57:35---[0, 'ps', []]
2018-09-27 11:57:35---[0, 'ws', [['exchange', 'USD', 229.08958891, 0, None], ['funding', 'USD', 0, 0, None], ['margin', 'USD', 193.58849932, 0, None], ['funding', 'EOS', 0.02105733, 0, None]]]
2018-09-27 11:57:35---[0, 'os', []]
2018-09-27 11:57:35---[0, 'fos', []]
2018-09-27 11:57:35---[0, 'fcs', []]
2018-09-27 11:57:35---[0, 'fls', []]
2018-09-27 11:58:06---{'event': 'info', 'version': 2, 'serverId': '716760b0-ac2f-4899-b52e-3c60c906fa45', 'platform': {'status': 1}}
2018-09-27 11:58:06---{'event': 'auth', 'status': 'FAILED', 'chanId': 0, 'code': 10114, 'msg': 'nonce: small'}
2018-09-27 11:58:27---{'event': 'info', 'version': 2, 'serverId': '91e978e6-9332-4268-b251-01f3f1b48067', 'platform': {'status': 1}}
2018-09-27 11:58:27---{'event': 'auth', 'status': 'OK', 'chanId': 0, 'userId': 2266936, 'auth_id': '565ea33f-f2ec-4912-a598-41d81eabd3a3', 'caps': {'orders': {'read': 1, 'write': 1}, 'account': {'read': 1, 'write': 0}, 'funding': {'read': 1, 'write': 1}, 'history': {'read': 1, 'write': 0}, 'wallets': {'read': 1, 'write': 1}, 'withdraw': {'read': 0, 'write': 0}, 'positions': {'read': 1, 'write': 1}}}
2018-09-27 11:58:27---[0, 'ps', []]
2018-09-27 11:58:27---[0, 'ws', [['exchange', 'USD', 229.08958891, 0, None], ['funding', 'USD', 0, 0, None], ['margin', 'USD', 193.58849932, 0, None], ['funding', 'EOS', 0.02105733, 0, None]]]
2018-09-27 11:58:27---[0, 'os', []]
2018-09-27 11:58:27---[0, 'fos', []]
2018-09-27 11:58:27---[0, 'fcs', []]
2018-09-27 11:58:27---[0, 'fls', []]
2018-09-27 12:17:05---{'event': 'info', 'version': 2, 'serverId': '2e7e80ca-5429-42c5-91e5-da373048ee34', 'platform': {'status': 1}}
2018-09-27 12:17:05---{'event': 'auth', 'status': 'FAILED', 'chanId': 0, 'code': 10114, 'msg': 'nonce: small'}
2018-09-27 12:17:26---{'event': 'info', 'version': 2, 'serverId': '1eca6b8c-8ce4-4972-b9a4-5d271732483e', 'platform': {'status': 1}}
2018-09-27 12:17:26---{'event': 'auth', 'status': 'OK', 'chanId': 0, 'userId': 2266936, 'auth_id': 'b46f54b3-92ff-4a6b-9e8d-e12635768a66', 'caps': {'orders': {'read': 1, 'write': 1}, 'account': {'read': 1, 'write': 0}, 'funding': {'read': 1, 'write': 1}, 'history': {'read': 1, 'write': 0}, 'wallets': {'read': 1, 'write': 1}, 'withdraw': {'read': 0, 'write': 0}, 'positions': {'read': 1, 'write': 1}}}
2018-09-27 12:17:26---[0, 'ps', []]
2018-09-27 12:17:26---[0, 'ws', [['exchange', 'USD', 229.08958891, 0, None], ['funding', 'USD', 0, 0, None], ['margin', 'USD', 193.58849932, 0, None], ['funding', 'EOS', 0.02105733, 0, None]]]
2018-09-27 12:17:26---[0, 'os', []]
2018-09-27 12:17:26---[0, 'fos', []]
2018-09-27 12:17:26---[0, 'fcs', []]
2018-09-27 12:17:26---[0, 'fls', []]
2018-09-27 12:26:43---{'event': 'info', 'version': 2, 'serverId': '6c1fb10d-932e-40e3-8584-468b0c2832d8', 'platform': {'status': 1}}
2018-09-27 12:26:43---{'event': 'auth', 'status': 'FAILED', 'chanId': 0, 'code': 10114, 'msg': 'nonce: small'}
2018-09-27 12:27:04---{'event': 'info', 'version': 2, 'serverId': 'bbf92310-08fa-4bf5-98a2-776a0d4b0217', 'platform': {'status': 1}}
2018-09-27 12:27:05---{'event': 'auth', 'status': 'OK', 'chanId': 0, 'userId': 2266936, 'auth_id': '324f6b2c-e0fc-46d7-a097-d119ccd468a9', 'caps': {'orders': {'read': 1, 'write': 1}, 'account': {'read': 1, 'write': 0}, 'funding': {'read': 1, 'write': 1}, 'history': {'read': 1, 'write': 0}, 'wallets': {'read': 1, 'write': 1}, 'withdraw': {'read': 0, 'write': 0}, 'positions': {'read': 1, 'write': 1}}}
2018-09-27 12:27:05---[0, 'ps', []]
2018-09-27 12:27:05---[0, 'ws', [['exchange', 'USD', 229.08958891, 0, None], ['funding', 'USD', 0, 0, None], ['margin', 'USD', 193.58849932, 0, None], ['funding', 'EOS', 0.02105733, 0, None]]]
2018-09-27 12:27:05---[0, 'os', []]
2018-09-27 12:27:05---[0, 'fos', []]
2018-09-27 12:27:05---[0, 'fcs', []]
2018-09-27 12:27:05---[0, 'fls', []]
2018-09-27 12:42:01---{'event': 'info', 'version': 2, 'serverId': '2c1e79e0-d3ba-49ed-9170-27c5ad075565', 'platform': {'status': 1}}
2018-09-27 12:42:02---{'event': 'auth', 'status': 'FAILED', 'chanId': 0, 'code': 10114, 'msg': 'nonce: small'}
2018-09-27 12:42:23---{'event': 'info', 'version': 2, 'serverId': '6202a6a1-e15d-49ee-a4d4-ac1e44ce1ab8', 'platform': {'status': 1}}
2018-09-27 12:42:23---{'event': 'auth', 'status': 'OK', 'chanId': 0, 'userId': 2266936, 'auth_id': '596ea46a-746f-449c-9f51-d37a42d2554b', 'caps': {'orders': {'read': 1, 'write': 1}, 'account': {'read': 1, 'write': 0}, 'funding': {'read': 1, 'write': 1}, 'history': {'read': 1, 'write': 0}, 'wallets': {'read': 1, 'write': 1}, 'withdraw': {'read': 0, 'write': 0}, 'positions': {'read': 1, 'write': 1}}}
2018-09-27 12:42:23---[0, 'ps', []]
2018-09-27 12:42:23---[0, 'ws', [['exchange', 'USD', 229.08958891, 0, None], ['funding', 'USD', 0, 0, None], ['margin', 'USD', 193.58849932, 0, None], ['funding', 'EOS', 0.02105733, 0, None]]]
2018-09-27 12:42:23---[0, 'os', []]
2018-09-27 12:42:23---[0, 'fos', []]
2018-09-27 12:42:23---[0, 'fcs', []]
2018-09-27 12:42:23---[0, 'fls', []]
2018-09-27 12:47:07---{'event': 'info', 'version': 2, 'serverId': '85ca86ae-5477-482c-9c5f-5b14e9870dab', 'platform': {'status': 1}}
2018-09-27 12:47:08---{'event': 'auth', 'status': 'FAILED', 'chanId': 0, 'code': 10114, 'msg': 'nonce: small'}
2018-09-27 12:47:29---{'event': 'info', 'version': 2, 'serverId': '69441c55-e18c-4075-97ec-1494c3912605', 'platform': {'status': 1}}
2018-09-27 12:47:29---{'event': 'auth', 'status': 'OK', 'chanId': 0, 'userId': 2266936, 'auth_id': '0dcc955f-8b3d-4c2f-bcf7-0d832e9f1ac3', 'caps': {'orders': {'read': 1, 'write': 1}, 'account': {'read': 1, 'write': 0}, 'funding': {'read': 1, 'write': 1}, 'history': {'read': 1, 'write': 0}, 'wallets': {'read': 1, 'write': 1}, 'withdraw': {'read': 0, 'write': 0}, 'positions': {'read': 1, 'write': 1}}}
2018-09-27 12:47:29---[0, 'ps', []]
2018-09-27 12:47:29---[0, 'ws', [['exchange', 'USD', 229.08958891, 0, None], ['funding', 'USD', 0, 0, None], ['margin', 'USD', 193.58849932, 0, None], ['funding', 'EOS', 0.02105733, 0, None]]]
2018-09-27 12:47:29---[0, 'os', []]
2018-09-27 12:47:29---[0, 'fos', []]
2018-09-27 12:47:29---[0, 'fcs', []]
2018-09-27 12:47:29---[0, 'fls', []]
gaorufeng commented 6 years ago

I found one thing, I don't know if it is correct.


bitfinex/bitfinex/websockets/client.py

  def authenticate(self, callback, filters=None):
        nonce = int(wss_utils.UtcNow() * 1000000)

But the Time is already X10000000

bitfinex/bitfinex/websockets/wss_utils.py

def UtcNow():
    now = datetime.datetime.utcnow()
    return int(float(now.strftime("%s.%f"))*10000000)

bitfinex/bitfinex/websockets/client.py

other UtcNow()

  def ping(self, channel="auth"):
        """Ping bitfinex."""
        client_cid = wss_utils.UtcNow()
        data = {
            'event': 'ping',
            'cid': client_cid
        }

        client_order_id = wss_utils.UtcNow()
        order_op = {
            'cid': client_order_id,
            'type': order_type,
            'symbol': wss_utils.order_pair(pair),
            'amount': amount,
            'price': price,
            'hidden': hidden,
            "flags": sum(flags),
        }
ohenrik commented 6 years ago

Hmm that's true... I seem to remember messing a bit with this. So i guess i forgot to remove the multiplier in th authenticate message.

However the problem in this issue is related to the client_id (or client_cid :roll_eyes: as i managed to write this as) is not relevant here i think as it does not affect the nonce.

I think the problem is that the rest api is using a different way to calculate the nounce: https://github.com/ohenrik/bitfinex/blob/master/bitfinex/rest/restv2.py#L40-L56

This needs to be the same, using the same multiplier. Also the websocket version should allow users to change this.

As to your earlier comment. All the nonce to small messages are related to the client trying to connection with authentication. If you are using machines that have the clock set differently (and not just the time zones set differently) this might be the problem.

judasbenhur commented 6 years ago

might be worth to add that a complete new API key needs to be created once its been used with a websocket connection. For using the v2 rest api that is.

ohenrik commented 6 years ago

@judasbenhur thats a good point, we have now updated the nonce logic so that it is the same on all api versions. However people probably need to use a new key for websockets in version 1.0.0 of the library, so ill add a note about that.