search

LUCIT-Systems-and-Development / unicorn-fy

A Python SDK by LUCIT to convert received raw data from crypto exchange API endpoints into well-formed python dictionaries.
https://unicorn-fy.docs.lucit.tech
MIT License
54 stars 20 forks source link

Running for a few hours with this error #41

Open netsesame opened 2 years ago

netsesame commented 2 years ago

Version of this library.

Please post this to your github issue: unicorn_fy: 0.12.2 unicorn_binance_local_depth_cache: not found unicorn_binance_rest_api: not found unicorn_binance_trailing_stop_loss: not found unicorn_binance_websocket_api: 1.41.0

Solution to Issue cannot be found in the documentation or other Issues and also occurs in the latest version of this library.

Hardware?

VPS or other cloud hosting

Operating System?

Linux

Python version?

Python3.8

Installed packages

Package                       Version
----------------------------- -------------------
aioamqp                       0.14.0
aiodns                        2.0.0
aiohttp                       3.7.4.post0
aioquant                      1.0.7
aioredis                      2.0.1
akshare                       1.3.38
aniso8601                     9.0.1
anyio                         3.5.0
asgiref                       3.5.0
async-timeout                 3.0.1
attrs                         21.2.0
bcrypt                        3.2.0
beautifulsoup4                4.10.0
Brotli                        1.0.9
ccxt                          1.54.58
certifi                       2021.5.30
cffi                          1.14.6
chardet                       4.0.0
charset-normalizer            2.0.4
cheroot                       8.5.2
click                         8.0.1
colorama                      0.4.4
cryptography                  3.4.7
cycler                        0.10.0
dateparser                    1.0.0
decorator                     5.1.1
demjson                       2.2.4
et-xmlfile                    1.1.0
fastapi                       0.73.0
Flask                         2.0.1
Flask-Compress                1.10.1
Flask-RESTful                 0.3.9
gunicorn                      20.1.0
h11                           0.13.0
hiredis                       2.0.0
html5lib                      1.1
httptools                     0.3.0
idna                          3.2
itsdangerous                  2.0.1
jaraco.functools              3.3.0
Jinja2                        3.0.1
jsonpath                      0.82
kiwisolver                    1.3.2
lxml                          4.7.1
MarkupSafe                    2.0.1
matplotlib                    3.4.3
more-itertools                8.8.0
motor                         2.0.0
multidict                     5.1.0
numpy                         1.21.1
openpyxl                      3.0.9
pamqp                         2.3.0
pandas                        1.3.1
paramiko                      2.10.3
pathlib                       1.0.1
patsy                         0.5.1
pika                          1.2.0
Pillow                        8.3.2
pip                           21.2.2
psutil                        5.8.0
py                            1.11.0
py-mini-racer                 0.6.0
pycares                       4.0.0
pycparser                     2.20
pydantic                      1.9.0
pymongo                       3.12.0
PyNaCl                        1.5.0
pyparsing                     2.4.7
pypinyin                      0.44.0
python-binance                1.0.12
python-dateutil               2.8.2
python-dotenv                 0.19.2
pytz                          2021.1
PyYAML                        6.0
regex                         2021.8.3
rel                           0.4.7
requests                      2.26.0
retry                         0.9.2
scipy                         1.7.1
setuptools                    52.0.0.post20210125
six                           1.16.0
sniffio                       1.2.0
soupsieve                     2.3.1
starlette                     0.17.1
unicorn-fy                    0.12.2
urllib3                       1.26.6
uvicorn                       0.17.4
uvloop                        0.16.0
watchgod                      0.7
webencodings                  0.5.1
websocket-client              1.1.1
websockets                    10.3
Werkzeug                      2.0.1
wheel                         0.36.2
xlrd                          2.0.1
yarl                          1.6.3

Logging output

Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-88-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Mon 18 Jul 2022 04:19:29 PM CST

  System load:  0.12               Processes:              182
  Usage of /:   86.6% of 19.62GB   Users logged in:        0
  Memory usage: 35%                IPv4 address for ens17: 10.0.4.2
  Swap usage:   0%

  => / is using 86.6% of 19.62GB
  => There is 1 zombie process.

 * Super-optimized for small spaces - read how we shrank the memory
Traceback (most recent call last):
  File "/root/anaconda3/envs/binancebot/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/binancebot/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/binancebot/lib/python3.8/site-packages/unicorn_binance_websocket_api/manager.py", line 529, in _create_stream_thread
    loop.run_until_complete(socket.start_socket())
  File "/root/anaconda3/envs/binancebot/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/root/anaconda3/envs/binancebot/lib/python3.8/site-packages/unicorn_binance_websocket_api/sockets.py", line 139, in start_socket
    received_stream_data = self.unicorn_fy.binance_com_futures_websocket(received_stream_data_json)
  File "/root/anaconda3/envs/binancebot/lib/python3.8/site-packages/unicorn_fy/unicorn_fy.py", line 129, in binance_com_futures_websocket
    return UnicornFy.binance_futures_websocket(stream_data_json,
  File "/root/anaconda3/envs/binancebot/lib/python3.8/site-packages/unicorn_fy/unicorn_fy.py", line 647, in binance_futures_websocket  
    if stream_data['data']['e'] == 'aggTrade':
KeyError: 'data'

Used endpoint?

binance.com-futures

Issue

Running for a few hours with this error

from unicorn_binance_websocket_api.manager import BinanceWebSocketApiManager
import logging
import time
import threading
import os
from unicorn_fy.unicorn_fy import UnicornFy
from termcolor import colored, cprint
import traceback
from pydantic import BaseModel
import time
from functools import wraps

# create instances of BinanceWebSocketApiManager
binance_com_websocket_api_manager = BinanceWebSocketApiManager(exchange="binance.com-futures", output_default="UnicornFy",throw_exception_if_unrepairable=True)

# configure api key and secret for binance.com for Alice
alice_api_key = ""
alice_api_secret = ""
# create the userData streams
alice_stream_id = binance_com_websocket_api_manager.create_stream('arr', '!userData', stream_label="Alice",
                                                                  stream_buffer_name=True,
                                                                  api_key=alice_api_key, api_secret=alice_api_secret)

info_stream = binance_com_websocket_api_manager.get_stream_list()

print(alice_stream_id)
print('status',info_stream[alice_stream_id]['status'])
print('stop_request',info_stream[alice_stream_id]['stop_request'])#停止请求
print('crash_request',info_stream[alice_stream_id]['crash_request'])#崩溃请求
print('kill_request',info_stream[alice_stream_id]['kill_request'])#杀死请求
print('reconnects',info_stream[alice_stream_id]['reconnects'])#重连次数

# print(binance_com_websocket_api_manager.get_stream_statistic(alice_stream_id))

# 结果转成JSON

def str2json(result):
    return UnicornFy.binance_com_futures_websocket(result)

def getdate2(t):
    '''时间戳转换为时间'''
    t1 = float(t)/1000
    # print(timeStamp)
    t2 = time.localtime(t1)
    # print(timeArray)
    t3 = time.strftime("%Y-%m-%d %H:%M:%S", t2)
    # print(otherStyleTime)
    return t3

# # https://docs.python.org/3/library/logging.html#logging-levels
logging.basicConfig(level=logging.DEBUG,
                    filename=os.path.basename(__file__) + '.log',
                    format="{asctime} [{levelname:8}] {process} {thread} {module}: {message}",
                    style="{")

# Python线程退出条件有以下几种：
# 1、线程内的函数语句执行完毕，线程自动结束
# 2、线程内的函数抛出未处理的异常

def print_stream_buffer_data(binance_websocket_api_manager, stream_id):
    while True:
        #有停止请求，则返回True ，否则返回 'False'。
        try :
            if binance_websocket_api_manager.is_manager_stopping():
                print("遇到错误了，正常运行程序并退出程序")
                exit(0)
            oldest_stream_data_from_stream_buffer = binance_websocket_api_manager.pop_stream_data_from_stream_buffer(stream_id)
            if oldest_stream_data_from_stream_buffer is False:
                time.sleep(0.01)

            else:
                # print(oldest_stream_data_from_stream_buffer)
                #信号状态
                data = UnicornFy.binance_com_futures_websocket(oldest_stream_data_from_stream_buffer)
                event_type = data["event_type"]
                if event_type == "ORDER_TRADE_UPDATE":
                    #币种
                    symbol = data['symbol']
                    #方向
                    side = data['side']
                    #持仓方向
                    position_side = data['position_side']
                    #挂单价格
                    order_price = data['order_price']
                    #实际成交价格
                    order_avg_price = data['order_avg_price']
                    #挂单数量
                    order_quantity = data['order_quantity']
                    current_order_status = data['current_order_status']
                    current_execution_type =  data['current_execution_type']
                    transaction_time = getdate2( data['transaction_time'])

                    last_executed_price = data['last_executed_price']
                    #交易编号 ex android_mJlmVJJhRdpoL2fnSAhB
                    client_order_id =  data['client_order_id']
                    #交易ID  ex 60952023909
                    order_id =  data['order_id']
                    #这个目前没用
                    trade_id = data['trade_id']
                    #下单为 LIMIT  挂单止盈为TAKE_PROFIT
                    order_type = data['order_type']

                    trad_type = None 
                    if order_type == "LIMIT":
                        trad_type = "限价单"
                    elif order_type == "TAKE_PROFIT":
                        trad_type = "止盈止损单"
                    elif order_type == "MARKET":
                        trad_type = "市价单" 

                    temp ={ 
                        "BUYLONG":"买入开多",
                        "BUYSHORT":"买入平空",
                        "SELLSHORT":"卖出开空",
                        "SELLLONG":"卖出平多",
                        'NEWNEW':"挂单未成交",
                        'CANCELEDCANCELED':"撤单",
                        'TRADEFILLED':"挂单已成交",
                        'EXPIREDEXPIRED':"订单已过期",
                        'TRADEPARTIALLY_FILLED':'部分成交'
                    }

                    side_away = temp[side+position_side]
                    acction = temp[current_execution_type + current_order_status]
                    cprint(f"订单状态：{acction}  信号状态：{event_type} 币种：{symbol} 下单动作：{side_away} 挂单价格：{order_price} 成交价格：{order_avg_price} 平仓价格：{last_executed_price} 挂单数量：{order_quantity} 时间：{transaction_time}",
                        'green', 'on_red')

        except BaseException as e :
            print(e)

# start a worker process to move the received stream_data from the stream_buffer to a print function
worker_thread = threading.Thread(target=print_stream_buffer_data, args=(binance_com_websocket_api_manager,
                                                                        alice_stream_id))

# get info about the new stream
# second_multi_stream_info = binance_com_websocket_api_manager.get_stream_info(alice_stream_id)
# print(second_multi_stream_info)
worker_thread.start()
netsesame commented 2 years ago

Screenshot_3 I added a few lines of code to get some information

2022-07-19 12:04:06,547 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.is_stop_as_crash_request(df1e2f89508d-a362-08cb-f0ab-9e2079d5)
2022-07-19 12:04:06,547 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.set_heartbeat(df1e2f89508d-a362-08cb-f0ab-9e2079d5)
2022-07-19 12:04:07,306 [DEBUG   ] 3250714 139674514667264 protocol: % sending keepalive ping
2022-07-19 12:04:07,364 [DEBUG   ] 3250714 139674514667264 protocol: % received keepalive pong
2022-07-19 12:04:08,549 [DEBUG   ] 3250714 139674514667264 sockets: BinanceWebSocketApiSocket.start_socket(df1e2f89508d-a362-08cb-f0ab-9e2079d5, ['arr'], ['!userData'] - Received inner asyncio.TimeoutError
2022-07-19 12:04:08,549 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.is_stop_request(df1e2f89508d-a362-08cb-f0ab-9e2079d5)
2022-07-19 12:04:08,549 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.is_stop_as_crash_request(df1e2f89508d-a362-08cb-f0ab-9e2079d5)
2022-07-19 12:04:08,550 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.set_heartbeat(df1e2f89508d-a362-08cb-f0ab-9e2079d5)
2022-07-19 12:04:09,833 [DEBUG   ] 3250714 139674514667264 unicorn_fy: UnicornFy->binance_futures_websocket({"e": "listenKeyExpired", "E": 1658203450029})
2022-07-19 12:04:09,833 [DEBUG   ] 3250714 139674514667264 unicorn_fy: xiaox_UnicornFy->binance_futures_websocket(False) - key_error_error: 'data'
2022-07-19 12:04:09,833 [DEBUG   ] 3250714 139674514667264 unicorn_fy: xiaox_UnicornFy->stream_data({'e': 'listenKeyExpired', 'E': 1658203450029}) - key_error_error: 'data'
2022-07-19 12:04:09,834 [CRITICAL] 3250714 139674514667264 unicorn_fy: UnicornFy->binance_futures_websocket(False) - error: 'bool' object does not support item assignment
2022-07-19 12:04:09,834 [DEBUG   ] 3250714 139674514667264 unicorn_fy: UnicornFy->binance_futures_websocket(False)
2022-07-19 12:04:09,834 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.is_stop_request(df1e2f89508d-a362-08cb-f0ab-9e2079d5)
2022-07-19 12:04:09,834 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.is_stop_as_crash_request(df1e2f89508d-a362-08cb-f0ab-9e2079d5)
2022-07-19 12:04:09,834 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.set_heartbeat(df1e2f89508d-a362-08cb-f0ab-9e2079d5)
2022-07-19 12:04:09,834 [DEBUG   ] 3250714 139674514667264 unicorn_fy: UnicornFy->binance_futures_websocket({"e": "listenKeyExpired", "E": 1658203450031})
2022-07-19 12:04:09,835 [DEBUG   ] 3250714 139674514667264 **unicorn_fy: xiaox_UnicornFy->binance_futures_websocket(False) - key_error_error: 'data'**
2022-07-19 12:04:09,835 [DEBUG   ] 3250714 139674514667264 **unicorn_fy: xiaox_UnicornFy->stream_data({'e': 'listenKeyExpired', 'E': 1658203450031}) - key_error_error: 'data'**
2022-07-19 12:04:09,835 [CRITICAL] 3250714 139674514667264 unicorn_fy: UnicornFy->binance_futures_websocket(False) - error: 'bool' object does not support item assignment
2022-07-19 12:04:09,835 [DEBUG   ] 3250714 139674514667264 unicorn_fy: UnicornFy->binance_futures_websocket(False)
2022-07-19 12:04:09,835 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.is_stop_request(df1e2f89508d-a362-08cb-f0ab-9e2079d5)
2022-07-19 12:04:09,835 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.is_stop_as_crash_request(df1e2f89508d-a362-08cb-f0ab-9e2079d5)
2022-07-19 12:04:09,835 [DEBUG   ] 3250714 139674514667264 manager: BinanceWebSocketApiManager.set_heartbeat(df1e2f89508d-a362-08cb-f0ab-9e2079d5)

2022-07-19 12:04:09,833 [DEBUG ] 3250714 139674514667264 unicorn_fy: xiaox_UnicornFy->binance_futures_websocket(False) - key_error_error: 'data' 2022-07-19 12:04:09,833 [DEBUG ] 3250714 139674514667264 unicorn_fy: xiaox_UnicornFy->stream_data({'e': 'listenKeyExpired', 'E': 1658203450029}) - key_error_error: 'data'

Screenshot_5 unicorn-fy is no listenerKeyExpired。