Cannot run the event loop while another loop is running error when trying to run stream via celery task

[WildsG]

Check this or we will delete your issue. (fill in the checkbox with an X like so: [x])

• [x] I have searched for other issues with the same problem or similar feature requests.

Select one:

• [x] Bug
• [ ] Feature Request
• [ ] Technical Help
• [ ] Other

Environment

• [ ] Are you using the module on a VPS or other Cloud hosting?
• [ ] Are you using the module on a Raspberry Pi?

What kind of internet connection do you have?

Wifi, Download/Upload 100Mbps

Average system load (CPU)

30% 6 cores

Hardware specification

16 GB Ram, 256GB ssd, AMD Ryzen 5 4500U 3948Mhz, Amd Radeon(TM) Graphics 512MB

Operating System? (include version)

• [ ] macOS
• [x] Windows
• [ ] Linux (include flavour)

Options

• [x] stream_buffer
• [ ] process_stream_data

Which endpoint do you connect?

Endpoint - binance.com

Python Version Requirement

• [x] I am using Python 3.6.1 or above

Exact Python Version?

3.9.5

Pip Version?

21.0.1

Dependencies

Package                       Version
----------------------------- ---------
alembic                       1.6.5
amqp                          2.6.1
aniso8601                     9.0.1
astroid                       2.5.6
billiard                      3.6.4.0
celery                        4.4.7
certifi                       2021.5.30
cffi                          1.14.5
chardet                       4.0.0
cheroot                       8.5.2
click                         8.0.1
colorama                      0.4.4
Flask                         2.0.1
Flask-RESTful                 0.3.9
flower                        0.9.7
gevent                        21.1.2
greenlet                      1.1.0
humanize                      3.9.0
idna                          2.10
isort                         5.9.1
itsdangerous                  2.0.1
jaraco.functools              3.3.0
Jinja2                        3.0.1
kombu                         4.6.11
lazy-object-proxy             1.6.0
Mako                          1.1.4
MarkupSafe                    2.0.1
mccabe                        0.6.1
more-itertools                8.8.0
nest-asyncio                  1.5.1
pathlib                       1.0.1
pip                           21.0.1
prometheus-client             0.8.0
psutil                        5.8.0
psycopg2                      2.9.1
pycparser                     2.20
pylint                        2.8.3
python-dateutil               2.8.1
python-dotenv                 0.17.1
python-editor                 1.0.4
pytz                          2021.1
redis                         3.5.3
requests                      2.25.1
setuptools                    54.1.2
six                           1.16.0
SQLAlchemy                    1.4.18
toml                          0.10.2
tornado                       6.1
ujson                         4.0.2
unicorn-binance-websocket-api 1.30.0
unicorn-fy                    0.11.0
urllib3                       1.26.5
vine                          1.3.0
websocket-client              1.1.0
websockets                    8.1
Werkzeug                      2.0.1
wheel                         0.36.2
wrapt                         1.12.1
zope.event                    4.5.0
zope.interface                5.4.0

UNICORN Binance WebSocket API Version?

1.30.0

Description of your issue

I've written a script for scrapping data from binance via websockets, it works standalone - when I simply run the python script from command line. Unfortunately, this is not enough for me and I want to manage my scripts via "celery" task manager in python (package is listed above).

When I'm trying to run my code via a celery task, it fails on BinanceWebSocketApiManager._create_stream_thread() and shows me this error:

[2021-06-23 09:34:47,508: CRITICAL/MainProcess] BinanceWebSocketApiManager._create_stream_thread() stream_id=e5c60520-eb23-4410-9b0f-f9a5d7f12f99 error: 7 - Cannot run the event loop while another loop is running - if this stream did not restart after this error, please create an issue: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/issues/new/choose

[2021-06-23 11:20:31,852: WARNING/MainProcess] c:\users\{myuser}\appdata\local\pypoetry\cache\virtualenvs\humbility-jpvk7jk4-py3.8\lib\site-packages\unicorn_binance_websocket_api\unicorn_binance_websocket_api_manager.py:458: RuntimeWarning: coroutine 'BinanceWebSocketApiSocket.start_socket' was never awaited
  loop.close()

It appears to be that somehow multithreading in your package and in celery conflict with each other in some way, because as I mentioned earlier, my code works well when I just run it from command line.

To reproduce this error, you should simply setup celery on your machine and try to run a task, which has a " binance_websocket_api_manager.create_stream" call.

This should trigger the error, as I can't really run any tasks that would execute stream connection from your library.

Some Notes: I've been reading about similar issue others were having and it appears that the culprit is from asyncio package your code uses. Here's the post: https://stackoverflow.com/questions/46827007/runtimeerror-this-event-loop-is-already-running-in-python

[oliver-zehentleitner]

Hello WildsG!

I dont know celery, i only have read the abstract desctiption. From my understanding, it is made to organize multiple request based routines but websockets are not request based, its a persistent data stream. To work request based you should switch to REST api instead.

If you have 3 request based tasks you can schedule them with celery that seems to be similar to asyncio or actually is using asyncio in the background. create_stream() does not need such a handling, because its a not blocking function and running paralell in a separate thread.

What you can try is the nested_async implementation that is suggested in the posts of your investigation. But i am not sure if thats really the "right" way.

Best regards, Oliver

[oliver-zehentleitner]

Feel free to reopen if needed....