search

miguelgrinberg / Flask-SocketIO-Chat

A simple chat application that demonstrates how to structure a Flask-SocketIO application.
http://blog.miguelgrinberg.com/post/easy-websockets-with-flask-and-gevent
MIT License
673 stars 239 forks source link

How to use SocketIO for updating progress of a large task? #4

Closed jithurjacob closed 8 years ago

jithurjacob commented 8 years ago

Hi, I have a large multi user application on blue print which uses pandas and scikitlearn beneath. I would like to use SocketIO to update the user on the events happening on server like performing data cleansing, starting training, training ended, checking accuracy score etc.

Could some one please point me to a sample code for this?

Correct me if I'm wrong. I'm trying to establish a connection when the person clicks a button for training the model. This connection will remain open for a long time and as each stage is passed in server a notification is pushed to the user. There would be multiple users accessing the server at same time performing some heavy data analysis and machine learning.

miguelgrinberg commented 8 years ago

Not sure Socket.IO is really necessary in your case. How much data will need to be transferred between the server doing the heavy computations and the client? And how often?

If the amount of data necessary for the client to display updates is small, then maybe you should consider using regular HTTP polling as a first implementation, and only move to Socket.IO later if you find that you need that.

I have written an article regarding the use of Flask with Celery, which I think may fit your situation. The article is here: http://blog.miguelgrinberg.com/post/using-celery-with-flask. In one of the examples I present, a long running task sends progress updates to the client, which displays a progress bar.

jithurjacob commented 8 years ago

Wow this is exactly what I'm looking for. Thank you

irfanmaroof commented 4 years ago

@miguelgrinberg Your post make sense but why SocketIO is not best solution for this use case ? I need something similar in my application but i don't want to try celery because it would be something really new for my application and don't want to introduce just for for progress bar. Do you have an example with socketIO for long running backend task and realtime progress bar on frontend.

miguelgrinberg commented 4 years ago

@irfanmaroof Celery is not used for the progress bar, it is used for the heavy computational task. As a side effect of having a long task running as a Celery job, you can provide progress updates, but the point of Celery is to run the heavy task.

Unfortunately if you need to do heavy computation you cannot do it within the context of an asynchronous server, so as much as you don't want to introduce a new component, for this type of problem it is usually needed.