Is it possible making progress bar with "Celery + Spark"??

gregbaker / spark-celery

Helper to allow Python Celery tasks to do work in a Spark job.

Apache License 2.0

27 stars 6 forks source link

Is it possible making progress bar with "Celery + Spark"?? #1

Closed AmadeusJ closed 6 years ago

AmadeusJ commented 7 years ago

your 'spark-celery' is very cooool! Can I have a question?? I making django web application, and made the progress-bar with celery. And I want to make the progress-bar for spark appliction to show to the clients! Is it possible?? I got some errors like "ImportError: No module named app.task" like this... Can you give me some idea?? Thank you!

gregbaker commented 6 years ago

I don't think it's possible to communicate back from a task in Celery (other than returning a final result). The only option I see would be to break up the calculation into many tasks, and roll your own progress estimate from how many have completed.