Progress track to the user

[kopytjuk]

User thinks there is a bug in the website, while waiting ...

[linuswagner]

I looked into the code and think that this is relatively easy to achieve if check_availability is approached differently: Instead of retrieving data from justwatch, IMDB and joining the data based on the whole movie batch, this process should be repeated per movie. Then it is just a matter of telling _progresstracker after each iteration that a new movie is ready.

I tried to implement this into the code and a single threaded approach works just fine, but is slow. When using multiple threads I fail to update the state of the task embedded in the _progresstracker using update(): It appears like the message is received correctly by update(). However, the Uvicorn client still shows no progress.

Do you have any experience with updating the state of Celery tasks from within threads?

[kopytjuk]

Hi Linus,

an interesting approach! Currently I do not have much time to work on the website :(

I have no experience with threading and celery but as I know each celery function call is executed within a new process, if you populate the progress tracker there I do not know if it will work. I tried updating from within the justwatch thread pool for one evening but had no idea why it wasn't working ...

Somehow we need to pass a shared variable to all threads and this shared variable has to be part of the progress tracker ...

Best regards!

Linus Wagner @.***> schrieb am Mo., 5. Apr. 2021, 15:03:

I looked into the code and think that this is relatively easy to achieve if check_availability https://github.com/kopytjuk/imdb-checker/blob/982b7b35d0f711bb3d3d60b75d3db3576a6f0fb2/logic/availability.py#L10 is approached differently: Instead of retrieving data from justwatch, IMDB and joining the data based on the whole movie batch, this process should be repeated per movie. Then it is just a matter of telling progress_tracker after each iteration that a new movie is ready.

I tried to implement this into the code and a single threaded approach works just fine, but is slow. When using multiple threads I fail to update the state of the task embedded in the progress_tracker using update(): It appears like the message is received correctly by update(). However, the Uvicorn client still shows no progress.

Do you have any experience with updating the state of Celery tasks from within threads?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kopytjuk/imdb-checker/issues/14#issuecomment-813381362, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADBL2RYOCJCJBF3OIFH7ZM3THGYI5ANCNFSM4X7L4EJA .