search

oss-aspen / 8Knot

Dash app in development to serve open source community visualizations using GitHub data from Augur. Hosted app: https://eightknot.osci.io
MIT License
50 stars 60 forks source link

create visualizations in rq workers rather than server workers #173

Closed JamesKunstle closed 1 year ago

JamesKunstle commented 2 years ago

currently, server workers wait for rq workers to return the data that they need for visualizations, then process the data and create + return the Plotly figure.

this is wasteful of the web server workers- they need to have higher responsiveness and can't be bogged down processing visualizations.

we need to move the processing of visualizations over to the / another worker queue to optimize this.

cdolfi commented 2 years ago

@JamesKunstle lets discuss more on what this might look like. I really want to stay in the callback structure for visualizations if all possible

JamesKunstle commented 2 years ago

@cdolfi I understand what you mean- the general structure would be very similar. the problem being addressed is that for very large data sets (think ceph commits) the webserver workers are running out of memory and getting killed. In general, it is poor practice to have the webserver workers (the gunicorn workers that handle the incoming user's requests) do any heavy lifting, this bogs them down and makes the app feel slow. Since we already have the worker queue working well it would be pretty straightforward to have workers deal with the visualizations / computations too. I'll sketch up a demo of this design and show you the file structure.

JamesKunstle commented 1 year ago

Done- visualizations created in Celery background callbacks rather than in web workers.