Use a cadCAD task queue

commons-stack / commons-simulator

Giving people the ability to easily simulate their own custom Commons with cadCAD

https://sim.commonsstack.org/

33 stars 15 forks source link

Use a cadCAD task queue #1

Closed sembrestels closed 3 years ago

sembrestels commented 4 years ago

Our cadCAD servers will break down if multiple users try to simulate cadCAD at the same time

Use as inspiration: https://testdriven.io/blog/developing-an-asynchronous-task-queue-in-python/

BenSchZA commented 4 years ago

I've done some work with task queues in Python before, using Celery: see http://www.celeryproject.org/ & https://github.com/BenSchZA/wattie-whatsapp-bot

What exactly is the purpose of the task queue? Is this to queue up requests to the cadCAD model from different user sessions?

Is the limitation of running concurrent requests with cadCAD that the server can't handle the load, or that there is a limitation with access to shared resources?

GriffGreen commented 4 years ago

Its an internal cadCAD problem. My understanding is that the cadCAD software itself wasn't written with parallelization in mind... I might be getting some piece of the equation wrong... but for sure that is the core problem that blockscience warned us about...

BenSchZA commented 4 years ago

These are some of the points that came up during brainstorming session w/ Griff & Andrew:

caching / memoization of results (could be issue because of randomization of simulations)
task queue (potentially with limited number of simulations in parallel?)
the distinction between the issue of vectorization/parallel processing of cadCAD simulations vs. running simulation in separate process for each user session
UX - countdown (store average simulation calculation time and provide this, along with queue size, to frontend)
display previous results while you wait (considering parameters/results would be stored for scoring gamification down the line, we could potentially do this)
user sessions (we need to handle user requests to the web server well)

I'll write up a clearer spec/scope before going ahead with this. @sembrestels let me know if you have any thoughts about the above. I think there are a few options I can try.

Eithcowich commented 4 years ago

If there's an issue with the simulation taking longer to generate than the players would find acceptable, maybe don't allow a range of values? I mean the value a user enters doesn't need to be continuous between a and b, it could be just a small series, like a1... a10 and that's it. (of course this depends on the number of parameters involved.)

Then you cache all the options on the server, send json to the javascript front end, and everything flies.

I don't have nearly enough information on cadCAD, your servers, or the game, so this is based on a totally general speculation, sorry if I put noise into the system.

BenSchZA commented 4 years ago

@Eithcowich that is an option, and the parameters are currently limited to a range with a slider, but there is some randomness in the simulation. So we might have to remove that randomness if we can't handle actually allowing the users to run a new simulation each time. The other issue, although maybe less of an issue compared to running a new simulation each time, is that there will be a lot of permutations for the current number of parameters, as you mention. Good suggestion and something to consider :)

BenSchZA commented 4 years ago

I've done a few experiments, running multiple parallel requests, using seq 1 10 | xargs -n1 -P4 curl _.

In more recent versions, Flask is multi-threaded by default. Matplotlib is unfortunately not thread-safe, although after a few tweaks it does actually work, with minor issues.

The first issue is that the Matplotlib figures are saved and accessed as static files on the server, which means you can't have concurrent user sessions without a file access error being thrown. To solve this I've temporarily added threading.current_thread().ident as a makeshift ID to create a unique plot for each request. We do need to add proper file lifecycle management in future, but for now it just proved that Flask and cadCAD can in fact handle multi-threading.

Another option, if Matplotlib not being thread-safe causes issues, is using the Python multiprocessing library, but this is limited by the server cores.

BenSchZA commented 4 years ago

Currently, the implementation solves the issue of the server breaking down due to multi-threading issues, but the task queue may be necessary in future for server resource issues.