search

tweag / value-at-risk-poc

Proof-of-concept of a portfolio risk analysis tool
http://value-at-risk-poc.herokuapp.com
0 stars 0 forks source link

Scale the number of securities #9

Open jgarber opened 6 years ago

jgarber commented 6 years ago

An idea (misconception?) is a solution like yours won't scale to larger data-sets. I suspect this is easy to test by dialing up the securities count from 100 -> 5000 for a start. ...Then maybe pushing that number until the response-times get "bad" (A question for a UX designer, perhaps?) ...then, finally, thoughts on how hard it would be to extend existing architecture to solve the response-time issue.

jgarber commented 6 years ago

Started moar-securities branch, but it runs out of memory on Heroku because it's generating 10,000 securities in-memory. The JSON alone is 375 MB. 

2018-02-01T13:11:11.606245+00:00 heroku[web.1]: Process running mem=1264M(123.5%)
2018-02-01T13:11:11.606625+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)

Seems like to really flex the muscles of this PoC, we need something more sophisticated than an in-memory DB (#11) and we should exercise calculating VaR on the fly (#5).

jgarber commented 6 years ago

Solved the Heroku R14 issue. It was running like 5 gunicorn workers, which meant the 10,000 securities were being generated in-memory 5x. With one gunicorn worker and a good-sized dyno, it's decently snappy. With a hobby dyno, it's a tad sluggish, but what do you expect for $7/mo?

http://jg-pw-dropshare.s3.amazonaws.com/Screen-Recording-2018-02-01-08-26-26-rDcPQ6a4ov.mp4