openpredictionmarkets / socialpredict

Easy to Deploy Prediction Market Platform
https://github.com/openpredictionmarkets/socialpredict
MIT License
62 stars 11 forks source link

Load Testing #227

Open pwdel opened 3 months ago

pwdel commented 3 months ago
pwdel commented 3 months ago

Re: from @astrosnat , Discord

Using pre- v0.0.3, deployed on Digital Ocean.

Image

Image

FYI https://docs.digitalocean.com/products/droplets/how-to/graphs/

Image

Then...

Image

pwdel commented 3 months ago

Created +10 markets, for a total of 11 on https://openprediction.xyz/

Image

This is running on Digital Ocean.

Specs for the box it's running on?

Suggest running a load test again, see where we are at, and then based upon the results of that test, increase the markets up to the point that is significant but perhaps doesn't break it yet, repeat until able to break the website.

astrosnat commented 3 months ago

20240728_Specs 20240728_Graphs Graphs & specs pre load test - 4GB RAM 80 GB basic hard drive Ubuntu 22.04 LTS

Thank you so much for doing this, load testing starts after coffee

Do you have an idea of what will be significant @pwdel? Like say a rough % of CPU usage or something?

astrosnat commented 3 months ago

Load testing with 10 markets: requests start to fail at around 100k requests total with a concurrency of 1000 (think 1000 users trying to access the site at the same time)

20240728_LoadTest3.csv 20240728_LoadTest2.csv 20240728_LoadTest1.csv 20240728_LoadTest3_logs.txt 20240728_LoadTest4_logs.txt 20240728_LoadTest4.csv

Here is how CPU usage etc. got affected:

Image Image

Note that CPU % usage goes above 90% with 10m requests and a concurrency of 1000

pwdel commented 3 months ago

Edit... I may have read your post above wrong and the math below here might be incorrect. My assumptions were based upon 100k requests, the CPU maxing out at close to 90% along with other metrics you show.

If it was in fact 10M requests, then we can potentially multiply all of my calculations below by 100, e.g. 400 markets on the front page, CPU fails or gets close to failing at 2500 users.


OK so this means that there is a huge potential for improvement in serving static files.

That being said ... I wonder how geometric this would be. So if the markets/ page displayed 10X more markets, would that be the equivalent of 100 users accessing the site at the same time?

E.g., so does this mean we're only rated for displaying around say 200 markets on the front page, with 50 users simultaneously right now?

Or, the equivalent of 400 markets on the front page, 25 users (not that we would need 400 markets, but something equivalent, processing wise).

I don't necessarily see us needing to support 25 users with that much activity by say, September, but I could ask more from @jg-you and his class as a hypothetical. There could be a safety factor that we build in.

Just trying to think about project planning and whether it's worth it to build in, "serve static files for prod," right away now or keep it more as a dev product.

astrosnat commented 3 months ago

Well, I can create 90 dummy markets and see what happens. My feeling is that it makes sense to build in "serve static files for prod". I would be targeting ~100 scientists to use this - for example I could talk to the UK Reproducibility Network (a group of scientists focused on improving reproducibility) and the University of Sheffield metascience department. It also fits in pretty nicely with UKRI's new pivot to metascience. (UKRI is the umbrella research funder for the UK.)

Wonder if we could share the SocialPredict software with the Open Science Foundation too. I'm sure some OSF folks would be really into it.

astrosnat commented 3 months ago

So I didn't manage to create 90 extra dummy markets...I got up to about 50 and ran out of extra credits. And then when I tried to create a new account, this happened when trying to change its password:

20240730_JSONerror

Load testing today: 20240730_LoadTest2_logs.txt 20240730_LoadTest2.csv 20240730_LoadTest1_logs.txt 20240730_LoadTest1.csv

and screenshots: 20240730_LoadTesting2 20240730_LoadTesting1

pwdel commented 3 months ago

hey, I had already commented this on Discord but putting it here for posterity.

Get the logs from the various containers, probably frontend first, to see where that error came from. Post the error here when you can and we'll generate a ticket based off of that.

https://github.com/openpredictionmarkets/socialpredict/blob/main/README/LOCAL_SETUP.md#getting-logs-from-different-containers

astrosnat commented 3 months ago

20240802_ErrorLogs.txt logs from frontend + backend

astrosnat commented 1 month ago

Load testing on openprediction.xyz after QoL fix to leverage vite build (PR #341)

1GB RAM | 25GB Disk

20240930_LoadTest5_logs.txt Screenshot from 2024-09-30 15-09-34 20240930_DOMetrics.txt (I'll make the metrics into a nice graph later) 20240930_LoadTest5.csv

10000 requests, 1000 requests at a time

Note: we are already at 32.1% disk usage, which might imply needing to resize the droplet at some point in time

pwdel commented 1 month ago

Looks like less memory usage as well... down to ~55% of RAM from the previous 70%.

pwdel commented 1 month ago

From Discord:

image

BrierFoxForecast ... using old version of master from late August:

Disk:

image

Memory:

image

CPU:

image
pwdel commented 2 weeks ago

Just a quick update after having updated again, working on BrierFoxForecast, with SSL.

Image