Using the provided Docker image I noticed some issues with the setup that caused huge performance issues.
For the use case where I work at the status page may be hit by 3,000 - 4,000 people at once as a first load for them.
To see how this setup of cachet would handle this I ran a load test of 1,000 users per second for 20 seconds. It buckled under the first 1,000 with the majority ~65% being 500, ~5% being 502 and rest your usual 200.
Ran test on AWS EC2 instance t2.medium, 2 vCPU 4GB RAM and CPUs maxed while RAM was only about 40% usage.
After doing some digging into the dockerfile, compose file, and nginx config I found two significant things.
1.) The cache isn't used at often, at all?
Looking at the nginx setup there seems to be a header that indicates if the request was served a response from the cache.
I'm not getting this at all and looking at the incoming requests via streaming docker logs gives more evidence the cache isn't being used at all.
2.) *.js files are not being gzipped
vendor.js ~ 542kb
all.js ~ 919kb
These two files are sent over the wire uncompressed while the CSS is compressed through gzip. Looking at this through Chrome dev tools. It's about 80% of the total size of the total resources loaded for the page.
To get around these two issues to avoid scaling issues. I've gone ahead and used Chrome headless through puppeteer to run cachet locally, query localhost, scrap the generated html and then upload it to S3 to use in Cloudfront. This happens every minute from a cronjob. This works pretty well as a hack to avoid our status page going down as well.
Wrote this all out to see if I'm missing something? 🤣 Is there a way to serve up a static page for main status page without invoking any PHP without resorting to hack I described above?
Thanks for the project, it simplifies a-lot of other issues that the workaround I implemented above was well worth it!
Using the provided Docker image I noticed some issues with the setup that caused huge performance issues.
For the use case where I work at the status page may be hit by 3,000 - 4,000 people at once as a first load for them.
To see how this setup of cachet would handle this I ran a load test of 1,000 users per second for 20 seconds. It buckled under the first 1,000 with the majority ~65% being 500, ~5% being 502 and rest your usual 200.
Ran test on AWS EC2 instance t2.medium, 2 vCPU 4GB RAM and CPUs maxed while RAM was only about 40% usage.
After doing some digging into the dockerfile, compose file, and nginx config I found two significant things.
1.) The cache isn't used at often, at all?
Looking at the nginx setup there seems to be a header that indicates if the request was served a response from the cache.
fastcgi_cache_path /usr/share/nginx/cache/fcgi levels=1:2 keys_zone=microcache:10m max_size=1024m inactive=1h;add_header X-Cache $upstream_cache_status;I'm not getting this at all and looking at the incoming requests via streaming docker logs gives more evidence the cache isn't being used at all.
2.) *.js files are not being gzipped
vendor.js~ 542kball.js~ 919kbThese two files are sent over the wire uncompressed while the CSS is compressed through gzip. Looking at this through Chrome dev tools. It's about 80% of the total size of the total resources loaded for the page.
To get around these two issues to avoid scaling issues. I've gone ahead and used Chrome headless through puppeteer to run cachet locally, query localhost, scrap the generated html and then upload it to S3 to use in Cloudfront. This happens every minute from a cronjob. This works pretty well as a hack to avoid our status page going down as well.
Wrote this all out to see if I'm missing something? 🤣 Is there a way to serve up a static page for main status page without invoking any PHP without resorting to hack I described above?
Thanks for the project, it simplifies a-lot of other issues that the workaround I implemented above was well worth it!