Abnormally high Upstash redis reads per second triggered by HTTP health checker of AWS EC2 load balancer.

[waffo-js]

Describe the bug

I have morphic deployed on both Vercel and locally on AWS. The AWS version was deployed using docker-compose, with latest commits around one week later than the Vercel one.

And I got warning from Upstash that I've exceeded the free tier on the same day of AWS deployment, I added my credit card and found today that morphic has read 31GB of data with 1.7M reads. I shutdown the container and the reads went down to zero instantly.

The patter was highly regular and I suspected the trigger was the health check on the HTTP port, done by AWS EC2 load balancer. I turned down the frequency and the load went down as well.

I guess when everything HTTP health check reads the "/" path, the whole history is loaded and triggered many reads on Upstash.

Apart from the issue, I would highly suggest adding a local Redis option as Upstash doesn't cover global regions such as Singapore and Hong Kong.

Screenshots

[arsaboo]

Another vote for local redis #110

[arsaboo]

@miurla also, please note that it makes the app extremely slow. I have not used morphic a lot, but still it takes forever to get started. In the logs, it appears that loading history takes a LONG time:

You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
https://nextjs.org/telemetry
 ✓ Ready in 1041ms
 ○ Compiling /search/[id] ...
 ✓ Compiled /search/[id] in 76.1s
history loaded
history loaded
 GET / 200 in 72931ms

[miurla]

@waffo-js

Thank you for the report.

The AWS EC2 Load Balancer is frequently accessing the "/" path, which may be causing a large number of data requests to Upstash. It is also possible that the cache is not working effectively. The reads are abnormally high for the amount of writing. Even morphic.sh, hosted on Vercel, is still at 12GB of bandwidth and 0.6M reads. I believe it is unlikely to exceed the free tier of Upstash with normal personal use. Since we don't support the AWS environment, we don't know the details.

Upstash doesn't cover global regions such as Singapore and Hong Kong.

If you choose the global type, you can select the Singapore region as well.

[waffo-js]

Thanks Yoshiki. AWS is just the hosting environment, and I used docker for the deployment, nothing special.

The problem is indeed the "/" health check, which I don't think cache works in this case. Is there any other path that I can use for the health check monitoring without loading all the history?

Will check the Upstash region settings.

[miurla]

Since the search path also loads history, it seems best to create a custom path for health checks.

[waffo-js]

Yup, should be easy to fix.