zio / zio-http

A next-generation Scala framework for building scalable, correct, and efficient HTTP clients and servers
https://zio.dev/zio-http
Apache License 2.0
799 stars 403 forks source link

OOM when receiving large streaming requests #3173

Open davidlar opened 1 month ago

davidlar commented 1 month ago

Describe the bug We have a service that proxies large streams to different backends. It's compiled with GraalVM native-image and is running in docker with a quite small heap ~80Mb. It has worked fine for a long time but when we updated to zio-http 3.0.0 from 3.0.0-RC9 it started to get OOM instantly. I did some investigation and the problem is the unbounded queue in AsyncBody.asStream that was introduced. It completely disables back pressure for incoming streams. So if the producer is faster than the consumer, the data will be buffered in the unbounded queue. This could lead to OOM.

To Reproduce Steps to reproduce the behaviour:

  1. Create a service with request streaming enabled and limit the heap. Make processing of incoming data a bit slow
  2. Create a test case that sends a request larger than the heap of the service.
  3. Watch OOM in the service

Expected behaviour It should work like it did before (but without blocking netty, of course)

davidlar commented 1 week ago

If enabling request streaming it only takes one large request to kill the server...

jdegoes commented 4 days ago

/bounty $250

algora-pbc[bot] commented 4 days ago

💎 $250 bounty • ZIO

Steps to solve:

  1. Start working: Comment /attempt #3173 with your implementation plan
  2. Submit work: Create a pull request including /claim #3173 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to zio/zio-http!

Add a bounty • Share on socials

Attempt Started (GMT+0) Solution
🟢 @davidlar Nov 9, 2024, 8:26:35 AM #3174
davidlar commented 4 days ago

/attempt #3173

algora-pbc[bot] commented 4 days ago

💡 @davidlar submitted a pull request that claims the bounty. You can visit your bounty board to reward.