search

GoogleCloudPlatform / appengine-java-vm-runtime

Apache License 2.0
67 stars 34 forks source link

Error from HTTP-over-RPC proxy: Truncated body #163

Open erugeri opened 8 years ago

erugeri commented 8 years ago

We noticed that some HTTP responses are being truncated and the following message is added at the end of the truncated body "Error from HTTP-over-RPC proxy: Truncated body".

It seems it happens only when a server is running for a few days (in our case 3 days, but maybe the problem started earlier). Our servers have quite a lot of load (3 n1-standard-4 at 625 qps at peek) but are very stable.

The query is truncated at around 112 - 130ko. We saw it happen for mp4 downloads or big html responses. We use last Jersey version: 2.22.1, but I don't think it's linked to our code.

Do you have any idea? Thanks!

erugeri commented 8 years ago

FYI, after a server reboot (stop/start), the issue is fixed.

erugeri commented 8 years ago

Would you have an advice to setup an auto reboot for a managed vm instances after 48h of uptime for example? Or maybe you already fixed it and the fix will hit production soon? My only idea for now is to have another server automatically deploying a new version for each module and automatically setting it default. But I don't think it is elegant.

Indeed, this issue is really problematic as all http responses with a big body are truncated leading to multiple errors for users. I checked, it also happens for API calls returning JSON data so I guess the issue is there for all HTTP requests.

Thanks.

erugeri commented 8 years ago

We saw the issue on another app without having to wait for 3 days. So it was an important issue as all responses with content > 32ko had "Error from HTTP-over-RPC proxy: Truncated body" appended to the body making JSON communications fail.

https://code.google.com/p/googleappengine/issues/detail?id=12790 -> adding the filter ( https://gist.github.com/prebenl/7fee765211bfed2cc667 ) fixed the problem in case someone is interested.

Not sure yet if the original problem (after 3 days) is still there though.

speedplane commented 8 years ago

I've seen this before, but I'm seeing this a lot today.

wangzilong commented 8 years ago

I deploy my project today but sometimes the body of response is added at the end of the truncated body "Error from HTTP-over-RPC proxy: Truncated body".

e.g. {"id":....,"name":"hihihi"}Error from HTTP-over-RPC proxy: Truncated body

Is there any idea to resolve it?

ludoch commented 8 years ago

I've see an internal bug for the serving path that got fixed yesterday. It should appear soon in prod. Let us know if you still see it in a few days.