search

captbaritone / raven-for-redux

A Raven middleware for Redux
295 stars 25 forks source link

When the redux state is really big, the request to sentry fails due to `Request too large` #42

Closed davidfurlong closed 6 years ago

davidfurlong commented 6 years ago

I'm not sure this is the responsibility of this library, however I'm running into a 413 Request Entity too large issue with our redux state being too big.

https://github.com/getsentry/raven-js/issues/339

davidfurlong commented 6 years ago

I suppose I could use the stateTransformer option..

davidfurlong commented 6 years ago

But also the conditions here: https://docs.sentry.io/learn/quotas/#attributes-limits seem quite limiting

beaugunderson commented 6 years ago

Noting that now raven-js and @sentry/browser default to fetch() with { keepalive: true } if fetch() is available. { keepalive: true } limits POST requests to a 64k body, so overriding fetchParams with { keepalive: false } can help here too (Sentry's documented limit is 100k, and onpremise users have effectively no limit).

More info at https://github.com/getsentry/sentry-javascript/issues/1464

ndbroadbent commented 5 years ago

I just noticed this happening while I was testing errors in development, and now I'm really worried that I've been missing a lot of error reports in production! I'm using redux-undo, so my state gets really large after only a few changes. Interestingly, compressing it with gzip turns 220kb of JSON into 5kb.

This should be a huge warning at the top of the README. I also think that raven-for-redux should be responsible for handling 413 responses, and it could retry the request without any state data. I can try to work on a PR for this. I'm also going to try to gzip the data on the client, and will see if Sentry accepts these requests.

I imagine that this issue has probably caused thousands of error reports to be silently dropped, so this is a huge problem that users should be aware of.

ndbroadbent commented 5 years ago

@captbaritone I just saw your workaround here: https://github.com/getsentry/sentry-javascript/issues/339#issuecomment-340134753 I think it would be awesome if this was part of raven-for-redux

EDIT: Ah there was a bug in your code (or sentry-javascript has changed). We need to call originalTransport.call(Raven, opts); now, otherwise this is not correct inside _makeRequest.

EDIT 2: Should also note that the maximum payload size has been increased to 200KB: https://docs.sentry.io/clients/javascript/usage/#raven-js-additional-context

beaugunderson commented 5 years ago

one note; that workaround is out of date now (but sadly i didn’t keep the newer version i ended up with)

Sent from my phone.

On Oct 13, 2018, at 01:51, Nathan Broadbent notifications@github.com wrote:

@captbaritone I just saw your workaround here: getsentry/sentry-javascript#339 (comment) I think it would be awesome if this was part of raven-for-redux

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

ndbroadbent commented 5 years ago

@beaugunderson Yeah I needed to make a few changes to get it working. I ended up with a solution and have opened a PR: #95

I've tested this in development and have written some tests that are all passing, so it seems to work pretty well. I'm going to deploy it to production now and see if there are any issues.

ndbroadbent commented 5 years ago

Sooooo I've actually come to the conclusion that raven-for-redux and redux-raven-middleware are not the best idea. Even with all my 413 error handling and redux-undo history removal, Sentry is still aggressively trimming the "additional data" and removing large chunks of my state. See:

I came up with a new approach that I'm going to try out:

Maybe a better approach would be to just submit the state to my own server, store the compressed JSON in an S3 bucket, and only send the URL to Sentry. (The compression will actually make a huge difference, because I have a lot of repeated data in my state, so it can compress from 220KB down to 5KB.) And then I can just copy that URL, and paste it into the developer console to load the state. I could even set an expiration rule on the S3 bucket, so that old files are deleted after 3 months. I think I'm going to create an open source library that can manage all of this, specifically for Ruby on Rails. Will call it something like raven-redux-rails.

Basically I'm going stop sending any Redux state to Sentry. I'm going to add some API endpoints to my own application, and store the compressed state in my own S3 bucket. Then I'll just send that URL to Sentry. The API specification should be really easy to implement if you're already using S3 and you have authentication for admin users. Let me know if you're interested in contributing a library for other languages/frameworks, like Django, Laravel, Spring, etc.

I've started a repo here, in case anyone wants to follow along: https://github.com/FormAPI/raven-redux-rails)