Send basic error info in case payload is too big and #40 Failsafe error is reported

[ncri]

We have many of them and rollbar doesn't give us any clue about what they are about. Attaching basic error info would be very useful in these cases even if most of the payload is ignored. At least we would need the error message and stacktrace and maybe basic environment info.

[brianr]

Thanks @ncri.

I'm guessing these may be infinite recursion issues (i.e. really really long stack traces). The way to fix this is probably to truncate long arrays, in addition to long strings. Relevant code is here

[jasonl]

Hi, we are seeing this issue as well (e.g. https://rollbar.com/JobReady/IQ/items/4).

Ours are not being caused by recursive stack traces (when we inspect the application logs they are pretty standard exceptions that are otherwise raised), but are instead arising from a API endpoint where we POST multi-megabyte JSON bodies. We have tried excluding the data through Rollbar.configuration.scrub_fields, but that seems to make no difference.

This is with Rollbar 1.4.4.

[jondeandres]

@jasonl your point is that configuration.scrub_fields doesn't work properly?

@jasonl or @ncri, have you been able to get the payload before the truncation process?

Just before https://github.com/rollbar/rollbar-gem/blob/master/lib/rollbar.rb#L632. Truncation code is in https://github.com/rollbar/rollbar-gem/blob/master/lib/rollbar/truncation.rb and https://github.com/rollbar/rollbar-gem/tree/master/lib/rollbar/truncation.

[jondeandres]

@ncri @jasonl

When payload cannot be truncated the gem logs a message:

log_error "[Rollbar] Payload too large to be sent: #{MultiJson.dump(payload)}"

Could you find those messages and send us the payload to support@rollbar.com? That'd be very useful to resolve your problem.

[grosser]

@jondeandres running into the same issue ... how about reporting the sizes of all fields in that case so users can see what the trouble is and preemptive backtrace truncation if it's >100/500 calls ?

Failsafe from rollbar-gem. Could not send payload due to it being too large after truncating attempts. Original size: 230223 Final size: 220669 

Getting this once in a while and I don't have enough log retention to find what is causing this :(

[sheerun]

I get this issue as well for big JSON POST payloads

[s2t2]

Hi, I'm seeing frequent occurrences of the following error:

Failsafe from rollbar-gem. Could not send payload due to it being too large after truncating attempts. Original size: 221390 Final size: 200469

Is there a recommended fix to stop these errors from occurring?

Thanks!

[grosser]

make a PR here that fixes it ... the get it merged and deployed :trollface:

[jondeandres]

@s2t2 we send the payload to the logger in this cases with this code:

logger.error("[Rollbar] Payload too large to be sent for UUID #{uuid}: #{Rollbar::JSON.dump(payload)}")

Maybe knowing how your payload is, we can try to add a new truncation strategy for you.

BTW, seems that your paylopad is 216kb. We increased the max payload size to 512kb in 2.13.0 version. You can upgrade the gem and check if this resolves the problem.

[cforcey]

@jondeandres Thanks so much for your increase to the payload -- we are upgrading and will let you know if the new limit helps. Our error occurred when we get an extra large JSON post to our API and our application times out processing it. We then catch the RackTimeout error and the json API seems to be included in the backtrace

Nov 07 10:00:35 pennsylvania-production app/web.1:  Rack::Timeout::RequestTimeoutException (Request waited 0ms, then ran for longer than 15000ms): 
Nov 07 10:00:41 pennsylvania-production app/worker_high.1:  [Rollbar] Payload too large to be sent: {"access_token":"87aa6634ad9e49b1b765a84d5e02731d","data":

...

{"filename":"/app/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb","lineno":592,"method":"async_exec"}],"exception":{"class":"Rack::Timeout::RequestTimeoutException","message":"Request waited 0ms, then ran for longer than 15000ms"},"extra":{}}},"uuid":"df91f96b-8a64-4e31-8a2a-8536aee5c353","request":{"params":{"bill":{"bill_id":862009,"change_hash":"00449ff786474d2be3f9f8082b681721","last_push":"2016-04-17 02:02:59","reasons":{"2":"StatusChange","4":"Complete","6":"Description","8":"CommReport","12":"HistoryAdd","14":"HistoryRevised","15":"HistoryMajor","16":"HistoryMinor","23":"Vote","25":"Progress"},"session":{"session_id":1149,"session_name":"121st General Assembly","session_title":"121st General Assembly"},"url":"https://legiscan.com/SC/bill/H5001/2015","state_link":"http://www.scstatehouse.gov/billsearch.php?billnumbers=5001&session=121&summary=B","completed":1,"status":4,"status_date":"2016-07-13","progress":[{"date":"2016-03-09","event":1},{"date":"2016-03-23","event":2},{"date":"2016-03-23","event":9},{"date":"2016-06-01","event":3},{"date":"2016-06-08","event":5},{"date":"2016-06-15","event":7},{"date":"2016-06-15","event":7},{"date":"2016-07-13","event":8}],"state":"SC","state_id":40,"bill_number":"H5001","bill_type":"B","body":"H","body_id":85,"current_body":"S","current_body_id":86,"title":"Appropriations Bill, 2016-2017","description":"An Act To Make Appropriations And To Provide Revenues To Meet The Ordinary Expenses Of State Government For The Fiscal Year Beginning July 1, 2016, To Regulate The Expenditure Of Such Funds, And To Further Provide For The Operation Of State Government During This Fiscal Year And For Other Purposes","committee":[],"history":[{"date":"2016-03-09","action":"Introduced, read first time, placed on calendar without reference","chamber":"H","chamber_id":85,"importance":1},{"date":"2016-03-10","action":"Debate adjourned until Tues., 3-15-16","chamber":"H","chamber_id":85,"importance":0},{"date":"2016-03-15","action":"Debate adjourned until Wed., 3-16-16","chamber":"H","chamber_id":85,"importance":0},{"date":"2016-03-16","action":"Debate adjourned until Thur., 3-17-16","chamber":"H","chamber_id":85,"importance":0},{"date":"2016-03-17","action":"Special order, set for Monday, March 21 immediately after roll call","chamber":"H","chamber_id":85,"importance":0},{"date":"2016-03-21","action":"Amended","chamber":"H","chamber_id":85,"importance":0},{"date":"2016-03-21","action":"Debate interrupted","chamber":"H","chamber_id":85,"importance":0},{"date":"2016-03-22","action":"Amended","chamber":"H","chamber_id":85,"importance":0},{"date":"2016-03-22","action":"Read second time","chamber":"H","chamber_id":85,"importance":0},{"date":"2016-03-22","action":"Roll call Yeas-111 Nays-1","chamber":"H","chamber_id":85,"importance":0},{"date":"2016-03-23","action":"Read third time and sent to Senate","chamber":"H","chamber_id":85,"importance":1},{"date":"2016-03-23","action":"Roll call Yeas-111 Nays-1","chamber":"H","chamber_id":85,"importance":0},{"date":"2016-03-23","action":"Introduced and read first time","chamber":"S","chamber_id":86,"importance":0},{"date":"2016-03-23","action":"Referred to Committee on Finance","chamber":"S","chamber_id":86,"importance":1},{"date":"2016-04-27","action":"Committee report: Favorable with amendment Finance","chamber":"S","chamber_id":86,"importance":0},{"date":"2016-04-28","action":"Committee Amendment Adopted","chamber":"S","chamber_id":86,"importance":0},{"date":"2016-04-28","action":"Debate interrupted","chamber":"S","chamber_id":86,"importance":0},{"date":"2016-05-03","action":"Amended","chamber":"S","chamber_id":86,"importance":0},{"date":"2016-05-03","action":"Read second time","chamber":"S","chamber_id":86,"importance":0},{"date":"2016-05-03","action":"Roll call Ayes-36 Nays-5","chamber":"S","chamber_id":86,"importance":0},{"date":"2016-05-04","action":"Amended","chamber":"S","chamber_id":86,"importance":0},{"date":"2016-05-04","action":"Read third time and returned to House with amendments","chamber":"S","chamber_id":86,"importance":0}, 
Nov 07 10:07:49 pennsylvania-production app/web.1:    vendor/bundle/ruby/2.2.0/gems/rack-timeout-0.3.2/lib/rack/timeout/core.rb:125:in `block in call' 
Nov 07 10:07:49 pennsylvania-production app/web.1:    vendor/bundle/ruby/2.2.0/gems/rack-timeout-0.3.2/lib/rack/timeout/support/timeout.rb:19:in `call' 
Nov 07 10:07:49 pennsylvania-production app/web.1:    vendor/bundle/ruby/2.2.0/gems/rack-timeout-0.3.2/lib/rack/timeout/support/timeout.rb:19:in `timeout' 
Nov 07 10:07:49 pennsylvania-production app/web.1:    vendor/bundle/ruby/2.2.0/gems/rack-timeout-0.3.2/lib/rack/timeout/core.rb:124:in `call'

Thanks so much for this fast response! Charlie

[coryvirok]

@cforcey are you still experiencing this issue?

[rivkahstandig3636]

Hi there, I’m closing out all issues opened before 2018 that haven’t had any activity on them since the start of this year. If this is still an issue for you, please comment here and we can reopen this. Thanks!