salesforce / Argus

Time series monitoring and alerting platform.
BSD 3-Clause "New" or "Revised" License
505 stars 145 forks source link

Insufficient error message when pushing metrics to OpenTSDB fails #553

Open axdotl opened 7 years ago

axdotl commented 7 years ago

Scenario: Trying to put ~20k metrics in a single HTTP request to /argus/collection/metrics fails with misleading error message.

When pushing 20k metrics via /argus/collection/metrics, the request fails with a 500. Following insufficient log message occurs in log of the metric-client:

[ ARGUS | *NULLSESSION* | *NULLUSER* | *NULLTXID* | 2017-05-05 14:09:23.658 | ccommitclient-1 |  INFO ] Error occured while committing metrics. Reason com.salesforce.dva.argus.system.SystemException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: java.io.StringReader@385564dd; line: 1, column: 2]

The root cause for this error is that chunked requests to OpenTSDB are not allowed per default. So the REST call to openTSDBs /api/put returns 400 and some HTML content (which contains the actual error message 'Chunked request not supported.')

rajsarkapally commented 6 years ago

We opened a bug for this..