census-ecosystem / opencensus-go-exporter-stackdriver

OpenCensus Go exporter for Stackdriver Monitoring and Trace
Apache License 2.0
67 stars 79 forks source link

skip exporting points with no value due to prometheus staleness markers #303

Open kwiesmueller opened 2 years ago

kwiesmueller commented 2 years ago

This fixes cases where a staleness marker causes errors like:

rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: Field timeSeries[94].points[0].value had an invalid value: A point has an unrecognized value type.; Field timeSeries[95].points[0].value had an invalid value: A point has an unrecognized value type.\nerror details: name = Unknown  desc = total_point_count:155 success_point_count:153 errors:{status:{code:3} point_count:2}

By properly dropping any point that is a staleness marker instead of exporting them without a value.

The fixed issue was introduced with https://github.com/census-ecosystem/opencensus-go-exporter-stackdriver/pull/300. The bug should not cause any data loss, it just causes error logs and an increase in exporter error metrics.

kwiesmueller commented 2 years ago

/assign @dashpole

kwiesmueller commented 2 years ago

Actually, my test just failed as we're now sending timeseries with no points which is also not okay :-/ Needs more work.

kwiesmueller commented 2 years ago

Alright, works now in our collector. PTAL.

dashpole commented 2 years ago

/gcbrun