The error seen by sender in Prometheus logs is like this:
level=warn ts=2019-07-22T09:25:16.899437956Z caller=queue_manager.go:530 component=remote queue=0:https://cloud.weave.works./api/prom/push msg="Error sending samples to remote storage" count=1000 err="server returned HTTP status 400 Bad Request: label value too long: \"{\\\"apiVersion\\\":\\\"extensions/v1beta1\\\",\\\"kind\\\":\\\"DaemonSet\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"flux.weave.works/sync-checksum\\\":\\\"0b6f1382edb3f01d6a64cd775b46008b0f61003e\\\"},\\\"labels\\\":{\\\"flux.weave.works/sync-gc-mark\\\":\" metric \"kub"
level=warn ts=2019-07-22T09:25:32.00067315Z caller=queue_manager.go:530 component=remote queue=0:https://cloud.weave.works./api/prom/push msg="Error sending samples to remote storage" count=1000 err="server returned HTTP status 400 Bad Request: label value too long: \"{\\\"apiVersion\\\":\\\"extensions/v1beta1\\\",\\\"kind\\\":\\\"Deployment\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"flux.weave.works/automated\\\":\\\"true\\\",\\\"flux.weave.works/automated_msg\\\":\\\"test\\\",\\\"flux.weave.works/automated_user\\\":\\\"Maarten\" metric \""
It looks like something is truncating the message just as it gets to the metric=... part. Maybe put that before the value?
And it would also be useful to state the label name.
The error seen by sender in Prometheus logs is like this:
It looks like something is truncating the message just as it gets to the
metric=...
part. Maybe put that before the value? And it would also be useful to state the label name.