[mimir-distributed] Frequently Gaps Inside Mimir Metrics

[ghost]

Hello,

we got an running instance of mimir-distributed inside our kubernetes cluster. But there are still some problems. It seems to me that the samples, which are out of order, can not get ingested. I think prometheus tries to remote write old samples from its WAL. What do i need to configure in mimir to allow prometheus to send these old samples? How do i archive a gapless traffic of samples? If i take a look at mimir trough grafana i got some empty gaps (time ranges with no data) in the graph for every metric. What am i doing wrong here?

I added our config. Also logs of the mimir-distributed-distributor, mimir-distributed-ingester and prometheus-rancher-monitoring-prometheus pods.

Best Regards

[ghost]

Active Alerts:

• warning | MimirRequestLatency | mimir-distributed-distributor-headless api_v1_push is experiencing 9.88s 99th percentile latency.
• warning | MimirRequestLatency | mimir-distributed-gossip-ring api_v1_push is experiencing 9.83s 99th percentile latency.
• warning | MimirRequestLatency | mimir-distributed-distributor api_v1_push is experiencing 9.86s 99th percentile latency.
• warning | MimirStoreGatewayNoSyncedTenants | Mimir store-gateway mimir-distributed-store-gateway-0 in /cattle-monitoring-system is not syncing any blocks for any tenant.
• warning | PrometheusRemoteWriteDesiredShards

[ghost]

Config:

mimir:
  config: >
    {{- if not .Values.enterprise.enabled -}}

    multitenancy_enabled: false

    {{- end }}

    limits:
      compactor_blocks_retention_period: 604800s
      ingestion_rate: 60000
      max_global_series_per_metric: 2000000
      max_global_series_per_user: 2000000
      max_label_names_per_series: 60

    distributor:
      instance_limits:
        max_ingestion_rate: 0
        max_inflight_push_requests: 0
      remote_timeout: 30s

    ...

[ghost]

mimir-distributed-distributor log:

Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227192795Z caller=logging.go:72 traceID=1736da2b0734600d msg="POST /api/v1/push (500) 1.722399471s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15250; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227707732Z caller=logging.go:72 traceID=6e63569a6618e2db msg="POST /api/v1/push (500) 1.913782544s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16030; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227194307Z caller=logging.go:72 traceID=06c2a40f7d29110b msg="POST /api/v1/push (500) 486.407113ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14918; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227734096Z caller=logging.go:72 traceID=4576d9fdd7287b4b msg="POST /api/v1/push (500) 1.119823902s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16185; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227736782Z caller=logging.go:72 traceID=47da02c9af0d1a91 msg="POST /api/v1/push (500) 1.019907518s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14810; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227791683Z caller=logging.go:72 traceID=00807a367d15eb3f msg="POST /api/v1/push (500) 416.256225ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13813; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227814911Z caller=logging.go:72 traceID=1fa857fb9629012c msg="POST /api/v1/push (500) 510.711781ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14655; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227424118Z caller=logging.go:72 traceID=0bfd3f1a63b22e98 msg="POST /api/v1/push (500) 2.01935168s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15572; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.22782542Z caller=logging.go:72 traceID=0dfdd1bf9c261c4f msg="POST /api/v1/push (500) 2.104112757s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15516; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227498668Z caller=logging.go:72 traceID=27a84d3a1f41e2de msg="POST /api/v1/push (500) 2.216045364s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15175; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227448365Z caller=logging.go:72 traceID=5143feb6c199d63a msg="POST /api/v1/push (500) 1.592792258s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13028; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227747108Z caller=logging.go:72 traceID=47ab797531315361 msg="POST /api/v1/push (500) 2.314307316s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13239; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227876713Z caller=logging.go:72 traceID=099b9513c7b0d731 msg="POST /api/v1/push (500) 318.832623ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14863; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227768142Z caller=logging.go:72 traceID=74c404f3a2e04a89 msg="POST /api/v1/push (500) 1.312822963s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15778; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227760562Z caller=logging.go:72 traceID=1816442083514f24 msg="POST /api/v1/push (500) 517.110517ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 12800; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227770761Z caller=logging.go:72 traceID=1c04b551a4d83ce8 msg="POST /api/v1/push (500) 122.295527ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13004; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.22777875Z caller=logging.go:72 traceID=72cfe829ef2a0e6f msg="POST /api/v1/push (500) 1.212204093s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 12082; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227472749Z caller=logging.go:72 traceID=081936b6df936bab msg="POST /api/v1/push (500) 696.227745ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14713; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228180425Z caller=logging.go:72 traceID=5518a048bdaa5c5b msg="POST /api/v1/push (500) 1.690873069s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13039; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228243271Z caller=logging.go:72 traceID=7c46d278cadc9c00 msg="POST /api/v1/push (500) 1.613758561s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13927; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228248336Z caller=logging.go:72 traceID=3a18e7eb8f43f969 msg="POST /api/v1/push (500) 3.091865069s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16164; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228419214Z caller=logging.go:72 traceID=29fc982db856b100 msg="POST /api/v1/push (500) 1.017792748s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 12572; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228430603Z caller=logging.go:72 traceID=2673eab65d91e83b msg="POST /api/v1/push (500) 1.121916731s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15985; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228448025Z caller=logging.go:72 traceID=72e582882f8b065e msg="POST /api/v1/push (500) 2.722082583s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 17358; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228496778Z caller=logging.go:72 traceID=1c52bcdb21cf5efa msg="POST /api/v1/push (500) 1.909129961s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 12981; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228551919Z caller=logging.go:72 traceID=6c6ced0953f83e79 msg="POST /api/v1/push (500) 2.020535714s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 11016; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228598416Z caller=logging.go:72 traceID=757194e641fa476e msg="POST /api/v1/push (500) 3.321319526s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16142; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.22871595Z caller=logging.go:72 traceID=4757ec01b60357c4 msg="POST /api/v1/push (500) 2.516569804s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14389; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228791793Z caller=logging.go:72 traceID=701dd0f19ce9f80d msg="POST /api/v1/push (500) 86.595219ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16778; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228874816Z caller=logging.go:72 traceID=428fc95d7b0bb204 msg="POST /api/v1/push (500) 2.514256872s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14006; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.229037391Z caller=logging.go:72 traceID=469e54e69e538a94 msg="POST /api/v1/push (500) 520.816824ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14051; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.229044499Z caller=logging.go:72 traceID=1b7dbca99112c312 msg="POST /api/v1/push (500) 518.300411ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 12953; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.229054522Z caller=logging.go:72 traceID=6eec49cd41925e6a msg="POST /api/v1/push (500) 613.843239ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14773; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.22923861Z caller=logging.go:72 traceID=1857fecda2a465c9 msg="POST /api/v1/push (500) 3.419017489s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14995; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.229349614Z caller=logging.go:72 traceID=21b65e111d5cfb75 msg="POST /api/v1/push (500) 3.015330856s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14005; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.229356677Z caller=logging.go:72 traceID=6e33820aa370fa2f msg="POST /api/v1/push (500) 720.936237ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14085; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.233139646Z caller=logging.go:72 traceID=6dc90806423655a2 msg="POST /api/v1/push (500) 19.089005ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15493; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.241303811Z caller=logging.go:72 traceID=497a23a70122d437 msg="POST /api/v1/push (500) 27.302655ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16582; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "

[ghost]

mimir-distributed-ingester log:

Mon, 20 Jun 2022 13:32:39 | level=warn ts=2022-06-20T11:32:39.189854883Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=726.413µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:07:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"rbac.authorization.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.9\", resource=\"roles\", scope=\"namespace\", service=\"kubernetes\", verb=\"POST\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:33:54 | level=warn ts=2022-06-20T11:33:54.711395544Z caller=grpc_logging.go:38 duration=696.55µs method=/cortex.Ingester/Push err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"apiregistration.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"apiservices\", scope=\"cluster\", service=\"kubernetes\", verb=\"DELETE\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:33:58 | level=warn ts=2022-06-20T11:33:58.51486206Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=670.173µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"monitoring.coreos.com\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.5\", resource=\"alertmanagers\", scope=\"namespace\", service=\"kubernetes\", verb=\"PATCH\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:12 | level=warn ts=2022-06-20T11:34:12.589178909Z caller=grpc_logging.go:38 duration=510.352µs method=/cortex.Ingester/Push err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"admissionregistration.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.5\", resource=\"mutatingwebhookconfigurations\", scope=\"cluster\", service=\"kubernetes\", verb=\"GET\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:15 | level=warn ts=2022-06-20T11:34:15.551402567Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=748.994µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"catalog.cattle.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.5\", resource=\"operations\", scope=\"namespace\", service=\"kubernetes\", verb=\"DELETE\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:15 | level=warn ts=2022-06-20T11:34:15.831319781Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=805.451µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:32.121Z, series={__name__=\"istio_request_bytes_bucket\", connection_security_policy=\"unknown\", container=\"istio-proxy\", destination_app=\"web-app\", destination_canonical_revision=\"1938\", destination_canonical_service=\"web-app\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"web-app.testing.svc.cluster.local\", destination_service_name=\"web-app\", destination_service_namespace=\"testing\", destination_version=\"1938\", destination_workload=\"web-app\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.194:15090\", job=\"istio-ingressgateway\", le=\"600000\", namespace=\"istio-system\", pod=\"istio-ingressgateway-c594d897-g2h7q\", pod_name=\"istio-ingressgateway-c594d897-g2h7q\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"source\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"istio-ingressgateway\", source_app=\"istio-ingressgateway\", source_canonical_revision=\"latest\", source_canonical_service=\"istio-ingressgateway\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account\", source_version=\"unknown\", source_workload=\"istio-ingressgateway\", source_workload_namespace=\"istio-system\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:20 | level=warn ts=2022-06-20T11:34:20.166798909Z caller=grpc_logging.go:38 duration=615.585µs method=/cortex.Ingester/Push err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:13.201Z, series={__name__=\"istio_request_bytes_bucket\", connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"elasticsearch-es-default\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"elasticsearch-es-http.testing.svc.cluster.local\", destination_service_name=\"elasticsearch-es-http\", destination_service_namespace=\"testing\", destination_version=\"unknown\", destination_workload=\"elasticsearch-es-default\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.14.43:15090\", job=\"elasticsearch-es-transport\", le=\"2500\", namespace=\"testing\", pod=\"elasticsearch-es-default-0\", pod_name=\"elasticsearch-es-default-0\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"elasticsearch-es-transport\", source_app=\"flow-executor\", source_canonical_revision=\"1974\", source_canonical_service=\"flow-executor\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/testing/sa/default\", source_version=\"1974\", source_workload=\"flow-executor\", source_workload_namespace=\"testing\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:23 | level=warn ts=2022-06-20T11:34:23.211139804Z caller=grpc_logging.go:38 duration=521.226µs method=/cortex.Ingester/Push err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:23.611Z, series={__name__=\"istio_response_bytes_bucket\", connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"elasticsearch-es-default\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"elasticsearch-es-http.testing.svc.cluster.local\", destination_service_name=\"elasticsearch-es-http\", destination_service_namespace=\"testing\", destination_version=\"unknown\", destination_workload=\"elasticsearch-es-default\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.9.196:15090\", job=\"elasticsearch-es-transport\", le=\"300000\", namespace=\"testing\", pod=\"elasticsearch-es-default-1\", pod_name=\"elasticsearch-es-default-1\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"elasticsearch-es-transport\", source_app=\"flow-executor\", source_canonical_revision=\"1945\", source_canonical_service=\"flow-executor\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/testing/sa/default\", source_version=\"1945\", source_workload=\"flow-executor\", source_workload_namespace=\"testing\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:25 | level=warn ts=2022-06-20T11:34:25.259276937Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=439.795µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:20.335Z, series={__name__=\"envoy_server_live\", container=\"istio-proxy\", endpoint=\"15090\", instance=\"10.42.9.41:15090\", job=\"audit\", namespace=\"testing\", pod=\"audit-85d7fb579d-gk588\", pod_name=\"audit-85d7fb579d-gk588\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", service=\"audit\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:26 | level=warn ts=2022-06-20T11:34:26.288849589Z caller=grpc_logging.go:38 duration=514.87µs method=/cortex.Ingester/Push err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:47.121Z, series={__name__=\"istio_request_duration_milliseconds_bucket\", connection_security_policy=\"unknown\", container=\"istio-proxy\", destination_app=\"flow\", destination_canonical_revision=\"1911\", destination_canonical_service=\"flow\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"flow.testing.svc.cluster.local\", destination_service_name=\"flow\", destination_service_namespace=\"testing\", destination_version=\"1911\", destination_workload=\"flow\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.194:15090\", job=\"istio-ingressgateway\", le=\"0.5\", namespace=\"istio-system\", pod=\"istio-ingressgateway-c594d897-g2h7q\", pod_name=\"istio-ingressgateway-c594d897-g2h7q\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"source\", request_protocol=\"http\", response_code=\"304\", response_flags=\"-\", service=\"istio-ingressgateway\", source_app=\"istio-ingressgateway\", source_canonical_revision=\"latest\", source_canonical_service=\"istio-ingressgateway\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account\", source_version=\"unknown\", source_workload=\"istio-ingressgateway\", source_workload_namespace=\"istio-system\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:41 | level=warn ts=2022-06-20T11:34:41.442058796Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=573.971µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"pods\", verb=\"read\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:22 | level=warn ts=2022-06-20T11:35:22.792209788Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=495.438µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"policy\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"podsecuritypolicies\", scope=\"cluster\", service=\"kubernetes\", verb=\"POST\", version=\"v1beta1\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:39 | level=warn ts=2022-06-20T11:35:39.879543641Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=953.927µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"storage.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"volumeattachments\", scope=\"cluster\", service=\"kubernetes\", verb=\"PATCH\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:46 | level=warn ts=2022-06-20T11:35:46.674285571Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=491.576µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"networking.istio.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.9\", resource=\"virtualservices\", scope=\"namespace\", service=\"kubernetes\", verb=\"GET\", version=\"v1beta1\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:48 | level=warn ts=2022-06-20T11:35:48.242767891Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=577.645µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:49.922Z, series={__name__=\"envoy_listener_manager_lds_update_duration_bucket\", container=\"istio-proxy\", endpoint=\"15090\", instance=\"10.42.9.231:15090\", job=\"secret\", le=\"250\", namespace=\"cloud\", pod=\"secret-6b844b8985-p52zg\", pod_name=\"secret-6b844b8985-p52zg\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", service=\"secret\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:55 | level=warn ts=2022-06-20T11:35:55.173285815Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=691.691µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"apps\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"replicasets\", scope=\"namespace\", service=\"kubernetes\", verb=\"PUT\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:57 | level=warn ts=2022-06-20T11:35:57.575252365Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=915.975µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.9\", resource=\"services\", scope=\"namespace\", service=\"kubernetes\", verb=\"PATCH\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:01 | level=warn ts=2022-06-20T11:36:01.000311337Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=720.084µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"rbac.authorization.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.5\", resource=\"clusterroles\", scope=\"cluster\", service=\"kubernetes\", verb=\"DELETE\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:08 | level=warn ts=2022-06-20T11:36:08.380644566Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=596.909µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:47.121Z, series={__name__=\"istio_request_duration_milliseconds_count\", connection_security_policy=\"unknown\", container=\"istio-proxy\", destination_app=\"timeseries\", destination_canonical_revision=\"1973\", destination_canonical_service=\"timeseries\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"timeseries.testing.svc.cluster.local\", destination_service_name=\"timeseries\", destination_service_namespace=\"testing\", destination_version=\"1973\", destination_workload=\"timeseries\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.194:15090\", job=\"istio-ingressgateway\", namespace=\"istio-system\", pod=\"istio-ingressgateway-c594d897-g2h7q\", pod_name=\"istio-ingressgateway-c594d897-g2h7q\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"source\", request_protocol=\"http\", response_code=\"403\", response_flags=\"-\", service=\"istio-ingressgateway\", source_app=\"istio-ingressgateway\", source_canonical_revision=\"latest\", source_canonical_service=\"istio-ingressgateway\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account\", source_version=\"unknown\", source_workload=\"istio-ingressgateway\", source_workload_namespace=\"istio-system\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:15 | level=warn ts=2022-06-20T11:36:15.88950191Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=520.855µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:48.703Z, series={__name__=\"istio_request_bytes_bucket\", connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"elasticsearch-es-transform\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/sealed-services/sa/default\", destination_service=\"elasticsearch-es-http.sealed-services.svc.cluster.local\", destination_service_name=\"elasticsearch-es-http\", destination_service_namespace=\"sealed-services\", destination_version=\"unknown\", destination_workload=\"elasticsearch-es-transform\", destination_workload_namespace=\"sealed-services\", endpoint=\"15090\", instance=\"10.42.12.169:15090\", job=\"elasticsearch-es-transform\", le=\"50\", namespace=\"sealed-services\", pod=\"elasticsearch-es-transform-0\", pod_name=\"elasticsearch-es-transform-0\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"elasticsearch-es-transform\", source_app=\"unknown\", source_canonical_revision=\"latest\", source_canonical_service=\"elasticsearch-kb\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/sealed-services/sa/default\", source_version=\"unknown\", source_workload=\"elasticsearch-kb\", source_workload_namespace=\"sealed-services\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:25 | level=warn ts=2022-06-20T11:36:25.27399867Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=469.65µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:47.121Z, series={__name__=\"istio_response_bytes_bucket\", connection_security_policy=\"unknown\", container=\"istio-proxy\", destination_app=\"asset\", destination_canonical_revision=\"1958\", destination_canonical_service=\"asset\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"asset.testing.svc.cluster.local\", destination_service_name=\"asset\", destination_service_namespace=\"testing\", destination_version=\"1958\", destination_workload=\"asset\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.194:15090\", job=\"istio-ingressgateway\", le=\"10000\", namespace=\"istio-system\", pod=\"istio-ingressgateway-c594d897-g2h7q\", pod_name=\"istio-ingressgateway-c594d897-g2h7q\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"source\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"istio-ingressgateway\", source_app=\"istio-ingressgateway\", source_canonical_revision=\"latest\", source_canonical_service=\"istio-ingressgateway\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account\", source_version=\"unknown\", source_workload=\"istio-ingressgateway\", source_workload_namespace=\"istio-system\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:33 | level=warn ts=2022-06-20T11:36:33.279107798Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=650.877µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:45.424Z, series={__name__=\"istio_request_bytes_bucket\", connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"elasticsearch-es-transform\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"elasticsearch-es-http.testing.svc.cluster.local\", destination_service_name=\"elasticsearch-es-http\", destination_service_namespace=\"testing\", destination_version=\"unknown\", destination_workload=\"elasticsearch-es-transform\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.94:15090\", job=\"elasticsearch-es-http\", le=\"5\", namespace=\"testing\", pod=\"elasticsearch-es-transform-0\", pod_name=\"elasticsearch-es-transform-0\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"409\", response_flags=\"-\", service=\"elasticsearch-es-http\", source_app=\"unknown\", source_canonical_revision=\"latest\", source_canonical_service=\"elasticsearch-kb\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/testing/sa/default\", source_version=\"unknown\", source_workload=\"elasticsearch-kb\", source_workload_namespace=\"testing\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:39 | level=warn ts=2022-06-20T11:36:39.820365459Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=593.632µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"validatingwebhookconfigurations\", verb=\"read\"}" msg=gRPC
Mon, 20 Jun 2022 13:37:37 | level=warn ts=2022-06-20T11:37:37.549507926Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=568.603µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:11:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"kibana.k8s.elastic.co\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"kibanas\", scope=\"namespace\", service=\"kubernetes\", verb=\"PATCH\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:37:47 | level=warn ts=2022-06-20T11:37:47.819143996Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=477.755µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"rbac.authorization.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"clusterrolebindings\", scope=\"cluster\", service=\"kubernetes\", verb=\"GET\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:37:48 | level=warn ts=2022-06-20T11:37:48.48279175Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=762.907µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"kibana.k8s.elastic.co\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"kibanas\", scope=\"namespace\", service=\"kubernetes\", verb=\"UPDATE\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:37:51 | level=warn ts=2022-06-20T11:37:51.727862428Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=610.987µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:47.121Z, series={__name__=\"istio_requests_total\", connection_security_policy=\"unknown\", container=\"istio-proxy\", destination_app=\"alert\", destination_canonical_revision=\"1963\", destination_canonical_service=\"alert\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"alert.testing.svc.cluster.local\", destination_service_name=\"alert\", destination_service_namespace=\"testing\", destination_version=\"1963\", destination_workload=\"alert\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.194:15090\", job=\"istio-ingressgateway\", namespace=\"istio-system\", pod=\"istio-ingressgateway-c594d897-g2h7q\", pod_name=\"istio-ingressgateway-c594d897-g2h7q\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"source\", request_protocol=\"http\", response_code=\"403\", response_flags=\"-\", service=\"istio-ingressgateway\", source_app=\"istio-ingressgateway\", source_canonical_revision=\"latest\", source_canonical_service=\"istio-ingressgateway\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account\", source_version=\"unknown\", source_workload=\"istio-ingressgateway\", source_workload_namespace=\"istio-system\"}" msg=gRPC
Mon, 20 Jun 2022 13:37:53 | level=warn ts=2022-06-20T11:37:53.276950969Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=586.305µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:11:27.026Z, series={__name__=\"istio_request_bytes_bucket\", connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"elasticsearch-es-transform\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"elasticsearch-es-http.testing.svc.cluster.local\", destination_service_name=\"elasticsearch-es-http\", destination_service_namespace=\"testing\", destination_version=\"unknown\", destination_workload=\"elasticsearch-es-transform\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.94:15090\", job=\"elasticsearch-es-internal-http\", le=\"2500\", namespace=\"testing\", pod=\"elasticsearch-es-transform-0\", pod_name=\"elasticsearch-es-transform-0\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"elasticsearch-es-internal-http\", source_app=\"asset\", source_canonical_revision=\"1945\", source_canonical_service=\"asset\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/testing/sa/default\", source_version=\"1945\", source_workload=\"asset\", source_workload_namespace=\"testing\"}" msg=gRPC

[ghost]

prometheus-rancher-monitoring-prometheus log:

level=info ts=2022-06-20T11:38:33.057Z caller=main.go:443 msg="Starting Prometheus" version="(version=2.28.1, branch=HEAD, revision=b0944590a1c9a6b35dc5a696869f75f422b107a1)"
--
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.057Z caller=main.go:448 build_context="(go=go1.16.5, user=root@2915dd495090, date=20210701-15:20:10)"
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.057Z caller=main.go:449 host_details="(Linux 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64 prometheus-rancher-monitoring-prometheus-0 (none))"
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.057Z caller=main.go:450 fd_limits="(soft=1048576, hard=1048576)"
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.057Z caller=main.go:451 vm_limits="(soft=unlimited, hard=unlimited)"
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.061Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.061Z caller=main.go:824 msg="Starting TSDB ..."
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.062Z caller=tls_config.go:227 component=web msg="TLS is disabled." http2=false
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.062Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655056800062 maxt=1655121600000 ulid=01G5F0837YAC495GQ50V4FTZTV
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.063Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655121600063 maxt=1655186400000 ulid=01G5GY1QX1Z92BN4GPHK997M1A
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.063Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655186400014 maxt=1655251200000 ulid=01G5JVVEYCDFFMVEF2BHW0245Y
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.064Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655251200053 maxt=1655272800000 ulid=01G5KGC7CMZBMZ3F9WB59XH5VX
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.064Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655272800065 maxt=1655294400000 ulid=01G5M4Z54HNS4WH3TEV58Y2DP0
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.065Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655294400027 maxt=1655316000000 ulid=01G5MSJ8XD0959V50JK46YPMEH
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.065Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655316000094 maxt=1655337600000 ulid=01G5NE5EX24V3EAPY8K397VS72
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.066Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655337600082 maxt=1655359200000 ulid=01G5P2RM6H86EG3S8WB62EARG6
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.066Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655359200029 maxt=1655380800000 ulid=01G5PQBVAD8PFDG3C63AG4ZVAK
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.067Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655380800018 maxt=1655402400000 ulid=01G5QBYWZ7DSMAJ33Z59S15PAY
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.067Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655402400075 maxt=1655424000000 ulid=01G5R0J3EECE6EXART79PHG65S
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.067Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655424000005 maxt=1655445600000 ulid=01G5RN5ERF86JA4JS0VGR40T0R
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.068Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655445600001 maxt=1655467200000 ulid=01G5S9RDMFY0H57AEHV4FCE3NY
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.069Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655467200037 maxt=1655488800000 ulid=01G5SYBMNJGX7RMJKX599SDBA4
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.069Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655488800046 maxt=1655510400000 ulid=01G5TJYXKPTKM9HDQNCR5E0N5S
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.070Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655510400048 maxt=1655532000000 ulid=01G5V7HYTCK7YHRRN0YY3P6NRZ
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.071Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655532000024 maxt=1655553600000 ulid=01G5VW5BW2FKAAKMV6S4E25TGA
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.071Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655553600050 maxt=1655575200000 ulid=01G5WGRH7TKASRJ4KC5BFFP6CW
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.072Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655575200011 maxt=1655596800000 ulid=01G5X5BMH51P8K5GPNN274FKKY
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.073Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655596800053 maxt=1655618400000 ulid=01G5XSYTAS9DHZ8Z9K717BMGD7
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.074Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655618400034 maxt=1655640000000 ulid=01G5YEJ30TV86JSATFG3DPZGNY
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.075Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655640000011 maxt=1655661600000 ulid=01G5Z359YMR66GZHYEZWRGV553
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.075Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655661600041 maxt=1655683200000 ulid=01G5ZQRAXJJ7Y4139FC47C6R9Q
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.076Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655704800001 maxt=1655712000000 ulid=01G605E5Q66YYVD4EFBDJ0J1Q3
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.076Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655712000000 maxt=1655719200000 ulid=01G60C9WZCQTPRM2Z67ZKBCPFS
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.077Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655683200001 maxt=1655704800000 ulid=01G60CBFND7HYV9BWCBZMFJZ4K
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.877Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
Mon, 20 Jun 2022 13:38:36 | level=info ts=2022-06-20T11:38:36.079Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.20253601s
Mon, 20 Jun 2022 13:38:36 | level=info ts=2022-06-20T11:38:36.079Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
Mon, 20 Jun 2022 13:38:43 | level=info ts=2022-06-20T11:38:43.443Z caller=head.go:826 component=tsdb msg="WAL checkpoint loaded"
Mon, 20 Jun 2022 13:38:46 | level=info ts=2022-06-20T11:38:46.451Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16619 maxSegment=16630
Mon, 20 Jun 2022 13:38:49 | level=info ts=2022-06-20T11:38:49.473Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16620 maxSegment=16630
Mon, 20 Jun 2022 13:38:54 | level=info ts=2022-06-20T11:38:54.408Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16621 maxSegment=16630
Mon, 20 Jun 2022 13:38:55 | level=info ts=2022-06-20T11:38:55.863Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16622 maxSegment=16630
Mon, 20 Jun 2022 13:38:58 | level=info ts=2022-06-20T11:38:58.556Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16623 maxSegment=16630
Mon, 20 Jun 2022 13:39:01 | level=info ts=2022-06-20T11:39:01.472Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16624 maxSegment=16630
Mon, 20 Jun 2022 13:39:03 | level=info ts=2022-06-20T11:39:03.790Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16625 maxSegment=16630
Mon, 20 Jun 2022 13:39:06 | level=info ts=2022-06-20T11:39:06.553Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16626 maxSegment=16630
Mon, 20 Jun 2022 13:39:11 | level=info ts=2022-06-20T11:39:11.685Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16627 maxSegment=16630
Mon, 20 Jun 2022 13:39:14 | level=info ts=2022-06-20T11:39:14.656Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16628 maxSegment=16630
Mon, 20 Jun 2022 13:39:16 | level=info ts=2022-06-20T11:39:16.662Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16629 maxSegment=16630
Mon, 20 Jun 2022 13:39:16 | level=info ts=2022-06-20T11:39:16.662Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16630 maxSegment=16630
Mon, 20 Jun 2022 13:39:16 | level=info ts=2022-06-20T11:39:16.662Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=7.364090293s wal_replay_duration=33.219008395s total_replay_duration=42.7856811s
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.613Z caller=main.go:851 fs_type=EXT4_SUPER_MAGIC
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.613Z caller=main.go:854 msg="TSDB started"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.613Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
Mon, 20 Jun 2022 13:39:18 | ts=2022-06-20T11:39:18.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Starting WAL watcher" queue=742148
Mon, 20 Jun 2022 13:39:18 | ts=2022-06-20T11:39:18.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Starting scraped metadata watcher"
Mon, 20 Jun 2022 13:39:18 | ts=2022-06-20T11:39:18.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Replaying WAL" queue=742148
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.620Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.621Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.621Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.622Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.622Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.623Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.623Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.624Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.625Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.625Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.626Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.626Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.627Z caller=kubernetes.go:282 component="discovery manager notify" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.710Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=96.37768ms remote_storage=340.721µs web_handler=434ns query_engine=540ns scrape=209.884µs scrape_sd=7.039296ms notify=20.722µs notify_sd=22.71941ms rules=60.15021ms
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.710Z caller=main.go:796 msg="Server is ready to receive web requests."
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.710Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.716Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | ts=2022-06-20T11:39:18.716Z caller=level.go:63 component=k8s_client_runtime level=error func=ErrorDepth msg="pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: Get \"https://10.43.0.1:443/api/v1/namespaces/sealed-services/pods?allowWatchBookmarks=true&resourceVersion=495641055&timeout=6m49s&timeoutSeconds=409&watch=true\": context canceled"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.717Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.718Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.718Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.719Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.719Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.720Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.720Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.721Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.721Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.722Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.722Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.723Z caller=kubernetes.go:282 component="discovery manager notify" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.835Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=125.168082ms remote_storage=73.181µs web_handler=443ns query_engine=589ns scrape=39.52µs scrape_sd=6.998634ms notify=14.04µs notify_sd=777.09µs rules=111.319021ms
Mon, 20 Jun 2022 13:39:32 | ts=2022-06-20T11:39:32.987Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Done replaying WAL" duration=14.367819655s
Mon, 20 Jun 2022 13:39:40 | ts=2022-06-20T11:39:40.136Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=1 to=6
Mon, 20 Jun 2022 13:39:58 | ts=2022-06-20T11:39:58.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=6 to=10
Mon, 20 Jun 2022 13:40:08 | ts=2022-06-20T11:40:08.622Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=10 to=14
Mon, 20 Jun 2022 13:40:18 | ts=2022-06-20T11:40:18.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=14 to=19
Mon, 20 Jun 2022 13:40:28 | ts=2022-06-20T11:40:28.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=19 to=26
Mon, 20 Jun 2022 13:40:38 | ts=2022-06-20T11:40:38.621Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Currently resharding, skipping."
Mon, 20 Jun 2022 13:40:48 | ts=2022-06-20T11:40:48.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=26 to=43
Mon, 20 Jun 2022 13:41:08 | ts=2022-06-20T11:41:08.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=43 to=66
Mon, 20 Jun 2022 13:41:28 | ts=2022-06-20T11:41:28.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=66 to=97
Mon, 20 Jun 2022 13:41:48 | ts=2022-06-20T11:41:48.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Currently resharding, skipping."
Mon, 20 Jun 2022 13:41:58 | ts=2022-06-20T11:41:58.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=97 to=172
Mon, 20 Jun 2022 13:42:48 | ts=2022-06-20T11:42:48.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=172 to=348
Mon, 20 Jun 2022 13:43:18 | ts=2022-06-20T11:43:18.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Currently resharding, skipping."
Mon, 20 Jun 2022 13:43:48 | ts=2022-06-20T11:43:48.620Z caller=dedupe.go:112 component=remote level=warn remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to send batch, retrying" err="Post \"http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push\": context canceled"
Mon, 20 Jun 2022 13:43:48 | ts=2022-06-20T11:43:48.620Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="non-recoverable error" count=100 exemplarCount=0 err="context canceled"
Mon, 20 Jun 2022 13:43:48 | ts=2022-06-20T11:43:48.621Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to flush all samples on shutdown" count=1003390020
Mon, 20 Jun 2022 13:43:58 | ts=2022-06-20T11:43:58.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=348 to=802
Mon, 20 Jun 2022 13:44:28 | ts=2022-06-20T11:44:28.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Currently resharding, skipping."
Mon, 20 Jun 2022 13:44:58 | ts=2022-06-20T11:44:58.620Z caller=dedupe.go:112 component=remote level=warn remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to send batch, retrying" err="Post \"http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push\": context canceled"
Mon, 20 Jun 2022 13:44:58 | ts=2022-06-20T11:44:58.620Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="non-recoverable error" count=100 exemplarCount=0 err="context canceled"
Mon, 20 Jun 2022 13:44:58 | ts=2022-06-20T11:44:58.622Z caller=dedupe.go:112 component=remote level=warn remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to send batch, retrying" err="Post \"http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push\": context canceled"
Mon, 20 Jun 2022 13:44:58 | ts=2022-06-20T11:44:58.622Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to flush all samples on shutdown" count=3274907571
Mon, 20 Jun 2022 13:45:08 | ts=2022-06-20T11:45:08.632Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=802 to=1000
Mon, 20 Jun 2022 13:46:09 | ts=2022-06-20T11:46:09.029Z caller=dedupe.go:112 component=remote level=warn remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to send batch, retrying" err="Post \"http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push\": context canceled"
Mon, 20 Jun 2022 13:46:09 | ts=2022-06-20T11:46:09.029Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="non-recoverable error" count=100 exemplarCount=0 err="context canceled"
Mon, 20 Jun 2022 13:46:09 | ts=2022-06-20T11:46:09.316Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to flush all samples on shutdown" count=2876743850

[ghost]

Config:

admin_api:
  affinity: {}
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 60
  tolerations: []
alertmanager:
  affinity: {}
  annotations: {}
  enabled: true
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  persistentVolume:
    accessModes:
      - ReadWriteOnce
    annotations: {}
    enabled: true
    size: 10Gi
    subPath: ''
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  statefulSet:
    enabled: true
  statefulStrategy:
    type: RollingUpdate
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 60
  tolerations: []
compactor:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: target
                  operator: In
                  values:
                    - compactor
            topologyKey: kubernetes.io/hostname
          weight: 100
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistentVolume:
    accessModes:
      - ReadWriteOnce
    annotations: {}
    enabled: true
    size: 90Gi
    subPath: ''
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 60
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 512Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    type: RollingUpdate
  terminationGracePeriodSeconds: 240
  tolerations: []
distributor:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target
                operator: In
                values:
                  - distributor
          topologyKey: kubernetes.io/hostname
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 512Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 60
  tolerations: []
enterprise:
  enabled: false
  image:
    repository: grafana/enterprise-metrics
    tag: v2.1.0
  legacyLabels: false
externalConfigSecretName: '{{ include "mimir.resourceName" (dict "ctx" . "component" "config") }}'
externalConfigVersion: '0'
fullnameOverride: null
gateway:
  affinity: {}
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  ingress:
    annotations: {}
    enabled: false
    hosts:
      - host: gateway.gem.example.com
        paths:
          - path: /
    tls:
      - hosts:
          - gateway.gem.example.com
        secretName: gem-gateway-tls
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
    port: null
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 60
  tolerations: []
  useDefaultProxyURLs: true
global:
  clusterDomain: cluster.local
  dnsNamespace: kube-system
  dnsService: kube-dns
image:
  pullPolicy: IfNotPresent
  repository: grafana/mimir
  tag: 2.1.0
ingester:
  affinity: {}
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistentVolume:
    accessModes:
      - ReadWriteOnce
    annotations: {}
    enabled: true
    size: 90Gi
    subPath: ''
  podAnnotations: {}
  podDisruptionBudget:
    maxUnavailable: 1
  podLabels: {}
  podManagementPolicy: Parallel
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 60
  replicas: 3
  resources:
    requests:
      cpu: 100m
      memory: 512Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  statefulSet:
    enabled: true
  statefulStrategy:
    type: RollingUpdate
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 240
  tolerations: []
license:
  contents: NOTAVALIDLICENSE
  external: false
  secretName: '{{ include "mimir.resourceName" (dict "ctx" . "component" "license") }}'
memcached:
  architecture: high-availability
  arguments:
    - '-m 8192'
    - '-o'
    - modern
    - '-v'
    - '-I 1m'
    - '-c 4096'
  enabled: false
  image:
    repository: memcached
    tag: 1.6.9
  maxItemMemory: '1048576'
  metrics:
    enabled: true
    image:
      registry: quay.io
      repository: prometheus/memcached-exporter
      tag: v0.9.0
  replicaCount: 1
  resources:
    limits:
      memory: 9830Mi
    requests:
      cpu: 500m
      memory: 9830Mi
memcached-metadata:
  architecture: high-availability
  arguments:
    - '-m 512'
    - '-o'
    - modern
    - '-v'
    - '-I 1m'
    - '-c 1024'
  enabled: false
  image:
    repository: memcached
    tag: 1.6.9
  maxItemMemory: '1048576'
  metrics:
    enabled: true
    image:
      registry: quay.io
      repository: prometheus/memcached-exporter
      tag: v0.9.0
  replicaCount: 1
  resources:
    limits:
      memory: 614Mi
    requests:
      cpu: 500m
      memory: 614Mi
memcached-queries:
  architecture: high-availability
  arguments:
    - '-m 2048'
    - '-o'
    - modern
    - '-v'
    - '-I 15m'
    - '-c 1024'
  enabled: false
  image:
    repository: memcached
    tag: 1.6.9
  maxItemMemory: '15728640'
  metrics:
    enabled: true
    image:
      registry: quay.io
      repository: prometheus/memcached-exporter
      tag: v0.9.0
  replicaCount: 1
  resources:
    limits:
      memory: 2457Mi
    requests:
      cpu: 500m
      memory: 2457Mi
memcached-results:
  architecture: high-availability
  arguments:
    - '-m 512'
    - '-o'
    - modern
    - '-v'
    - '-I 1m'
    - '-c 1024'
  enabled: false
  image:
    repository: memcached
    tag: 1.6.9
  maxItemMemory: '1048576'
  metrics:
    enabled: true
    image:
      registry: quay.io
      repository: prometheus/memcached-exporter
      tag: v0.9.0
  replicaCount: 1
  resources:
    limits:
      memory: 614Mi
    requests:
      cpu: 500m
      memory: 614Mi
mimir:
  config: >
    {{- if not .Values.enterprise.enabled -}}

    multitenancy_enabled: false

    {{- end }}

    limits:
      compactor_blocks_retention_period: 604800s
      ingestion_rate: 60000
      max_global_series_per_metric: 2000000
      max_global_series_per_user: 2000000
      max_label_names_per_series: 60

    distributor:
      instance_limits:
        max_ingestion_rate: 0
        max_inflight_push_requests: 0
      remote_timeout: 30s

    activity_tracker:
      filepath: /data/metrics-activity.log

    alertmanager:
      data_dir: '/data'
      enable_api: true
      external_url: '/alertmanager'

    {{- if .Values.minio.enabled }}

    alertmanager_storage:
      backend: s3
      s3:
        endpoint: {{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000
        bucket_name: {{ include "mimir.minioBucketPrefix" . }}-ruler
        access_key_id: {{ .Values.minio.accessKey }}
        secret_access_key: {{ .Values.minio.secretKey }}
        insecure: true
    {{- end }}

    frontend_worker:
      frontend_address: {{ template "mimir.fullname" . }}-query-frontend-headless.{{ .Release.Namespace }}.svc:{{ include "mimir.serverGrpcListenPort" . }}

    ruler:
      enable_api: true
      rule_path: '/data'
      alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.{{ template "mimir.fullname" . }}-alertmanager-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}/alertmanager

    server:
      grpc_server_max_recv_msg_size: 104857600
      grpc_server_max_send_msg_size: 104857600
      grpc_server_max_concurrent_streams: 1000

    frontend:
      log_queries_longer_than: 10s
      align_queries_with_step: true
      {{- if index .Values "memcached-results" "enabled" }}
      results_cache:
        backend: memcached
        memcached:
          addresses: dns+{{ .Release.Name }}-memcached-results.{{ .Release.Namespace }}.svc:11211
          max_item_size: {{ (index .Values "memcached-results").maxItemMemory }}
      cache_results: true
      {{- end }}

    compactor:
      data_dir: "/data"

    ingester:
      ring:
        final_sleep: 0s
        num_tokens: 512
        unregister_on_shutdown: false

    ingester_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600

    runtime_config:
      file: /var/{{ include "mimir.name" . }}/runtime.yaml

    memberlist:
      abort_if_cluster_join_fails: false
      compression_enabled: false
      join_members:
      - {{ include "mimir.fullname" . }}-gossip-ring

    # This configures how the store-gateway synchronizes blocks stored in the
    bucket. It uses Minio by default for getting started (configured via flags)
    but this should be changed for production deployments.

    blocks_storage:
      backend: s3
      tsdb:
        dir: /data/tsdb
      bucket_store:
        sync_dir: /data/tsdb-sync
        {{- if .Values.memcached.enabled }}
        chunks_cache:
          backend: memcached
          memcached:
            addresses: dns+{{ .Release.Name }}-memcached.{{ .Release.Namespace }}.svc:11211
            max_item_size: {{ .Values.memcached.maxItemMemory }}
        {{- end }}
        {{- if index .Values "memcached-metadata" "enabled" }}
        metadata_cache:
          backend: memcached
          memcached:
            addresses: dns+{{ .Release.Name }}-memcached-metadata.{{ .Release.Namespace }}.svc:11211
            max_item_size: {{ (index .Values "memcached-metadata").maxItemMemory }}
        {{- end }}
        {{- if index .Values "memcached-queries" "enabled" }}
        index_cache:
          backend: memcached
          memcached:
            addresses: dns+{{ .Release.Name }}-memcached-queries.{{ .Release.Namespace }}.svc:11211
            max_item_size: {{ (index .Values "memcached-queries").maxItemMemory }}
        {{- end }}
      {{- if .Values.minio.enabled }}
      s3:
        endpoint: {{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000
        bucket_name: {{ include "mimir.minioBucketPrefix" . }}-tsdb
        access_key_id: {{ .Values.minio.accessKey }}
        secret_access_key: {{ .Values.minio.secretKey }}
        insecure: true
      {{- end }}

    {{- if .Values.minio.enabled }}

    ruler_storage:
      backend: s3
      s3:
        endpoint: {{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000
        bucket_name: {{ include "mimir.minioBucketPrefix" . }}-ruler
        access_key_id: {{ .Values.minio.accessKey }}
        secret_access_key: {{ .Values.minio.secretKey }}
        insecure: true
    {{- end }}

    {{- if .Values.enterprise.enabled }}

    multitenancy_enabled: true

    admin_api:
      leader_election:
        enabled: true
        ring:
          kvstore:
            store: "memberlist"

    {{- if .Values.minio.enabled }}

    admin_client:
      storage:
        type: s3
        s3:
          endpoint: {{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000
          bucket_name: enterprise-metrics-admin
          access_key_id: {{ .Values.minio.accessKey }}
          secret_access_key: {{ .Values.minio.secretKey }}
          insecure: true
    {{- end }}

    auth:
      type: enterprise

    cluster_name: "{{ .Release.Name }}"

    license:
      path: "/license/license.jwt"

    {{- if .Values.gateway.useDefaultProxyURLs }}

    gateway:
      proxy:
        default:
          url: http://{{ template "mimir.fullname" . }}-admin-api.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        admin_api:
          url: http://{{ template "mimir.fullname" . }}-admin-api.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        alertmanager:
          url: http://{{ template "mimir.fullname" . }}-alertmanager.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        compactor:
          url: http://{{ template "mimir.fullname" . }}-compactor.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        distributor:
          url: dns:///{{ template "mimir.fullname" . }}-distributor-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverGrpcListenPort" . }}
        ingester:
          url: http://{{ template "mimir.fullname" . }}-ingester.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        query_frontend:
          url: http://{{ template "mimir.fullname" . }}-query-frontend.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        ruler:
          url: http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        store_gateway:
          url: http://{{ template "mimir.fullname" . }}-store-gateway.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
    {{- end }}

    instrumentation:
      enabled: true
      distributor_client:
        address: 'dns:///{{ template "mimir.fullname" . }}-distributor-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverGrpcListenPort" . }}'

    {{- end }}
minio:
  accessKey: grafana-mimir
  buckets:
    - name: mimir-tsdb
      policy: none
      purge: false
    - name: mimir-ruler
      policy: none
      purge: false
    - name: enterprise-metrics-tsdb
      policy: none
      purge: false
    - name: enterprise-metrics-admin
      policy: none
      purge: false
    - name: enterprise-metrics-ruler
      policy: none
      purge: false
  enabled: true
  persistence:
    size: 270Gi
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  secretKey: supersecret
nameOverride: null
nginx:
  affinity: |
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: component
                  operator: In
                  values:
                    - nginx
            topologyKey: failure-domain.beta.kubernetes.io/zone
  annotations: {}
  autoscaling:
    enabled: false
    maxReplicas: 3
    minReplicas: 1
    targetCPUUtilizationPercentage: 60
    targetMemoryUtilizationPercentage: null
  basicAuth:
    enabled: false
    existingSecret: null
    htpasswd: >-
      {{ htpasswd (required "'nginx.basicAuth.username' is required"
      .Values.nginx.basicAuth.username) (required "'nginx.basicAuth.password' is
      required" .Values.nginx.basicAuth.password) }}
    password: null
    username: null
  containerSecurityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    readOnlyRootFilesystem: true
  deploymentStrategy:
    type: RollingUpdate
  enabled: true
  extraArgs: []
  extraEnv: []
  extraEnvFrom: []
  extraVolumeMounts: []
  extraVolumes: []
  image:
    pullPolicy: IfNotPresent
    registry: docker.io
    repository: nginxinc/nginx-unprivileged
    tag: 1.19-alpine
  ingress:
    annotations: {}
    enabled: false
    hosts:
      - host: nginx.loki.example.com
        paths:
          - path: /
    tls:
      - hosts:
          - nginx.loki.example.com
        secretName: loki-nginx-tls
  nginxConfig:
    file: |
      worker_processes  5;  ## Default: 1
      error_log  /dev/stderr;
      pid        /tmp/nginx.pid;
      worker_rlimit_nofile 8192;

      events {
        worker_connections  4096;  ## Default: 1024
      }

      http {
        client_body_temp_path /tmp/client_temp;
        proxy_temp_path       /tmp/proxy_temp_path;
        fastcgi_temp_path     /tmp/fastcgi_temp;
        uwsgi_temp_path       /tmp/uwsgi_temp;
        scgi_temp_path        /tmp/scgi_temp;

        default_type application/octet-stream;
        log_format   {{ .Values.nginx.nginxConfig.logFormat }}

        {{- if .Values.nginx.verboseLogging }}
        access_log   /dev/stderr  main;
        {{- else }}

        map $status $loggable {
          ~^[23]  0;
          default 1;
        }
        access_log   /dev/stderr  main  if=$loggable;
        {{- end }}

        sendfile     on;
        tcp_nopush   on;
        resolver {{ .Values.global.dnsService }}.{{ .Values.global.dnsNamespace }}.svc.{{ .Values.global.clusterDomain }};

        {{- with .Values.nginx.nginxConfig.httpSnippet }}
        {{ . | nindent 2 }}
        {{- end }}

        server {
          listen 8080;

          {{- if .Values.nginx.basicAuth.enabled }}
          auth_basic           "Mimir";
          auth_basic_user_file /etc/nginx/secrets/.htpasswd;
          {{- end }}

          location = / {
            return 200 'OK';
            auth_basic off;
          }

          {{- if not (include "mimir.calculatedConfig" . | fromYaml).multitenancy_enabled }}
          proxy_set_header X-Scope-OrgID 0;
          {{- end }}

          # Distributor endpoints
          location /distributor {
            proxy_pass      http://{{ template "mimir.fullname" . }}-distributor-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location = /api/v1/push {
            proxy_pass      http://{{ template "mimir.fullname" . }}-distributor-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          # Alertmanager endpoints
          location {{ template "mimir.alertmanagerHttpPrefix" . }} {
            proxy_pass      http://{{ template "mimir.fullname" . }}-alertmanager.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location = /multitenant_alertmanager/status {
            proxy_pass      http://{{ template "mimir.fullname" . }}-alertmanager.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location = /api/v1/alerts {
            proxy_pass      http://{{ template "mimir.fullname" . }}-alertmanager.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          # Ruler endpoints
          location {{ template "mimir.prometheusHttpPrefix" . }}/config/v1/rules {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location {{ template "mimir.prometheusHttpPrefix" . }}/api/v1/rules {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          location /api/v1/rules {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location {{ template "mimir.prometheusHttpPrefix" . }}/api/v1/alerts {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location {{ template "mimir.prometheusHttpPrefix" . }}/rules {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location = /ruler/ring {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          # Rest of {{ template "mimir.prometheusHttpPrefix" . }} goes to the query frontend
          location {{ template "mimir.prometheusHttpPrefix" . }} {
            proxy_pass      http://{{ template "mimir.fullname" . }}-query-frontend.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          # Buildinfo endpoint can go to any component
          location = /api/v1/status/buildinfo {
            proxy_pass      http://{{ template "mimir.fullname" . }}-query-frontend.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          {{- with .Values.nginx.nginxConfig.serverSnippet }}
          {{ . | nindent 4 }}
          {{- end }}
        }
      }
    httpSnippet: ''
    logFormat: |-
      main '$remote_addr - $remote_user [$time_local]  $status '
              '"$request" $body_bytes_sent "$http_referer" '
              '"$http_user_agent" "$http_x_forwarded_for"';
    serverSnippet: ''
  nodeSelector: {}
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  podSecurityContext:
    fsGroup: 101
    runAsGroup: 101
    runAsNonRoot: true
    runAsUser: 101
  priorityClassName: null
  readinessProbe:
    httpGet:
      path: /
      port: http-metric
    initialDelaySeconds: 15
    timeoutSeconds: 1
  replicas: 1
  resources: {}
  service:
    annotations: {}
    clusterIP: null
    labels: {}
    loadBalancerIP: null
    nodePort: null
    port: 80
    type: ClusterIP
  terminationGracePeriodSeconds: 30
  tolerations: []
  verboseLogging: true
overrides_exporter:
  affinity: {}
  annotations: {}
  enabled: true
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  livenessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 60
  tolerations: []
querier:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: target
                  operator: In
                  values:
                    - querier
            topologyKey: kubernetes.io/hostname
          weight: 100
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 2
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 180
  tolerations: []
query_frontend:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: target
                  operator: In
                  values:
                    - query-frontend
            topologyKey: kubernetes.io/hostname
          weight: 100
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 180
  tolerations: []
rbac:
  pspEnabled: true
ruler:
  affinity: {}
  annotations: {}
  enabled: true
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 180
  tolerations: []
runtimeConfig: {}
serviceAccount:
  annotations: {}
  create: true
  name: null
serviceMonitor:
  annotations: {}
  enabled: false
  interval: null
  labels: {}
  namespace: null
  namespaceSelector: {}
  relabelings: []
  scheme: http
  scrapeTimeout: null
  tlsConfig: null
store_gateway:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target
                operator: In
                values:
                  - store-gateway
          topologyKey: kubernetes.io/hostname
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistentVolume:
    accessModes:
      - ReadWriteOnce
    annotations: {}
    enabled: true
    size: 90Gi
    subPath: ''
  podAnnotations: {}
  podDisruptionBudget:
    maxUnavailable: 1
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 60
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 512Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    type: RollingUpdate
  terminationGracePeriodSeconds: 240
  tolerations: []
tokengenJob:
  annotations: {}
  enable: true
  env: []
  extraArgs: {}
  initContainers: []
useExternalConfig: false

[krajorama]

Hi, out-of-order sample support is an upcoming feature in Mimir (https://github.com/grafana/mimir/pull/2187). Possibly @codesome can comment on this issue?

[codesome]

Mimir 2.2 will have experimental support for accepting out of order samples. The release will come with docs on how to enable this feature and use this. Stay tuned!

[ghost]

Very nice. But what about a alternative solution until then? How can i tell prometheus not to send old samples? How can i disable this behaviour? I know this is a question for another issue in another place but i am curious. :)