search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
12.82k stars 2.04k forks source link

Thanos receive context deadline exceeded: Permanent error: Post \"http://obs-thanos-receive.xxxsvc.cluster.local:19291/api/v1/receive\": context deadline exceeded #7257

Open yliu138repo opened 3 months ago

yliu138repo commented 3 months ago

Thanos, Prometheus and Golang version used: Thanos: v0.34.1

Object Storage Provider: MinIO object storage

What happened: Our pipeline has a OpenTelemetry collector in front of the thanos Receiver stacks.

We performed load testing with TPS (Transaction Per Second)=2500 using k6 testing tool, we observed that there are significant data drops, and errors from Opentelemetry collector shows some context deadline timeout issue, e.g.

2024-04-04T07:14:46.107Z        error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"http://obs-thanos-receive.engcoeliberty-dev-di1001.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded; Permanent error: Permanent error: Post \"http://obs-thanos-receive.engcoeliberty-dev-di1001.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded", "errorCauses": [{"error": "Permanent error: Permanent error: Post \"http://obs-thanos-receive.xxx.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded"}, {"error": "Permanent error: Permanent error: Post \"http://obs-thanos-receive.engcoeliberty-dev-di1001.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded"}], "dropped_items": 304868}

However, we are not able to identify any errors from the receiver logs except some out-of-order samples, which is expected.

We don't have any limits configured on Thanos receiver, and as far as we are concerned, By default all the limits and gates are disabled on receivers.

Receiver config:

        - receive
        - '--log.level=debug'
        - '--log.format=logfmt'
        - '--grpc-address=0.0.0.0:10901'
        - '--http-address=0.0.0.0:10902'
        - '--remote-write.address=0.0.0.0:19291'
        - '--receive.hashrings-algorithm=ketama'
        - '--objstore.config=$(OBJSTORE_CONFIG)'
        - '--tsdb.path=/var/thanos/receive'
        - '--label=replica="$(NAME)"'
        - '--label=receive="true"'
        - '--tsdb.retention=2h'
        - >-
          --receive.local-endpoint=$(NAME).obs-thanos-receive-headless.$(NAMESPACE).svc.cluster.local:10901

Receiver distributor config

        - receive
        - '--log.level=debug'
        - '--log.format=logfmt'
        - '--grpc-address=0.0.0.0:10901'
        - '--http-address=0.0.0.0:10902'
        - '--remote-write.address=0.0.0.0:19291'
        - '--label=replica="$(NAME)"'
        - '--label=receive="true"'
        - '--receive.hashrings-file=/var/lib/thanos-receive/hashrings.json'
        - '--receive.hashrings-file-refresh-interval=3s'
        - '--receive.replication-factor=1'

Hashring

[{"hashring":"default","endpoints":["obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901","obs-thanos-receive-1.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901"]}]

It works perfectly with TPS=1500

What you expected to happen: There should be minimal data drops with TPS=2500 as we have similar load testing against Prometheus before, which works fine.

How to reproduce it (as minimally and precisely as possible):

  1. Deploy the pipeline with Opentelmetry collector and Thanos receiver stacks
  2. Load test with k6 using following script together with command 
    
import { group, check } from 'k6';
import http from 'k6/http';
import { sleep } from 'k6';

export const options = { systemTags: ['status', 'proto', 'method', 'name', 'group', 'error', 'error_code', 'scenario', 'expected_response'], insecureSkipTLSVerify: true, discardResponseBodies: true,

scenarios: { contacts: { executor: 'ramping-arrival-rate', // Requests per iteration = 200 // Start iterations per timeUnit startRate: 1, // Start startRate iterations per second timeUnit: '1s', // Pre-allocate necessary VUs. preAllocatedVUs: 5000, stages: [ // Start 300 iterations per timeUnit for the first minute. { target: 25, duration: '15m' }, // Linearly ramp-up to starting 600 iterations per timeUnit over the following two minutes. { target: 25, duration: '45m' }, // Continue starting 600 iterations per timeUnit for the following four minutes. //{ target: 200, duration: '1m' }, // Linearly ramp-down to starting 60 iterations per timeUnit over the last two minutes. //{ target: 10, duration: '1m' }, ], }, }, };

export default function () {

// Generate a random number between 1 and 100000 (adjust the range as needed) //const randomNumber = Math.floor(Math.random() * 100000) + VU + ITER;

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName1 01 Load Alpha Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Microfocus.html, {tags: { name: 'AppOrModuleName1 02 Load Beta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName1 03 Load Gamma Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName1 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName1 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName1 01 Load Alpha Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName1 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName1 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName1 07 Download File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName1 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName1 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName1 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName2 01 Load Login Page'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName2 02 Login'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName2 03 View Dashboard'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName2 04 Search'},});

// Execute the HTTP request

http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName2 04 Search'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName2 05 Modify Entity'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName2 06 Create New Entity'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName2 07 Link Entity'},});

// Execute the HTTP request

http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName3 01 Load Alpha Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Microfocus.html, {tags: { name: 'AppOrModuleName3 02 Load Beta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName3 03 Load Gamma Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName3 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName3 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName3 01 Load Alpha Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName3 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName3 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName3 07 Download File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName3 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName3 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName3 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName4 01 Load Login Page'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName4 02 Login'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName4 03 View Dashboard'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName4 04 Search'},});

// Execute the HTTP request

http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName4 04 Search'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName4 05 Modify Entity'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName4 06 Create New Entity'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName4 07 Link Entity'},});

// Execute the HTTP request

http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName5 01 Load Alpha Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Microfocus.html, {tags: { name: 'AppOrModuleName5 02 Load Beta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName5 03 Load Gamma Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName5 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName5 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName5 01 Load Alpha Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName5 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName5 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName5 07 Download File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName5 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName5 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName5 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName6 01 Load Login Page'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName6 02 Login'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName6 03 View Dashboard'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName6 04 Search'},});

// Execute the HTTP request

http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName6 04 Search'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName6 05 Modify Entity'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName6 06 Create New Entity'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName6 07 Link Entity'},});

// Execute the HTTP request

http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName7 01 Load Alpha Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Microfocus.html, {tags: { name: 'AppOrModuleName7 02 Load Beta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName7 03 Load Gamma Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName7 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName7 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName7 01 Load Alpha Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName7 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName7 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName7 07 Download File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName7 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName7 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName7 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName8 01 Load Login Page'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName8 02 Login'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName8 03 View Dashboard'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName8 04 Search'},});

// Execute the HTTP request

http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName8 04 Search'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName8 05 Modify Entity'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName8 06 Create New Entity'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName8 07 Link Entity'},});

// Execute the HTTP request

http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName9 01 Load Alpha Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Microfocus.html, {tags: { name: 'AppOrModuleName9 02 Load Beta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName9 03 Load Gamma Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName9 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName9 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName9 01 Load Alpha Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName9 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName9 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName9 07 Download File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName9 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName9 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName9 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName10 01 Load Login Page'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName10 02 Login'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName10 03 View Dashboard'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName10 04 Search'},});

// Execute the HTTP request

http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName10 04 Search'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName10 05 Modify Entity'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName10 06 Create New Entity'},});

// Execute the HTTP request http.get('https://appau101mel1108.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName10 07 Link Entity'},});

// Execute the HTTP request http.get(https://appau101mel1108.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName11 01 Load Alpha Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Microfocus.html, {tags: { name: 'AppOrModuleName11 02 Load Beta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName11 03 Load Gamma Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName11 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName11 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName11 01 Load Alpha Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName11 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName11 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName11 07 Download File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName11 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName11 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName11 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName12 01 Load Login Page'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName12 02 Login'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName12 03 View Dashboard'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName12 04 Search'},});

// Execute the HTTP request

http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName12 04 Search'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName12 05 Modify Entity'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName12 06 Create New Entity'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName12 07 Link Entity'},});

// Execute the HTTP request

http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName13 01 Load Alpha Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Microfocus.html, {tags: { name: 'AppOrModuleName13 02 Load Beta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName13 03 Load Gamma Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName13 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName13 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName13 01 Load Alpha Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName13 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName13 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName13 07 Download File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName13 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName13 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName13 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName14 01 Load Login Page'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName14 02 Login'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName14 03 View Dashboard'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName14 04 Search'},});

// Execute the HTTP request

http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName14 04 Search'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName14 05 Modify Entity'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName14 06 Create New Entity'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName14 07 Link Entity'},});

// Execute the HTTP request

http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName15 01 Load Alpha Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Microfocus.html, {tags: { name: 'AppOrModuleName15 02 Load Beta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName15 03 Load Gamma Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName15 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName15 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName15 01 Load Alpha Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName15 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName15 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName15 07 Download File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName15 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName15 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName15 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName16 01 Load Login Page'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName16 02 Login'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName16 03 View Dashboard'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName16 04 Search'},});

// Execute the HTTP request

http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName16 04 Search'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName16 05 Modify Entity'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName16 06 Create New Entity'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName16 07 Link Entity'},});

// Execute the HTTP request

http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName17 01 Load Alpha Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Microfocus.html, {tags: { name: 'AppOrModuleName17 02 Load Beta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName17 03 Load Gamma Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName17 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName17 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName17 01 Load Alpha Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName17 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName17 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName17 07 Download File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName17 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName17 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName17 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName18 01 Load Login Page'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName18 02 Login'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName18 03 View Dashboard'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName18 04 Search'},});

// Execute the HTTP request

http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName18 04 Search'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName18 05 Modify Entity'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName18 06 Create New Entity'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName18 07 Link Entity'},});

// Execute the HTTP request

http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName19 01 Load Alpha Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Microfocus.html, {tags: { name: 'AppOrModuleName19 02 Load Beta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/ABCNews.html', {tags: { name: 'AppOrModuleName19 03 Load Gamma Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName19 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName19 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName19 01 Load Alpha Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName19 04 Load Delta Website'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName19 05 Load Epsilon Website'},});

// Execute the HTTP request http.get(https://appau101mel1107.globaltest.blabla.com/websites/Google.html, {tags: { name: 'AppOrModuleName19 07 Download File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName19 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName19 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName19 08 Upload File'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName20 01 Load Login Page'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName20 02 Login'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName20 03 View Dashboard'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName20 04 Search'},});

// Execute the HTTP request

http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName20 04 Search'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName20 05 Modify Entity'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/NYTimes.html', {tags: { name: 'AppOrModuleName20 06 Create New Entity'},});

// Execute the HTTP request http.get('https://appau101mel1107.globaltest.blabla.com/websites/RevolutionIT.html', {tags: { name: 'AppOrModuleName20 07 Link Entity'},});

}


**Full logs to relevant components**:

<!--
Uncomment if you would like to post collapsible logs:

<details>Logs
<p>

2024-04-04T07:14:46.107Z error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"http://obs-thanos-receive.engcoeliberty-dev-di1001.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded; Permanent error: Permanent error: Post \"http://obs-thanos-receive.engcoeliberty-dev-di1001.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded", "errorCauses": [{"error": "Permanent error: Permanent error: Post \"http://obs-thanos-receive.engcoeliberty-dev-di1001.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded"}, {"error": "Permanent error: Permanent error: Post \"http://obs-thanos-receive.engcoeliberty-dev-di1001.svc.cluster.local:19291/api/v1/receive\": context deadline exceeded"}], "dropped_items": 304868} go.opentelemetry.io/collector/exporter/exporterhelper.(retrySender).send go.opentelemetry.io/collector/exporter@v0.80.0/exporterhelper/queued_retry.go:391 go.opentelemetry.io/collector/exporter/exporterhelper.(metricsSenderWithObservability).send go.opentelemetry.io/collector/exporter@v0.80.0/exporterhelper/metrics.go:125 go.opentelemetry.io/collector/exporter/exporterhelper.(queuedRetrySender).start.func1 go.opentelemetry.io/collector/exporter@v0.80.0/exporterhelper/queued_retry.go:195 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(boundedMemoryQueue).StartConsumers.func1 go.opentelemetry.io/collector/exporter@v0.80.0/exporterhelper/internal/bounded_memory_queue.go:47



</p>
</details>
-->

**Anything else we need to know**:

<!--
Uncomment and fill if you use not casual environment or if it might be relevant.

**Environment**:
- OS (e.g. from /etc/os-release):
- Kernel (e.g. `uname -a`):
- Others:
Openshift cluster
-->
GiedriusS commented 3 months ago

Maybe you could try the version from main in your tests and check whether https://github.com/thanos-io/thanos/commit/daa34a52cc41d609d184013707c55f0ad28b60e2 helps? :thinking:

yliu138repo commented 3 months ago

@GiedriusS I have tried with main-2024-04-03-f80fd94, I am still seeing following error context deadline exceeded. I've also tried to configure otel collector timeout. Does it mean I will need to increase aysnc workers --receive.forward.async-workers=x with number of async workers higher than 5?

2024-04-05T04:19:49.099Z        error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data.       {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 75215 samples: out of order sample; forwarding request to endpoint obs-th; Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 8251 samples: out of order sample; forwarding request to endpoint obs-tha; Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 72800 samples: out of order sample; forwarding request to endpoint obs-th; Permanent error: remote write returned HTTP status 500 Internal Server Error; err = %!w(<nil>): context deadline exceeded\n", "errorCauses": [{"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 75215 samples: out of order sample; forwarding request to endpoint obs-th"}, {"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 8251 samples: out of order sample; forwarding request to endpoint obs-tha"}, {"error": "Permanent error: Permanent error: remote write returned HTTP status 409 Conflict; err = %!w(<nil>): 2 errors: forwarding request to endpoint obs-thanos-receive-0.obs-thanos-receive-headless.engcoeliberty-dev-di1001.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = add 72800 samples: out of order sample; forwarding request to endpoint obs-th"}, {"error": "Permanent error: remote write returned HTTP status 500 Internal Server Error; err = %!w(<nil>): context deadline exceeded\n"}], "dropped_items": 461230}