search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.09k stars 2.1k forks source link

msg="reading timestamps failed" err="read shipper meta file: open /home/server/prometheus/data/thanos.shipper.json: no such file or directory" #2007

Closed aarono0 closed 4 years ago

aarono0 commented 4 years ago

Hi, When I started thanos, I uploaded the file to S3 for storage. The file thanos.shipper. Json could not be found in the log error message，can you help me?

Here is the configuration information：

Environment:

thanos sidecar start command:

/usr/local/prometheus/plugins/thanos sidecar \
--http-address="0.0.0.0:19191" \
--grpc-address="0.0.0.0:19091" \
--prometheus.url="http://127.0.0.1:9090" \
--tsdb.path="/home/server/prometheus/data" \
--objstore.config-file="/home/server/supervisor/etc/s3.yml"

s3.yml:

type: S3
config:
  bucket: "***"
  endpoint: "s3.cn-north-1.amazonaws.com.cn"
  region: "cn-north-1"
  access_key: "***"
  insecure: false
  signature_version2: true
  encrypt_sse: false
  secret_key: "***"
  put_user_metadata: {}
  http_config:
    idle_conn_timeout: 120s
    response_header_timeout: 2m
    insecure_skip_verify: false
  trace:
    enable: false
  part_size: 134217728

log:

level=warn ts=2020-01-16T02:29:21.29481209Z caller=sidecar.go:320 msg="reading timestamps failed" err="read shipper meta file: open /home/server/prometheus/data/thanos.shipper.json: no such file or directory"
level=warn ts=2020-01-16T02:29:51.680871959Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: 400 Bad Request" uploaded=0
level=warn ts=2020-01-16T02:29:51.68097365Z caller=sidecar.go:320 msg="reading timestamps failed" err="read shipper meta file: open /home/server/prometheus/data/thanos.shipper.json: no such file or directory"
level=warn ts=2020-01-16T02:30:21.286514522Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: 400 Bad Request" uploaded=0
level=warn ts=2020-01-16T02:30:21.286588814Z caller=sidecar.go:320 msg="reading timestamps failed" err="read shipper meta file: open /home/server/prometheus/data/thanos.shipper.json: no such file or directory"
aarono0 commented 4 years ago

I don't know how the file "thanos.shipper. Json" was generated, thanks.

GiedriusS commented 4 years ago

You were supposed to get other warnings about updating meta file failed. Could you please paste them as well?

aarono0 commented 4 years ago

Thanks, I later used tencent cos to solve this problem, which is probably because signature_version2 on S3 storage is not supported, it needs v3v4 version.

aarono0 commented 4 years ago

If there are policy Settings on s3, I hope to get your some hints, thank you very much. @GiedriusS

FUSAKLA commented 4 years ago

Hi, do I understand well that you managed to resolve the issue or do you still need help with? :)

caarlos0 commented 4 years ago

Similar issue here with s3, thanos 0.10.0 (official docker img), prometheus 2.15.2 (official docker img) running on kubernetes:

prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T01:00:21.723850686Z caller=shipper.go:349 msg="upload new block" id=01DZ2ST1N5WTGPTQABN3Y79SGT
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T03:00:21.712021051Z caller=shipper.go:349 msg="upload new block" id=01DZ30NRX4WW65Q5B36K29QKQR
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T05:00:21.717237396Z caller=shipper.go:349 msg="upload new block" id=01DZ37HG547QFWX49E2FBVYPQR
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T07:00:21.71461379Z caller=shipper.go:349 msg="upload new block" id=01DZ3ED7D53TZNPPDDNFSYFHXG
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T09:00:21.720872786Z caller=shipper.go:349 msg="upload new block" id=01DZ3N8YN4M3GXAVXCGWC1CPA0
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T11:00:21.716841143Z caller=shipper.go:349 msg="upload new block" id=01DZ3W4NX5P9CXKPCXF39A4TBX
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T13:00:21.707624929Z caller=shipper.go:349 msg="upload new block" id=01DZ430D554J83V5YBS6R9MV1K
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T15:00:21.728489652Z caller=shipper.go:349 msg="upload new block" id=01DZ49W4D5NTWHRC1QV5WVZBZE
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T17:00:21.721834631Z caller=shipper.go:349 msg="upload new block" id=01DZ4GQVN5VE8YY6PG16NEC4K3
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T19:00:21.710419328Z caller=shipper.go:349 msg="upload new block" id=01DZ4QKJX4VWMHDMB9555HDKM7
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T21:00:21.756248643Z caller=shipper.go:349 msg="upload new block" id=01DZ4YFA55JQP91DPMVEEMD18M
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-21T23:00:21.715856853Z caller=shipper.go:349 msg="upload new block" id=01DZ55B1D48MWRA3FH3J7R8NSV
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T01:00:21.722042912Z caller=shipper.go:349 msg="upload new block" id=01DZ5C6RN4MX3XWTAFASZQN0JA
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T03:00:21.726451717Z caller=shipper.go:349 msg="upload new block" id=01DZ5K2FX5CQSX17PVED7AAJ1Q
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T05:00:21.717652742Z caller=shipper.go:349 msg="upload new block" id=01DZ5SY755584SNNFYST1QY6RA
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T07:00:21.714523221Z caller=shipper.go:349 msg="upload new block" id=01DZ60SYD49SDF7HMH96P4Z6QG
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T09:00:21.716141193Z caller=shipper.go:349 msg="upload new block" id=01DZ67NNN4AGD18GABT22F7MP6
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T11:00:21.717361126Z caller=shipper.go:349 msg="upload new block" id=01DZ6EHCX4110EG9H86NGNYP9Z
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T13:00:21.705228484Z caller=shipper.go:349 msg="upload new block" id=01DZ6ND455FYFYGKKGKF480QES
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T15:00:21.711048305Z caller=shipper.go:349 msg="upload new block" id=01DZ6W8VD55YDXYWHVB632ZJGB
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T17:00:21.709521236Z caller=shipper.go:349 msg="upload new block" id=01DZ734JN4FWJBW4JWW5C01SQM
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=warn ts=2020-01-22T19:03:51.390387908Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X4DJ5Q88PGNQK2JFPX/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.0.10:32795->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=warn ts=2020-01-22T19:07:18.534930136Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X4DJ5Q88PGNQK2JFPX/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.0.10:60281->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=warn ts=2020-01-22T19:10:49.017667136Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X4DJ5Q88PGNQK2JFPX/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.0.10:48997->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=warn ts=2020-01-22T19:14:21.439272627Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X4DJ5Q88PGNQK2JFPX/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.0.10:43793->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=warn ts=2020-01-22T19:17:53.407819894Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X4DJ5Q88PGNQK2JFPX/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.0.10:47677->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=warn ts=2020-01-22T19:21:32.31221203Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X4DJ5Q88PGNQK2JFPX/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.0.10:57422->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=warn ts=2020-01-22T19:25:12.873461756Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X4DJ5Q88PGNQK2JFPX/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.0.10:60807->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T19:25:13.462732204Z caller=shipper.go:349 msg="upload new block" id=01DZ7A09X4DJ5Q88PGNQK2JFPX
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T21:00:21.729498706Z caller=shipper.go:349 msg="upload new block" id=01DZ7GW154X8J8172H5JEA5ZG7
prometheus-thanos-prometheus-operator-prometheus-1 thanos-sidecar level=info ts=2020-01-22T23:00:21.716900644Z caller=shipper.go:349 msg="upload new block" id=01DZ7QQRD42XZVFEGSCJ4YPSPP
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T01:00:25.786976843Z caller=shipper.go:349 msg="upload new block" id=01DZ2ST1N5ES8QVPGN4ZCMST3H
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T03:00:25.784086534Z caller=shipper.go:349 msg="upload new block" id=01DZ30NRX5Q9XWE2ZAGJ8J0HG4
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T05:00:25.793478411Z caller=shipper.go:349 msg="upload new block" id=01DZ37HG55Y5SZ5PQN64WQCX24
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T07:00:25.797855539Z caller=shipper.go:349 msg="upload new block" id=01DZ3ED7D5JK5512NZBCSP1R0M
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T09:00:25.786203085Z caller=shipper.go:349 msg="upload new block" id=01DZ3N8YN55E79VEN91KSKQDJ6
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T11:00:25.796718251Z caller=shipper.go:349 msg="upload new block" id=01DZ3W4NX561ATEKKV1GRZAEYB
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T13:00:25.820571801Z caller=shipper.go:349 msg="upload new block" id=01DZ430D551JMSYV3JQ289R7R3
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T15:00:25.792281417Z caller=shipper.go:349 msg="upload new block" id=01DZ49W4D57KA6H44RGSXWTE9A
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T17:00:25.78688602Z caller=shipper.go:349 msg="upload new block" id=01DZ4GQVN5Z7CDFFN54HR1SJFV
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T19:00:25.785316169Z caller=shipper.go:349 msg="upload new block" id=01DZ4QKJX5RRKGA7EV8PDV0F2R
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T21:00:25.783926154Z caller=shipper.go:349 msg="upload new block" id=01DZ4YFA5514KN9XC49CZ8E6N3
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-21T23:00:25.785819526Z caller=shipper.go:349 msg="upload new block" id=01DZ55B1D5KHFNQFAGCF27W1WC
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T01:00:25.78937559Z caller=shipper.go:349 msg="upload new block" id=01DZ5C6RN5RSAEKTCMR0X5JEAH
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T03:00:25.790867866Z caller=shipper.go:349 msg="upload new block" id=01DZ5K2FX5WP25HPYH1X5SHFTA
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T05:00:25.783384762Z caller=shipper.go:349 msg="upload new block" id=01DZ5SY755TWSPBBC31E1XQ7WR
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T07:00:25.786750642Z caller=shipper.go:349 msg="upload new block" id=01DZ60SYD5N1X3SP4TE2TRPKJ8
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T09:00:25.788203533Z caller=shipper.go:349 msg="upload new block" id=01DZ67NNN5AHNEKRQF90WREPZ9
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T11:00:25.786970862Z caller=shipper.go:349 msg="upload new block" id=01DZ6EHCX5CWKP4C8P9F2J8FF8
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T13:00:25.804080041Z caller=shipper.go:349 msg="upload new block" id=01DZ6ND455PV6CKCB4D0CWQNTG
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T15:00:25.796386432Z caller=shipper.go:349 msg="upload new block" id=01DZ6W8VD5V965SD6XV4PYEJGR
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T17:00:25.785215562Z caller=shipper.go:349 msg="upload new block" id=01DZ734JN5TZ05BJRDNY30PV6Z
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=warn ts=2020-01-22T19:03:55.984981571Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X57MV6S2H0Y9WMYGHT/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.2.8:34883->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=warn ts=2020-01-22T19:07:16.003646734Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X57MV6S2H0Y9WMYGHT/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.2.8:59813->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=warn ts=2020-01-22T19:11:03.067578488Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X57MV6S2H0Y9WMYGHT/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.2.8:39282->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=warn ts=2020-01-22T19:14:39.663398742Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X57MV6S2H0Y9WMYGHT/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.2.8:53716->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=warn ts=2020-01-22T19:18:15.262516564Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X57MV6S2H0Y9WMYGHT/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.2.8:39405->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=warn ts=2020-01-22T19:21:36.560898543Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X57MV6S2H0Y9WMYGHT/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.2.8:52289->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=warn ts=2020-01-22T19:24:59.587062087Z caller=sidecar.go:315 err="iter local block metas: check exists: stat s3 object: Head https://my-bucket.s3.dualstack.us-west-1.amazonaws.com/01DZ7A09X57MV6S2H0Y9WMYGHT/meta.json: dial tcp: lookup my-bucket.s3.dualstack.us-west-1.amazonaws.com on 172.20.0.10:53: read udp 10.0.2.8:33814->172.20.0.10:53: i/o timeout" uploaded=0
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T19:25:15.175666154Z caller=shipper.go:349 msg="upload new block" id=01DZ7A09X57MV6S2H0Y9WMYGHT
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T21:00:25.786623549Z caller=shipper.go:349 msg="upload new block" id=01DZ7GW1551X5MGEXPJ7EJY6JS
prometheus-thanos-prometheus-operator-prometheus-0 thanos-sidecar level=info ts=2020-01-22T23:00:25.788537448Z caller=shipper.go:349 msg="upload new block" id=01DZ7QQRD5085BE6FTBJFZHYM6

seems like some sort of race condition?

stale[bot] commented 4 years ago

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.