search

nerc-project / operations

Issues related to the operation of the NERC OpenShift environment
1 stars 0 forks source link

Monitor effects on timeouts and performance. (follow up #473 & #596) #618

Open schwesig opened 6 days ago

schwesig commented 6 days ago

Follow up from

Known checks

Status

level=info ts=2024-06-24T13:46:37.338578312Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=893.994626ms duration_ms=893 cached=2128 returned=694 partial=0

level=warn ts=2024-06-24T13:46:44.853880072Z caller=bucket.go:637 msg="loading block failed" elapsed=7.514862006s id=01HQ6GM... err="create index header reader: write index header: new index reader: get TOC from object storage of 01HQ6GM.../index: Get \"https://s3.openshift-storage.svc/observability-a6581571-.../01HQ6GM.../index\": Connection closed by foreign host https://s3.openshift-storage.svc/observability-a6581571-.../01HQ6GM.../index. Retry again."

schwesig commented 2 days ago

check warning on observability-thanos-store-shard-0-0

https://access.redhat.com/support/cases/#/case/03764352

observability-thanos-store-shard-0-0 link

level=warn ts=2024-06-28T08:52:39.405735916Z caller=bucket.go:637 msg="loading block failed"
elapsed=2.930443482s id=01HQ6GMBQ9EWTE00ZSCME83RT3
err="create index header reader: write index
header: new index reader: get TOC from object storage of 01HQ6GMBQ9EWTE00ZSCME83RT3/index:
Get \"https://s3.openshift-storage.svc/observability-a6581571-5ded-446c-8ab8-9008e45e7e33/01HQ6GMBQ9EWTE00ZSCME83RT3/index\":
Connection closed by foreign host https://s3.openshift-storage.svc/observability-a6581571-5ded-446c-8ab8-9008e45e7e33/01HQ6GMBQ9EWTE00ZSCME83RT3/index. Retry again."
schwesig commented 2 days ago

check retention on OBS

https://access.redhat.com/support/cases/#/case/03861871

schwesig commented 2 days ago

/CC @computate