search

canonical / opensearch-operator

OpenSearch operator
Apache License 2.0
11 stars 6 forks source link

[backup] Tests occasionally fail after the 30m mark with "repository-s3 unreachable" #301

Closed phvalguima closed 4 months ago

phvalguima commented 4 months ago

I am noticing that, occasionally, we get test failures because the OpenSearch is not able to reach its S3 repository any longer:

The CI run eventually fails and the logs will point to:

2024-05-06T19:31:54.7439602Z unit-main-0: 19:26:12 ERROR unit.main/0.juju-log s3-credentials:19: Request PUT to https://10.81.35.210:9200/_snapshot/s3-repository with payload: {'type': 's3', 'settings': {'endpoint': 'http://localhost', 'protocol': 'http', 'bucket': 'error', 'base_path': '/', 'region': 'default'}} failed.(Attempts left: 5)
2024-05-06T19:31:54.7443652Z    Error: 500 Server Error: Internal Server Error for url: https://10.81.35.210:9200/_snapshot/s3-repository
2024-05-06T19:31:54.7447603Z unit-main-0: 19:26:14 ERROR unit.main/0.juju-log s3-credentials:19: Request PUT to https://10.81.35.210:9200/_snapshot/s3-repository with payload: {'type': 's3', 'settings': {'endpoint': 'http://localhost', 'protocol': 'http', 'bucket': 'error', 'base_path': '/', 'region': 'default'}} failed.(Attempts left: 4)
2024-05-06T19:31:54.7451431Z    Error: 500 Server Error: Internal Server Error for url: https://10.81.35.210:9200/_snapshot/s3-repository
2024-05-06T19:31:54.7455157Z unit-main-0: 19:26:15 ERROR unit.main/0.juju-log s3-credentials:19: Request PUT to https://10.81.35.210:9200/_snapshot/s3-repository with payload: {'type': 's3', 'settings': {'endpoint': 'http://localhost', 'protocol': 'http', 'bucket': 'error', 'base_path': '/', 'region': 'default'}} failed.(Attempts left: 3)
2024-05-06T19:31:54.7458823Z    Error: 500 Server Error: Internal Server Error for url: https://10.81.35.210:9200/_snapshot/s3-repository
2024-05-06T19:31:54.7462450Z unit-main-0: 19:26:17 ERROR unit.main/0.juju-log s3-credentials:19: Request PUT to https://10.81.35.210:9200/_snapshot/s3-repository with payload: {'type': 's3', 'settings': {'endpoint': 'http://localhost', 'protocol': 'http', 'bucket': 'error', 'base_path': '/', 'region': 'default'}} failed.(Attempts left: 2)
2024-05-06T19:31:54.7466642Z    Error: 500 Server Error: Internal Server Error for url: https://10.81.35.210:9200/_snapshot/s3-repository
2024-05-06T19:31:54.7471900Z unit-main-0: 19:26:18 ERROR unit.main/0.juju-log s3-credentials:19: Request PUT to https://10.81.35.210:9200/_snapshot/s3-repository with payload: {'type': 's3', 'settings': {'endpoint': 'http://localhost', 'protocol': 'http', 'bucket': 'error', 'base_path': '/', 'region': 'default'}} failed.(Attempts left: 1)
2024-05-06T19:31:54.7475420Z    Error: 500 Server Error: Internal Server Error for url: https://10.81.35.210:9200/_snapshot/s3-repository
2024-05-06T19:31:54.7477620Z unit-main-0: 19:26:19 ERROR unit.main/0.juju-log s3-credentials:19: Failed to setup backup service with state repository s3 is unreachable
github-actions[bot] commented 4 months ago

https://warthogs.atlassian.net/browse/DPE-4303