search

splunk / splunk-operator

Splunk Operator for Kubernetes
Other
210 stars 115 forks source link

Smartstore: Failed to execute runSync() #1332

Closed nfilip21 closed 1 day ago

nfilip21 commented 6 months ago

Please select the type of request

Bug

Tell us more

Describe the request

CloudTrail can be used to confirm that the token has expired.

Expected behavior

Splunk setup on K8S

Splunk Setup

K8s environment

Proposed changes(optional)

K8s collector data(optional)

Additional context(optional)

raghukumarc commented 3 months ago

We have been seeing similar errors as reported. Appreciate help here: 07-17-2024 06:03:09.545 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:03:10.832 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:03:13.120 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:03:17.398 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:03:25.683 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:03:41.973 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:04:14.255 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:05:14.537 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:06:14.820 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:07:15.099 +0000 ERROR AwsCredentials [143795 CMUploadReplicatedBucketThread] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:08:06.263 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:08:07.551 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:08:09.830 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:08:14.112 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:08:22.396 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:08:38.702 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:09:10.986 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:10:11.274 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:11:11.554 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:12:11.834 +0000 ERROR AwsCredentials [568873 FilesystemOpExecutorWorker-0] - Failed to execute runSync() for transaction with uri=https://sts.amazonaws.com http_code=400 07-17-2024 06:12:11.848 +0000 ERROR S3Client [568873 FilesystemOpExecutorWorker-0] - command=head transactionId=0x7f0a3a71e000 rTxnId=0x7f0a2c7f5f10 status=completed success=N uri=https://-splunk-smartstore.s3-eu-west-1.amazonaws.com/single-site/index_stg/db/8b/35/18~54BB2B8B-8FD4-45FE-92FD-61161215F597/receipt.json statusCode=400 statusDescription="Bad Request" payload="" 07-17-2024 06:12:11.848 +0000 ERROR RetryableClientTransaction [568873 FilesystemOpExecutorWorker-0] - transactionDone() exiting: groupId=(nil) rTxnId=0x7f0a2c7f5f10 transactionId=0x7f0a3a71e000 status=fatal success=N HTTP-statusCode=400 HTTP-statusDescription="Bad Request" retries=0 retry=N no_retry_reason="transaction had fatal error" remainingTxns=0 07-17-2024 06:12:11.848 +0000 ERROR DatabaseDirectoryManager [568873 FilesystemOpExecutorWorker-0] - failed to check if cache_id="bid|index_stg~18~54BB2B8B-8FD4-45FE-92FD-61161215F597|" is stable with CacheManager due to exception=Unable to check If cache_id="bid|index_stg~18~54BB2B8B-8FD4-45FE-92FD-61161215F597|" exists on remote storage or not as the check could not complete due to an error="Bad Request"

vivekr-splunk commented 1 month ago

upcoming splunk enterprise release should fix this issue.

akondur commented 1 week ago

Hi @nfilip21 , the fix for AWS IRSA JWT token expiring has been fixed in splunk 9.3.2. Please try it and let us know if you are running into further issues.

nfilip21 commented 1 day ago

Hi, now is everything working as aspected.

akondur commented 1 day ago

Closing the issue per the comment above.