Closed pbundac closed 1 year ago
Hey @pbundac I've put the replies in-line
Is there a configuration we can use to send core dumps to a Google Storage Bucket? The extraEnvVars capability was added for this scenario. https://github.com/IBM/core-dump-handler/blob/8576047c6ba1b582e0876bc504e2bb6f64ed83be/FAQ.md#how-do-i-use-the-custom-endpoint Also check the s3 rust sample to see how this maps https://github.com/durch/rust-s3/blob/master/examples/gcs-tokio.rs#L9 GCP has compatibility with the S3 API using google cloud storage but there are configuration differences - this article seems to outline it quite well. https://vamsiramakrishnan.medium.com/a-study-on-using-google-cloud-storage-with-the-s3-compatibility-api-324d31b8dfeb
If not, what is the recommended S3 storage solution to use in a GKE implementation? Would AWS be preferred, or would something local like minio be feasible? If option 1 doesn't work for your scenario I would of course recommend IBM Cloud first :smile: It's very cost effective for object storage. But AWS or Minio will work you might want to consider using minio as a frontend for GCP. https://blog.min.io/minio-object-storage-running-on-the-google-cloud-platform/
With AWS, is there a way to use temporary credentials instead of using the permanent access key and secret access key? This will likely require substantial additional modifications to the agent as it uses the key found in the environment of the service which means a restart would be needed which is far from ideal. I would consider a PR for this but it's not something we are looking to build right now.
Hi @No9, thank you for your insights!
I'm continuing to tinker with the extraEnvVars and the GCP compatibility configurations, but have hit an error regarding our generated keys:
[2023-01-09T18:02:10Z INFO core_dump_agent] INotify Initialised...
[2023-01-09T18:02:10Z INFO core_dump_agent] INotify watching : /home/kubernetes/cores
[2023-01-09T18:17:54Z INFO core_dump_agent] Setting s3 endpoint location to: https://storage.googleapis.com
[2023-01-09T18:17:54Z INFO core_dump_agent] Uploading: /home/kubernetes/cores/7e6de3aa-836d-45ca-af78-dd46f04eecbf-dump-1673288274-segfaulter-segfaulter-1-4.zip
[2023-01-09T18:17:54Z INFO core_dump_agent] zip size is 29579
[2023-01-09T18:17:54Z ERROR core_dump_agent] Upload Failed Got HTTP 403 with content '<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>AWS4-HMAC-SHA256
20230109T181754Z
20230109/us-central1/s3/aws4_request
57c6395829a16f3f0bbe4167512d0451f1713e93eae038d835d5e3241bd5d6b1</StringToSign><CanonicalRequest>PUT
/test-bucket-us-central1/7e6de3aa-836d-45ca-af78-dd46f04eecbf-dump-1673288274-segfaulter-segfaulter-1-4.zip
content-length:29579
content-md5:ZoxCFOzt3aYbLR8IWX1W9Q==
content-type:application/octet-stream
host:storage.googleapis.com
x-amz-content-sha256:7ad3253b0ac1e62cfbb90e20f8a72c386ebdc76b096925ab3df5406690ab414a
x-amz-date:20230109T181754Z
content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date
7ad3253b0ac1e62cfbb90e20f8a72c386ebdc76b096925ab3df5406690ab414a</CanonicalRequest></Error>'
Right now I'm not totally sure if the issue is with the key that GCP generated or if there's an issue with accessing the bucket. Currently trying to debug using a few online links, but no luck so far: https://stackoverflow.com/questions/30518899/amazon-s3-how-to-fix-the-request-signature-we-calculated-does-not-match-the-s
Hi @pbundac The following config works fine for me:
manageStoreSecret: true
s3AccessKey: MY_USER_ACCESS_KEY
s3Secret: MY_USER_SECRET
s3BucketName: test-bucket-core-dump
s3Region: europe-west2
extraEnvVars: |
- name: S3_ENDPOINT
value: https://storage.googleapis.com
I created a single region bucket and used the "Create Key" button at the bottom of the Bucket > Settings > INTEROPERABILITY tab on GCP and it worked as expected. Not sure what is causing your issue but I would check your bucket permissions and confirm that the region config is correct.
Hey @pbundac Closing this as it's working as expected and I think the issue is outside of this project. Please feel free to reopen or start another issue if you come across something else.
Hello team!
I am attempting to install the core dump handler on a GKE cluster (the nodes are using 1.24.7-gke.900). The [Kubernetes Service Compatibility Matrix] (https://github.com/IBM/core-dump-handler#kubernetes-service-compatibility) shows that the core dump handler is functional on GKE-cos_containerd, but I don't believe Google Storage Buckets are S3 compatible. I had a handful of questions regarding the implementation: