Clarify compatibility with Google Kubernetes

pbundac commented 1 year ago

Hello team!

I am attempting to install the core dump handler on a GKE cluster (the nodes are using 1.24.7-gke.900). The [Kubernetes Service Compatibility Matrix] (https://github.com/IBM/core-dump-handler#kubernetes-service-compatibility) shows that the core dump handler is functional on GKE-cos_containerd, but I don't believe Google Storage Buckets are S3 compatible. I had a handful of questions regarding the implementation:

Is there a configuration we can use to send core dumps to a Google Storage Bucket?
If not, what is the recommended S3 storage solution to use in a GKE implementation? Would AWS be preferred, or would something local like minio be feasible?
With AWS, is there a way to use temporary credentials instead of using the permanent access key and secret access key?

No9 commented 1 year ago

Hey @pbundac I've put the replies in-line

Is there a configuration we can use to send core dumps to a Google Storage Bucket? The extraEnvVars capability was added for this scenario. https://github.com/IBM/core-dump-handler/blob/8576047c6ba1b582e0876bc504e2bb6f64ed83be/FAQ.md#how-do-i-use-the-custom-endpoint Also check the s3 rust sample to see how this maps https://github.com/durch/rust-s3/blob/master/examples/gcs-tokio.rs#L9 GCP has compatibility with the S3 API using google cloud storage but there are configuration differences - this article seems to outline it quite well. https://vamsiramakrishnan.medium.com/a-study-on-using-google-cloud-storage-with-the-s3-compatibility-api-324d31b8dfeb
If not, what is the recommended S3 storage solution to use in a GKE implementation? Would AWS be preferred, or would something local like minio be feasible? If option 1 doesn't work for your scenario I would of course recommend IBM Cloud first :smile: It's very cost effective for object storage. But AWS or Minio will work you might want to consider using minio as a frontend for GCP. https://blog.min.io/minio-object-storage-running-on-the-google-cloud-platform/
With AWS, is there a way to use temporary credentials instead of using the permanent access key and secret access key? This will likely require substantial additional modifications to the agent as it uses the key found in the environment of the service which means a restart would be needed which is far from ideal. I would consider a PR for this but it's not something we are looking to build right now.

pbundac commented 1 year ago

Hi @No9, thank you for your insights!

I'm continuing to tinker with the extraEnvVars and the GCP compatibility configurations, but have hit an error regarding our generated keys:

[2023-01-09T18:02:10Z INFO  core_dump_agent] INotify Initialised...
[2023-01-09T18:02:10Z INFO  core_dump_agent] INotify watching : /home/kubernetes/cores
[2023-01-09T18:17:54Z INFO  core_dump_agent] Setting s3 endpoint location to: https://storage.googleapis.com
[2023-01-09T18:17:54Z INFO  core_dump_agent] Uploading: /home/kubernetes/cores/7e6de3aa-836d-45ca-af78-dd46f04eecbf-dump-1673288274-segfaulter-segfaulter-1-4.zip
[2023-01-09T18:17:54Z INFO  core_dump_agent] zip size is 29579
[2023-01-09T18:17:54Z ERROR core_dump_agent] Upload Failed Got HTTP 403 with content '<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>AWS4-HMAC-SHA256
    20230109T181754Z
    20230109/us-central1/s3/aws4_request
    57c6395829a16f3f0bbe4167512d0451f1713e93eae038d835d5e3241bd5d6b1</StringToSign><CanonicalRequest>PUT
    /test-bucket-us-central1/7e6de3aa-836d-45ca-af78-dd46f04eecbf-dump-1673288274-segfaulter-segfaulter-1-4.zip

    content-length:29579
    content-md5:ZoxCFOzt3aYbLR8IWX1W9Q==
    content-type:application/octet-stream
    host:storage.googleapis.com
    x-amz-content-sha256:7ad3253b0ac1e62cfbb90e20f8a72c386ebdc76b096925ab3df5406690ab414a
    x-amz-date:20230109T181754Z

    content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date
    7ad3253b0ac1e62cfbb90e20f8a72c386ebdc76b096925ab3df5406690ab414a</CanonicalRequest></Error>'

Right now I'm not totally sure if the issue is with the key that GCP generated or if there's an issue with accessing the bucket. Currently trying to debug using a few online links, but no luck so far: https://stackoverflow.com/questions/30518899/amazon-s3-how-to-fix-the-request-signature-we-calculated-does-not-match-the-s

No9 commented 1 year ago

Hi @pbundac The following config works fine for me:

  manageStoreSecret: true
  s3AccessKey: MY_USER_ACCESS_KEY
  s3Secret: MY_USER_SECRET
  s3BucketName: test-bucket-core-dump
  s3Region: europe-west2
  extraEnvVars: |
    - name: S3_ENDPOINT
      value: https://storage.googleapis.com

I created a single region bucket and used the "Create Key" button at the bottom of the Bucket > Settings > INTEROPERABILITY tab on GCP and it worked as expected. Not sure what is causing your issue but I would check your bucket permissions and confirm that the region config is correct.

No9 commented 1 year ago

Hey @pbundac Closing this as it's working as expected and I think the issue is outside of this project. Please feel free to reopen or start another issue if you come across something else.

IBM / core-dump-handler

Clarify compatibility with Google Kubernetes #125