grafana / mimir

Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
https://grafana.com/oss/mimir/
GNU Affero General Public License v3.0
3.86k stars 470 forks source link

mimirtool bucket-validation not working with GCS service account json #8570

Open Ivaka opened 4 days ago

Ivaka commented 4 days ago

Describe the bug

When running mimirtool bucket-validation for data in GCS bucket, the command reports the following error:

mimirtool: error: failed to create the bucket client: failed to create credentials from JSON: unexpected end of JSON input, try --help

To Reproduce

Steps to reproduce the behavior:

  1. Mimirtool, version 2.12.0 (branch: HEAD, revision: c7aab9e03)
  2. Try to run mimirtool bucket-validation
mimirtool bucket-validation --bucket-config='-backend=gcs -gcs.bucket-name=mimir_experiments -gcs.service-account={"type":"service_account","project_id":"fake-project-id-12345","private_key_id":"some_id","private_key":"-----BEGIN PRIVATE KEY-----A....-----END PRIVATE KEY-----","client_email":"mimir-experiments@fake-project-id-12345.iam.gserviceaccount.com","client_id":"1234","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":"https://www.googleapis.com/robot/v1/metadata/x509/mimir-experimentsfake-project-id-12345.iam.gserviceaccount.com","universe_domain":"googleapis.com"}'

Expected behavior

bucket-validation is ran

Additional Context

It seems that the issue stems from the way the bucket-config is passed and parsed - https://github.com/grafana/mimir/blob/main/pkg/mimirtool/commands/bucket_validation.go#L185 Because of the spaces in the private_key the gcs.service-account is not parsed correctly. The only way I could figure to fix this is to pass the service account as base64 encoded string and decode it in the code. Here is an implementation in my fork - https://github.com/Ivaka/mimir/commit/f09edae9d7be6eb7fe00cf31557eaa923d8adf3b

aknuds1 commented 3 days ago

I could reproduce the problem, and agree that it's due to -bucket-config's value being split on spaces. Maybe you could provide a PR for fixing it?

Ivaka commented 3 days ago

I could reproduce the problem, and agree that it's due to -bucket-config's value being split on spaces. Maybe you could provide a PR for fixing it?

Of course. However, the only feasible way I managed to fix it without introducing complexity into the parsing logic, is to expect the -gcs.service-account to be passed as base64 encoded string. This might "break" a bit UX, as all other places where -gcs.service-account is passed it is expected as JSON. If this solution is acceptable, then I can open a PR and move forward with it.