search

GoogleCloudPlatform / berglas

A tool for managing secrets on Google Cloud
https://cloud.google.com/secret-manager
Apache License 2.0
1.24k stars 96 forks source link

Unable to authenticate against sample application when webhook is set to require authenticated users #109

Closed saranicole closed 4 years ago

saranicole commented 4 years ago

I'm trying to use the mutating webhook with allAuthenticatedUsers instead of leaving it wide open. I'm passing in my authentication with curl but I'm still not seeing the secret decrypted. What am I missing?

I've tried both

curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" http://${IP}

and

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" http://${IP}

but no luck. It would be nice to have some way to debug this at the very least.

sethvargo commented 4 years ago

Hi @saranicole

Could you tell me a little more about the changes you made the webhook? How is it deployed? What config changes were made? What error are you getting with those curl commands?

saranicole commented 4 years ago

Hi @sethvargo , thanks for your time and effort putting this project together and maintaining it.

I followed the Kubernetes instructions exactly but there is a point where it asked to allow unauthenticated invocations. This seems unsafe to me so I said no and instead added the iam policy to allow all authenticated users. Nothing actually fails but when I try to view the output with curl to see the secret substituted in it it has the berglas://secret-name syntax, not the secret content.

Saras-MBP:kubernetes sarajarjoura$ gcloud functions deploy berglas-secrets-webhook \
>   --project ${PROJECT_ID} \
>   --runtime go111 \
>   --entry-point F \
>   --trigger-http
Allow unauthenticated invocations of new function
[berglas-secrets-webhook]? (y/N)?  n

WARNING: Function created with limited-access IAM policy. To enable unauthorized access consider "gcloud alpha functions add-iam-policy-binding berglas-secrets-webhook --member=allUsers --role=roles/cloudfunctions.invoker"
Deploying function (may take a while - up to 2 minutes)...done.
availableMemoryMb: 256
entryPoint: F
httpsTrigger:
  url: https://us-central1-<<PROJECTID>>.cloudfunctions.net/berglas-secrets-webhook
ingressSettings: ALLOW_ALL
labels:
  deployment-tool: cli-gcloud
name: projects/<<PROJECTID>>/locations/us-central1/functions/berglas-secrets-webhook
runtime: go111
serviceAccountEmail: <<YADAYADA>>
status: ACTIVE
timeout: 60s
updateTime: '2020-03-18T19:19:43.899Z'
versionId: '1'
gcloud alpha functions add-iam-policy-binding berglas-secrets-webhook --member=allAuthenticatedUsers --role=roles/cloudfunctions.invoker --project=${PROJECT_ID}

What I see as output is:

Saras-MBP:kubernetes sarajarjoura$ curl -H "Authorization: Bearer $(gcloud auth print-access-token)" $IP
{"API_KEY":"berglas://backend/development/devops/test-secret" ...}

instead of the desired {"API_KEY":"testing" ...}

Does that make sense?

saranicole commented 4 years ago

Actually I have found doc that suggests I have to use an OAuth token, not an access token, to do this. https://cloud.google.com/functions/docs/securing/authenticating. If that is the case it may not be possible to do this just with curl and I will need to set up a quick python script to get the token for me. Let me try that and post my findings.

saranicole commented 4 years ago

Ok I went through the full song and dance of getting an OAuth token and had the same bad result of getting the berglas:// syntax instead of the secret.

For anyone who wants to reproduce what I did, I followed this tutorial https://requests-oauthlib.readthedocs.io/en/latest/examples/google.html after creating an OAuth2 client id and secret.

Repasted in an easily pastable format, modified slightly for Python3:

client_id = $CLIENT_ID
client_secret = $CLIENT_SECRET
redirect_uri = $WEBHOOK_URL #for lack of a better url to use
authorization_base_url = "https://accounts.google.com/o/oauth2/v2/auth"
token_url = "https://www.googleapis.com/oauth2/v4/token"
scope = ["https://www.googleapis.com/auth/cloud-platform"]
from requests_oauthlib import OAuth2Session
google = OAuth2Session(client_id, scope=scope, redirect_uri=redirect_uri)
authorization_url, state = google.authorization_url(authorization_base_url, access_type="offline", prompt="select_account")
print('Please go here and authorize,', authorization_url)  # pause here to authorize through the browser
redirect_response = input('Paste the full redirect URL here:')
google.fetch_token(token_url, client_secret=client_secret,authorization_response=redirect_response)

Fetch token outputted an access token that I then used with my curl command and ... no luck.

We're currently evaluating using Google KMS and berglas since Google Secret Manager is not yet covered under SLA. I'm hoping I can resolve this in order to use this solution.

sethvargo commented 4 years ago

The auth is for Kubernetes to talk to the webhook endpoint, not for you or your app to authenticate. It's pretty safe to have that endpoint publicly accessible because it doesn't actually do anything but accept some YAML, mutate that YAML, and then return the YAML. Kubernetes calls the mutating webhook controller long before your application code executes.

saranicole commented 4 years ago

Ok, the way I was picturing this was that any actor could access the webhook and retrieve secrets if it didn't require authentication. Is there an architecture diagram that illustrates this flow? I want to be well educated as to how this works.

sethvargo commented 4 years ago

Hi @saranicole

The mutating webhook has no permissions to manage secrets. It does not fetch or access secrets of have access to KMS. It mutates the incoming spec (YAML) before it's submitted to the kubelet.

I don't have an architecture diagram, but suppose you had the following:

containers:
- name: "myapp"
  image: "mycompany/myapp"
  entrypoint: "/bin/myapp"
  args: ["-config", "/etc/config"]
  env:
  - name: "FOO"
    value: "berglas://my-bucket/my-secret"

The mutating webhook would modify the definition to:

containers:
- name: "myapp"
  image: "mycompany/myapp"
  entrypoint: "/bin/berglas"
  args: ["exec", "--", "/bin/myapp", "-config", "/etc/config"]
  env:
  - name: "FOO"
    value: "berglas://my-bucket/my-secret"

Specifically it:

At this point, the YAML is passed to the kubelet where it's scheduled. At this point, no secrets have been accessed. Once the container starts, the berglas exec starts (see berglas exec -h), parses the environment, accesses any secrets, updates the environment, and then spawns your subprocess with the updated environment. You can verify this by running kubectl describe pod NAME.

Now, this does omit one point: how does the berglas binary get into your container? Well, that's a bit more complicated and I omitted it above because it clogs the explanation. The mutating webhook actually mutates more than containers and their entrypoints, it:

  1. Creates a shared volume mount
  2. Adds an initContainer with the berglas binary (gcr.io/berglas/berglas)
  3. Moves the berglas binary into the shared volume mount
  4. Adds the volume mount to all containers that need berglas

You'll see these mounts when you run kubectl describe pod NAME.

sethvargo commented 4 years ago

We're currently evaluating using Google KMS and berglas since Google Secret Manager is not yet covered under SLA.

Secret Manager is GA as of yesterday.

saranicole commented 4 years ago

@sethvargo thank you for the architecture description - that clears up my confusion.

Is there a link to an announcement somewhere for Google Secret Manager General Availability? Reason being I need to show them the SLA. Googling is ironically not revealing anything...

saranicole commented 4 years ago

Found it! Not the SLA but the fact that it's generally available is noted on the release page https://cloud.google.com/secret-manager/docs/release-notes

Closing this out as we've exhausted the topic - thanks so much!