Network Policy for Google Cloud Storage

jeunii commented 5 years ago

Hello

This is more of a general question. Would you have an example of a Network Policy that allows access to Storage buckets in the same Project ? I have a GKE cluster running that hosts an app that needs access to the storage buckets. The current Network Policy is very restrictive and I cannot access the buckets at all unless I take down the Network Policy. Would you know how I can find the relevant Url OR IPs/Port so that I can whitelist them ?

EDIT:

With my policy in place, if I login to my Pod and run gsutil ls, I get

jenkins@ptbkn:~$ gsutil ls
INFO 0826 18:13:03.702595 retry_util.py] Retrying request, attempt #1...
INFO 0826 18:14:05.556362 retry_util.py] Retrying request, attempt #2...
INFO 0826 18:15:09.331261 retry_util.py] Retrying request, attempt #3...

But the moment I take down the Network Policy, I can see all my buckets.

jenkins@ptbkn:~$ gsutil ls
gs://abc/
gs://xyz/
.
.

Now Im unsure of how can I specify the Network Policy rules. Meaning that Im unaware of what IP ranges / hostnames / Ports should I add to my network policy to whitelist Google Cloud Storage.

chrislovecnm commented 5 years ago

That is a great question. Not certain we do, but let me reach out to someone.

jeunii commented 5 years ago

@chrislovecnm Thanks. I added a bit of logs too to my original post.

thomasfricke commented 5 years ago

Hi, I run in the same problem.

Looks like the address for storage.googleapis.com is extremely unpredictable and volatile.

Would like to see a predictable set of cidrs for this. nslookup presents a single IP, and it is different all the time. We need help for this, otherwise we cannot use buckets in security critical environments

gsutil -D ls

WARNING

*** You are running gsutil with debug output enabled.

*** Be aware that debug output includes authentication credentials.

*** Make sure to remove the value of the Authorization header for

*** each HTTP request printed to the console prior to posting to

*** a public medium such as a forum post or Stack Overflow.

WARNING

gsutil version: 4.44

checksum: e2af2d34cc25b33a166680f83089faee (OK)

boto version: 2.49.0

python version: 2.7.16 (default, May 6 2019, 19:28:45) [GCC 8.3.0]

OS: Linux 4.14.137+

multiprocessing available: False

using cloud sdk: True

pass cloud sdk credentials to gsutil: True

config path(s): /tmp/.config/gcloud/legacy_credentials/xxxxxxxxxxxxxxxxx/.boto

gsutil path: /google-cloud-sdk/bin/gsutil

compiled crcmod: True

installed via package manager: False

editable install: False

Command being run: /google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=zz-muster -D ls

config_file_list: ['/tmp/.config/gcloud/legacy_credentials/xxxxxxxxxxxxxxxxxx/.boto']

config: [('debug', '0'), ('working_dir', '/mnt/pyami'), (u'https_validate_certificates', 'true'), ('debug', '0'), ('working_dir', '/mnt/pyami'), (u'default_project_id', u'zz-muster')]

DEBUG 1015 15:08:05.735010 multiprocess_file_storage.py] Read credential file

INFO 1015 15:08:05.735191 client.py] access_token is expired. Now: 2019-10-15 15:08:05.735172, token_expiry: 2019-10-15 14:33:29

DEBUG 1015 15:08:05.735759 multiprocess_file_storage.py] Read credential file

INFO 1015 15:08:05.736997 base_api.py] Calling method storage.buckets.list with StorageBucketsListRequest: <StorageBucketsListRequest

maxResults: 1000

project: u'zz-muster'

projection: ProjectionValueValuesEnum(noAcl, 1)>

INFO 1015 15:08:05.738046 base_api.py] Making http GET to https://storage.googleapis.com/storage/v1/b?project=zz-muster&fields=nextPageToken%2Citems%2Fid&alt=json&projection=noAcl&maxResults=1000

INFO 1015 15:08:05.738615 base_api.py] Headers: {'accept': 'application/json',

'accept-encoding': 'gzip, deflate',

'content-length': '0',

'user-agent': 'apitools gsutil/4.44 Python/2.7.16 (linux2) google-cloud-sdk/266.0.0 analytics/disabled'}

INFO 1015 15:08:05.738781 base_api.py] Body: (none)

connect: (storage.googleapis.com, 443)

send: 'GET /storage/v1/b?project=zz-muster&fields=nextPageToken%2Citems%2Fid&alt=json&projection=noAcl&maxResults=1000 HTTP/1.1\r\nHost: storage.googleapis.com\r\ncontent-length: 0\r\nauthorization: Bearer ya29.ImWbB2JsJbmHr5l9CMGrVmuNGU9L4gJnWS96sbuCtFPO9pwlJFc97UgkaopYX2_hbQ01hKOemgvAFJKnn8-SlbEHVz_HVpSDPEBUig0jnvOk1WF5dCDx01Af0SkzL8iwLz7uaDR4uw\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: apitools gsutil/4.44 Python/2.7.16 (linux2) google-cloud-sdk/266.0.0 analytics/disabled\r\n\r\n'

reply: 'HTTP/1.1 401 Unauthorized\r\n'

header: X-GUploader-UploadID: AEnB2UqkJG3pZO51HvYEZegDdXMeFPQMGFusODqDX0koCJQp_GmQqzo97q3ihXLIosJeiw6NBl4uOA139BR4UuBZN2ODcqOridYYeAuZONoGktqrzmxVPb0

header: Content-Type: application/json; charset=UTF-8

header: Date: Tue, 15 Oct 2019 15:08:05 GMT

header: Vary: Origin

header: Vary: X-Origin

header: WWW-Authenticate: Bearer realm="https://accounts.google.com/", error=invalid_token

header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate

header: Expires: Mon, 01 Jan 1990 00:00:00 GMT

header: Pragma: no-cache

header: Content-Length: 285

header: Server: UploadServer

INFO 1015 15:08:05.943315 transport.py] Refreshing due to a 401 (attempt 1/2)

DEBUG 1015 15:08:05.944178 multiprocess_file_storage.py] Read credential file

INFO 1015 15:08:05.944321 client.py] access_token is expired. Now: 2019-10-15 15:08:05.944308, token_expiry: 2019-10-15 14:33:29

DEBUG 1015 15:08:05.944707 multiprocess_file_storage.py] Read credential file

INFO 1015 15:08:05.944816 reauth_creds.py] Refreshing access_token

connect: (oauth2.googleapis.com, 443)

send: u'POST /token HTTP/1.1\r\nHost: oauth2.googleapis.com\r\nContent-Length: 174\r\ncontent-type: application/x-www-form-urlencoded\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/0.11.3 (gzip)\r\n\r\nclient_secret=ZmssLNjJy2998hD4CTg2ejr2&grant_type=refresh_token&refresh_token=1%2FxLAid1pO_W2MrKG_kRoxq-Q0sAlcM8TmiRTHuaa3tJg&client_id=32555940559.apps.googleusercontent.com'

reply: 'HTTP/1.1 200 OK\r\n'

header: Content-Type: application/json; charset=utf-8

header: Vary: Origin

header: Vary: X-Origin

header: Vary: Referer

header: Content-Encoding: gzip

header: Date: Tue, 15 Oct 2019 15:08:06 GMT

header: Server: scaffolding on HTTPServer2

header: Cache-Control: private

header: X-XSS-Protection: 0

header: X-Frame-Options: SAMEORIGIN

header: X-Content-Type-Options: nosniff

header: Transfer-Encoding: chunked

DEBUG 1015 15:08:06.164385 multiprocess_file_storage.py] Read credential file

DEBUG 1015 15:08:06.164941 multiprocess_file_storage.py] Wrote credential file /tmp/.gsutil/credstore2.

send: 'GET /storage/v1/b?project=zz-muster&fields=nextPageToken%2Citems%2Fid&alt=json&projection=noAcl&maxResults=1000 HTTP/1.1\r\nHost: storage.googleapis.com\r\ncontent-length: 0\r\nauthorization: Bearer ya29.ImWbB-OmTvbo6W0_xtzW9GRJcsTiHxVNdFzoLAagWpUdREGfEBe-UUwl2M5u_5WzNYIQl5NnzhhOeZUkSxHXOhzx5OsbFX_MarSj9Rn8lT6YhCHLwWT5lg-38MIqLRpIDdisgeKIYw\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: apitools gsutil/4.44 Python/2.7.16 (linux2) google-cloud-sdk/266.0.0 analytics/disabled\r\n\r\n'

reply: 'HTTP/1.1 200 OK\r\n'

header: X-GUploader-UploadID: AEnB2UpbgTsTeSKw-ayr8DAghbX214RHwNrU61XTbwfvyOC1nuUhOiNSXbTaEmU6eMiFS-pKdbIviWxrt4c3k5ftg3_2ZvTSmWkblgyAnzCgsE0wabkl2M0

header: Content-Type: application/json; charset=UTF-8

header: Date: Tue, 15 Oct 2019 15:08:06 GMT

header: Vary: Origin

header: Vary: X-Origin

header: Cache-Control: private, max-age=0, must-revalidate, no-transform

header: Expires: Tue, 15 Oct 2019 15:08:06 GMT

header: Content-Length: 64

header: Server: UploadServer

INFO 1015 15:08:06.419345 base_api.py] Response of type Buckets: <Buckets

items: [<Bucket

acl: []

cors: []

defaultObjectAcl: []

id: u'test_bucket_spon'>]>

gs://test_bucket_spon/

bash-5.0$ nslookup storage.googleapis.com

nslookup: can't resolve '(null)': Name does not resolve

Name: storage.googleapis.com

Address 1: 209.85.234.128 jn-in-f128.1e100.net

Address 2: 2607:f8b0:4001:c12::80

bash-5.0$ gsutil -D ls

WARNING

*** You are running gsutil with debug output enabled.

*** Be aware that debug output includes authentication credentials.

*** Make sure to remove the value of the Authorization header for

*** each HTTP request printed to the console prior to posting to

*** a public medium such as a forum post or Stack Overflow.

WARNING

gsutil version: 4.44

checksum: e2af2d34cc25b33a166680f83089faee (OK)

boto version: 2.49.0

python version: 2.7.16 (default, May 6 2019, 19:28:45) [GCC 8.3.0]

OS: Linux 4.14.137+

multiprocessing available: False

using cloud sdk: True

pass cloud sdk credentials to gsutil: True

config path(s): /tmp/.config/gcloud/legacy_credentials/xxxxxxxxxx/.boto

gsutil path: /google-cloud-sdk/bin/gsutil

compiled crcmod: True

installed via package manager: False

editable install: False

Command being run: /google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=zz-muster -D ls

config_file_list: ['/tmp/.config/gcloud/legacy_credentials/xxxxxxxxxxxxx/.boto']

config: [('debug', '0'), ('working_dir', '/mnt/pyami'), (u'https_validate_certificates', 'true'), ('debug', '0'), ('working_dir', '/mnt/pyami'), (u'default_project_id', u'zz-muster')]

DEBUG 1015 15:09:40.532640 multiprocess_file_storage.py] Read credential file

INFO 1015 15:09:40.534110 base_api.py] Calling method storage.buckets.list with StorageBucketsListRequest: <StorageBucketsListRequest

maxResults: 1000

project: u'zz-muster'

projection: ProjectionValueValuesEnum(noAcl, 1)>

INFO 1015 15:09:40.535085 base_api.py] Making http GET to https://storage.googleapis.com/storage/v1/b?project=zz-muster&fields=nextPageToken%2Citems%2Fid&alt=json&projection=noAcl&maxResults=1000

INFO 1015 15:09:40.535679 base_api.py] Headers: {'accept': 'application/json',

'accept-encoding': 'gzip, deflate',

'content-length': '0',

'user-agent': 'apitools gsutil/4.44 Python/2.7.16 (linux2) google-cloud-sdk/266.0.0 analytics/disabled'}

INFO 1015 15:09:40.535859 base_api.py] Body: (none)

connect: (storage.googleapis.com, 443)

send: 'GET /storage/v1/b?project=zz-muster&fields=nextPageToken%2Citems%2Fid&alt=json&projection=noAcl&maxResults=1000 HTTP/1.1\r\nHost: storage.googleapis.com\r\ncontent-length: 0\r\nauthorization: Bearer ya29.ImWbB-OmTvbo6W0_xtzW9GRJcsTiHxVNdFzoLAagWpUdREGfEBe-UUwl2M5u_5WzNYIQl5NnzhhOeZUkSxHXOhzx5OsbFX_MarSj9Rn8lT6YhCHLwWT5lg-38MIqLRpIDdisgeKIYw\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: apitools gsutil/4.44 Python/2.7.16 (linux2) google-cloud-sdk/266.0.0 analytics/disabled\r\n\r\n'

reply: 'HTTP/1.1 200 OK\r\n'

header: X-GUploader-UploadID: AEnB2UrjQ570DSiLuf8Ozl4_sI8CW0xgxoTrWVpaIhAHrTtjRd_8byViAkRVEBzE4rgtdvla-BiRjapDxR4hZKCgrAQ3Kzr9ZoRpC7oLswNATif_bkcEexE

header: Vary: Origin

header: Vary: X-Origin

header: Content-Type: application/json; charset=UTF-8

header: Expires: Tue, 15 Oct 2019 15:09:41 GMT

header: Date: Tue, 15 Oct 2019 15:09:41 GMT

header: Cache-Control: private, max-age=0, must-revalidate, no-transform

header: Content-Length: 55

header: Server: UploadServer

INFO 1015 15:09:41.017758 base_api.py] Response of type Buckets: <Buckets

items: [<Bucket

acl: []

cors: []

defaultObjectAcl: []

id: u'test_bucket_spon'>]>

gs://test_bucket_spon/

bash-5.0$ nslookup storage.googleapis.com

nslookup: can't resolve '(null)': Name does not resolve

Name: storage.googleapis.com

Address 1: 209.85.234.128 jn-in-f128.1e100.net

Address 2: 2607:f8b0:4001:c12::80

bash-5.0$ nslookup storage.googleapis.com

nslookup: can't resolve '(null)': Name does not resolve

Name: storage.googleapis.com

Address 1: 172.217.212.128

Address 2: 2607:f8b0:4001:c03::80 ie-in-x80.1e100.net

bash-5.0$ nslookup storage.l.googleusercontent.com

nslookup: can't resolve '(null)': Name does not resolve

Name: storage.l.googleusercontent.com

Address 1: 209.85.234.128 jn-in-f128.1e100.net

Address 2: 2607:f8b0:4001:c1c::80 iz-in-x80.1e100.net

bash-5.0$ nslookup storage.l.googleusercontent.com

nslookup: can't resolve '(null)': Name does not resolve

Name: storage.l.googleusercontent.com

Address 1: 209.85.234.128 jn-in-f128.1e100.net

Address 2: 2607:f8b0:4001:c1c::80 iz-in-x80.1e100.net

bash-5.0$ nslookup storage.l.googleusercontent.com

nslookup: can't resolve '(null)': Name does not resolve

Name: storage.l.googleusercontent.com

Address 1: 172.217.214.128

Address 2: 2607:f8b0:4001:c07::80

bash-5.0$

------------- networkpolicy ---------------------

kind: NetworkPolicy

apiVersion: extensions/v1beta1

metadata:

generation: 1

labels:

app: gcloud

name: gcloud

namespace: dev-zz-muster

spec:

podSelector:

matchLabels:

  app: gcloud

policyTypes:

Egress

egress:

- ports:

- port: 443

protocol: TCP
- to:
  
  https://cloud.google.com/vpc/docs/private-access-options#example_2
  - ipBlock:
    
    cidr: 199.36.153.4/30
  Name: dl.googleapis.com
  
  Address 1: 108.177.111.95
  
  Address 2: 172.217.214.95
  
  Address 3: 172.217.212.95
  
  Address 4: 173.194.192.95 jc-in-f95.1e100.net
  
  Address 5: 2607:f8b0:4001:c14::5f
  - ipBlock:
    
    cidr: 172.217.214.95/32
  - ipBlock:
    
    cidr: 172.217.212.95/32
  - ipBlock:
    
    cidr: 108.177.111.95/32
  - ipBlock:
    
    cidr: 173.194.192.95/32
  Name: www.googleapis.com
  
  Address 1: 173.194.74.95 jb-in-f95.1e100.net
  
  Address 2: 172.217.212.95
  
  Address 3: 172.217.214.95
  - ipBlock:
    
    cidr: 173.194.74.95/32
  storage.googleapis.com
  - ipBlock:
    
    cidr: 172.217.212.128/32
  - ipBlock:
    
    cidr: 108.177.112.128/32

mnacharov commented 3 years ago

Here is an explanation of what IP ranges / hostnames / Ports should be added to network policy to whitelist Google Cloud Storage: https://cloud.google.com/storage/docs/troubleshooting#proxy-server

GoogleCloudPlatform / gke-network-policy-demo

Network Policy for Google Cloud Storage #28

- ports:

- port: 443

protocol: TCP

https://cloud.google.com/vpc/docs/private-access-options#example_2

Name: dl.googleapis.com

Address 1: 108.177.111.95

Address 2: 172.217.214.95

Address 3: 172.217.212.95

Address 4: 173.194.192.95 jc-in-f95.1e100.net

Address 5: 2607:f8b0:4001:c14::5f

Name: www.googleapis.com

Address 1: 173.194.74.95 jb-in-f95.1e100.net

Address 2: 172.217.212.95

Address 3: 172.217.214.95

storage.googleapis.com