switching gsutil copy to gsutil rsync

[shraykay]

when I attempt to use the bucket sync method, I noticed that the directory is empty when it attempts to clone it into the volume mount. I was able to replicate this locally:

# check bucket for items
$ gsutil ls gs://forseti-server-obfusc/
gs://forseti-server-obfusc/configs/
gs://forseti-server-obfusc/inventory_summary/
gs://forseti-server-obfusc/policy-library/
gs://forseti-server-obfusc/rules/
gs://forseti-server-obfusc/scanner_violations/

# attempt recursive copy
$ gsutil cp -r gs://forseti-server-obfusc/policy-library/ .

# check local for copied items
$ tree
.

0 directories, 0 files

$ ls -altr
total 0
drwxr-xr-x  15 shray  staff  480 May  6 15:16 ..
drwxr-xr-x   2 shray  staff   64 May  6 15:18 .

# run rsync instead of cp

$ gsutil rsync -r gs://forseti-server-obfusc/policy-library/ .
Building synchronization state...
Starting synchronization...
Copying gs://forseti-server-obfusc/policy-library/...
Skipping attempt to download to filename ending with slash (./). This
typically happens when using gsutil to download from a subdirectory
created by the Cloud Console (https://cloud.google.com/console)
Copying gs://forseti-server-obfusc/policy-library/policies/constraints/constraints/enforce_label.yaml...
Copying gs://forseti-server-obfusc/policy-library/policies/constraints/constraints/gcp_enforce_labels_v1.yaml...
Copying gs://forseti-server-obfusc/policy-library/policies/templates/...
Skipping attempt to download to filename ending with slash
(./policies/templates/). This typically happens when using gsutil to
download from a subdirectory created by the Cloud Console
(https://cloud.google.com/console)

==> NOTE: You are performing a sequence of gsutil operations that may
run significantly faster if you instead use gsutil -m rsync ... Please
see the -m section under "gsutil help options" for further information
about when gsutil -m can be advantageous.

Copying gs://forseti-server-obfusc/policy-library/policies/templates/constraints.rego...
Copying gs://forseti-server-obfusc/policy-library/policies/templates/util.rego...
\ [4 files][ 14.2 KiB/ 14.2 KiB]
Operation completed over 6 objects/14.2 KiB.

# confirm items exist
$ tree
.
└── policies
    ├── constraints
    │   └── constraints
    │       ├── enforce_label.yaml
    │       └── gcp_enforce_labels_v1.yaml
    └── templates
        ├── constraints.rego
        └── util.rego

4 directories, 4 files

$ ls -altr
total 0
drwxr-xr-x  15 shray  staff  480 May  6 15:16 ..
drwxr-xr-x   3 shray  staff   96 May  6 15:24 .
drwxr-xr-x   4 shray  staff  128 May  6 15:24 policies

[gkowalski-google]

Hi @shraykay, sorry for the delay. Looks like you have addressed an issue, just curious if you would also like to include the -m option to gsutil and the -d option to rysnc.

Including -m will help the performance of gsutil. Maybe not a big deal since we are just coping constraints, these shouldn't be large in size.

Including -d will mirror source and destination.

If you are happy as is, then I can approve and merge.

[shraykay]

@gkowalski-google thanks, added