Cannot access test data following instruction on Readme

[miaoneng]

I am following readme to get test data, but after following instruction as

First, make sure you have configured the authentication with user credientials and configured Docker with the following commands:

gcloud auth login gcloud auth configure-docker

Next, you can download the test data for the dynamical core and the physics tests.

cd fv3core make get_test_data

TEST_DATA_ROOT=/scratch/git/pace/fv3core/test_data/ TARGET=dycore EXPERIMENT=c12_6ranks_standard make -C .. get_test_data
make[1]: Entering directory '/scratch/git/pace'
if [ -z "" ] ; then \
    if [ ! -f "/scratch/git/pace/fv3core/test_data//8.1.1/c12_6ranks_standard/dycore/input.nml" ] || \
    [ "$(gsutil cp gs://vcm-fv3gfs-serialized-regression-data/8.1.1/c12_6ranks_standard/dycore/md5sums.txt -)" != "$(cat /scratch/git/pace/fv3core/test_data//8.1.1/c12_6ranks_standard/dycore/md5sums.txt)" ] ; then \
        rm -rf /scratch/git/pace/fv3core/test_data//8.1.1/c12_6ranks_standard/dycore ; \
        make sync_test_data ; \
        make unpack_test_data ; \
    fi ; \
else \
    make sync_test_data_from_ftp ; \
    make unpack_test_data ; \
fi
make[2]: Entering directory '/scratch/git/pace'
mkdir -p /scratch/git/pace/fv3core/test_data//8.1.1/c12_6ranks_standard/dycore && gsutil -m rsync -r gs://vcm-fv3gfs-serialized-regression-data/8.1.1/c12_6ranks_standard/dycore/ /scratch/git/pace/fv3core/test_data//8.1.1/c12_6ranks_standard/dycore

WARNING: gsutil rsync uses hashes when modification time is not available at
both the source and destination. Your crcmod installation isn't using the
module's C extension, so checksumming will run very slowly. If this is your
first rsync since updating gsutil, this rsync can take significantly longer than
usual. For help installing the extension, please see "gsutil help crcmod".

Building synchronization state...
Caught non-retryable exception while listing gs://vcm-fv3gfs-serialized-regression-data/8.1.1/c12_6ranks_standard/dycore/: AccessDeniedException: 403 xxxxxxxx@gmail.com does not have storage.objects.list access to the Google Cloud Storage bucket.
CommandException: Caught non-retryable exception - aborting rsync
make[2]: *** [Makefile.data_download:19: sync_test_data] Error 1
make[2]: Leaving directory '/scratch/git/pace'
make[2]: Entering directory '/scratch/git/pace'
if [ -f /scratch/git/pace/fv3core/test_data//8.1.1/c12_6ranks_standard/dycore/dat_files.tar.gz ]; then \
    cd /scratch/git/pace/fv3core/test_data//8.1.1/c12_6ranks_standard/dycore && tar -xf dat_files.tar.gz && \
    rm dat_files.tar.gz; \
fi
make[2]: Leaving directory '/scratch/git/pace'
make[1]: Leaving directory '/scratch/git/pace'

[elynnwu]

I think we still have some permission issues that prevent external users to download from the google bucket, we will try and get that sorted out. As a workaround, we recommend pulling the data from the ftp site, you can do this by:

make USE_FTP=yes get_test_data

[jdahm]

Hey @miaoneng, Thanks for filing the issue! At one point that bucket was public with requester pays, but it looks like a some point it flipped back to private. I set it back how it was, so you should be able to access it now. I'll close this for now, but let me know if you still have issues.

[jdahm]

We had to revert this because some people on the team had trouble accessing the data. I'll keep this open as a reminder to fix it soon.