Create an cloud space for all the data

[jwestw]

Probably a Google bucket with read-only access

Deliverables:

• Create a Google bucket
• change the data source to the bucket- mount the bucket as drive
• test that data loads

[jwestw]

Look at the code in https://github.com/ONSdigital/research-and-development/blob/develop/src/pipeline.py for inspiration on how to create conditional imports based on a config setting (which indicates choice of environment). In our case we may want to do the same re: imports but the main thing to focus on is the creation of paths (local) vs. signed-urls (cloud). Then hopefully most of our existing file read functions will take the signed-url as a "path", just like pandas' dataframe constructor.

We want to avoid lots of if/else statements throughout the code if possible.

[jwestw]

May want to write to bucket which if we do we need to have two things:

• function to write to bucket in the GcpBucket class
• permissions for that service account to write (may already exist)

[paigeh-fsa]

Going to download files and host on OneDrive.