Script to upload Camus dropped dirs to gcs.

[olessia]

Create a script that goes over the directories dropped by Camus and uploads data in whitelisted topics to gcs.

• Utilizes regular copy for small number of files (< 8) and distributed copy for larger number of files.
• The whitelist is currently limited to topics used by refinery.
• Updates folders if new data is dropped, but does not overwrite existing files.
• Checks if all files if hdfs were uploaded and fails otherwise.
• Can be configured to look at any number of most recent Camus executions (currently set to 1, the latest one)

[dterror-zz]

We'll have to set-up the upload of /u/apps/{target}/current/upload_to_gcs.py and /u/apps/{target}/current/upload_topics_to_gcs too

(https://github.com/Shopify/camus/blob/master/camus-shopify/script/upload#L15-L18)

[olessia]

Upload should be done already: https://github.com/Shopify/camus/pull/114/files#diff-044216a1e3439de25ef5b93559cbfc15 🤞