issues
search
Shopify
/
camus
Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.
7
stars
4
forks
source link
Better gcs upload script.
#120
Closed
olessia
closed
6 years ago
olessia
commented
6 years ago
Post run time, number of directories uploaded and exit status to Datadog
Make script more resilient by ignoring stderr output that messes up the directory list every so often
Other improvements:
Upload a specific execution dir
When uploading multiple execution dirs, upload duplicate directories only once
Check the last two executions, but only upload if not marked as uploaded