issues
search
Shopify
/
camus
Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.
7
stars
4
forks
source link
Improvements to gcs upload.
#127
Closed
olessia
closed
6 years ago
olessia
commented
6 years ago
Post run time, number of directories uploaded and exit status to Datadog
Make script more resilient by ignoring stderr output that messes up the directory list every so often
Other improvements:
Option to upload a specific execution dir
When uploading multiple execution dirs, upload duplicate directories only once
Check the last three executions, but only upload if not marked as uploaded
Other improvements: