Extract file uploading to GCS to a separate method and adding re-tries

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.

7 stars 4 forks source link

Extract file uploading to GCS to a separate method and adding re-tries #150

Closed dterror-zz closed 6 years ago

dterror-zz commented 6 years ago

A lot of times, it seems like the upload fails because of intermittent infrastructure issues, we're implementing re-tries to diminish the number of files we have to manually upload.

@olessia wdyt?

dterror-zz commented 6 years ago

fixed