Changing from overwriting files to swapping symlinks

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.

7 stars 4 forks source link

Changing from overwriting files to swapping symlinks #124

Closed dterror-zz closed 6 years ago

dterror-zz commented 6 years ago

Instead of overwriting files (more importantly the main jar) per deploy, we'll now upload multiple versions and keep a symlink to the canonical location (current/). This will allow us to safely deploy even if Camus is running 😰😰

dterror-zz commented 6 years ago

it'll require one last coordinated deployment, tho.