Try 2 of camus at shopify (long live albert?, I wish we called it king then I could say long live the king)
The following are changes that are included in this PR:
Camus shopify module
- Builds a shaded jar for use in production
- Shopify data partitioner to make the transition process to albert easy peasy
Circle and Shipit support
Compiled with CDH libraries to run on shopify cluster
What we do not support yet:
Camus sweeper to get around small file problem (Upstream and my branch have significantly diverged and they are refactoring it too. I'm in touch with maintainers to get most of my changes in). We will support this soon enough.
Please review
@drdee @airhorns
cc @Shopify/data-engineers
cc @wvanbergen @snormore @eapache
airhorns
commented
9 years ago
Try 2 of camus at shopify (long live albert?, I wish we called it king then I could say long live the king)
The following are changes that are included in this PR:
What we do not support yet: Camus sweeper to get around small file problem (Upstream and my branch have significantly diverged and they are refactoring it too. I'm in touch with maintainers to get most of my changes in). We will support this soon enough.
Please review @drdee @airhorns cc @Shopify/data-engineers cc @wvanbergen @snormore @eapache