search

cloudera-labs / envelope

Build configuration-driven ETL pipelines on Apache Spark
Apache License 2.0
158 stars 89 forks source link

CDH Env requirements. #17

Closed ShawnHouCHN closed 7 years ago

ShawnHouCHN commented 7 years ago

Hi Jared. I think the project is also an excellent tool which can encapsulate lots of coding works behind the scene. Currently, my company has already started using spark 2, kafka 0.10 along with CDH 5.10 without Kudu. In the doc, you wrote about using it aong with kafka 0.9. Will the distinct version (as well as maven dependencies) cause an issue? Besides, do you have plan, at some point, to upgrade the project to spark 2?

jeremybeard commented 7 years ago

I haven't tested Envelope 0.3 against either Spark 2 or Kafka 0.10, but my guess is that the first would break it and the second probably wouldn't.

The good news is that Envelope 0.4 will be based on Spark 2.1 and Kafka 0.10. It is being very actively worked on, and although we don't have a firm release date I am hoping it will be out over the next few weeks.

If you've got any more questions please drop us a line on the Cloudera Community forum: http://community.cloudera.com/t5/Cloudera-Labs/bd-p/ClouderaLabs

Thanks!

jeremybeard commented 6 years ago

@ShawnHouCHN FYI, Envelope 0.4.0 has been released and is based on Spark 2.1 and Kafka 0.10.