Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.
7 stars 4 forks source link

CDH 4.7 #25

Closed datwright closed 9 years ago

datwright commented 9 years ago

To be deployed after tomorrow's CDH 4.7.x upgrade

Upgrade to Hadoop-:

Client 4.7.1: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-client/2.0.0-mr1-cdh4.7.1/

Core 4.7.1: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/2.0.0-mr1-cdh4.7.1/

Common 4.7.1: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-common/2.0.0-cdh4.7.1/

Not really sure how to verify this works, but I did successfully mvn install and mvn clean package on my machine.

@yagnik @airhorns

airhorns commented 9 years ago

Looks good to me. There aren't subordinate pom files inside any of the component projects eh?

datwright commented 9 years ago

Maven does "Dependency mediation" similar to Bundler, so I think it's taken care of

this determines what version of a dependency will be used when multiple versions of an artifact are encountered. Currently, Maven 2.0 only supports using the "nearest definition" which means that it will use the version of the closest dependency to your project in the tree of dependencies.

You can always guarantee a version by declaring it explicitly in your project's POM. (which we did)

datwright commented 9 years ago

Oh sorry, I didn't realize how the project was set up with all the subprojects. In any case, there's no other references to CDH in those other POMs.

yagnik commented 9 years ago

Lgtm, testing it is really running it against the cluster so go ahead and deploy.

datwright commented 9 years ago

Rolled back to 4.7.0 since cloudera doesn't have a 4.7.1 version for the rest of the upgrade

datwright commented 9 years ago

mvn install and mvn clean package, again, both worked :+1:

datwright commented 9 years ago

ping @airhorns @udnay

airhorns commented 9 years ago

:shipit: