search

sakserv / hadoop-mini-clusters

hadoop-mini-clusters provides an easy way to test Hadoop projects directly in your IDE
Apache License 2.0
289 stars 105 forks source link

How can this rep help me? #28

Closed augustoamerico closed 7 years ago

augustoamerico commented 7 years ago

Sorry, this is really not an issue. I tried to get your email from your profile, but no success.

I'm trying to study the hadoop ecosystems to see how I can use it to develop a system but one which I can debug on my localhost.

After spending a few minutes checking the project and cloning the project, I still don't understand how can I use this.

Can you please enlighten me?

timvw commented 7 years ago

I use this repository as a means to quickly set up an environment which i can use to run integration tests on..

Here is an example (using scala) that demonstrates how to have spark streaming jobs (twitter->kafka and kafka->kafka) running on an in memory zookeeper/kafka cluster: https://github.com/timvw/demo-spark-structured-streaming

awesome ;)

sakserv commented 7 years ago

Thanks @timvw for your response!

@augustoamerico - This project is intended for Java developers that would like to write integration tests without needing a development Hadoop cluster or, say, the Hortonworks Sandbox virtual machine. As part of your test code, you can start a "Mini Cluster" without ever having to install Hadoop or start a virtual machine.

You don't say much about your project, but I would suggest looking at the *IntegrationTest.java files for examples on how to use each of the modules. Here is the HDFS example.

augustoamerico commented 7 years ago

Thank you both! I will take a look on that. Regarding my project , I have a web server (java) that provides an app with some analytics and predictive functionalities (R code) , but now the project needs to scale and therefore I'm trying to learn the hadoop ecosystem and the hadoop ways.

I came across this repo and felt that this could be useful somehow, and that's why I asked the question.

Again, thank you guys!

On Mar 2, 2017 12:12, "Shane Kumpf" notifications@github.com wrote:

Thanks @timvw https://github.com/timvw for your response!

@augustoamerico https://github.com/augustoamerico - This project is intended for Java developers that would like to write integration tests without needing a development Hadoop cluster or, say, the Hortonworks Sandbox virtual machine. As part of your test code, you can start a "Mini Cluster" without ever having to install Hadoop or start a virtual machine.

You don't say much about your project, but I would suggest looking at the *IntegrationTest.java files for examples on how to use each of the modules. Here https://github.com/sakserv/hadoop-mini-clusters/blob/master/hadoop-mini-clusters-hdfs/src/test/java/com/github/sakserv/minicluster/impl/HdfsLocalClusterIntegrationTest.java is the HDFS example.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sakserv/hadoop-mini-clusters/issues/28#issuecomment-283637520, or mute the thread https://github.com/notifications/unsubscribe-auth/ASdDOs0aCfwykX159_VtObtsWB-YJdgRks5rhrIqgaJpZM4MNffH .

sakserv commented 7 years ago

No problem, @augustoamerico - I'm going to close this issue for now, but I'll keep an eye on any updates in case you have questions. Thanks for your interest.