search

amplab / spark-ec2

Scripts used to setup a Spark cluster on EC2
Apache License 2.0
392 stars 299 forks source link

Is there a handy Hadoop cluster running? #70

Open biolearning opened 7 years ago

biolearning commented 7 years ago

With Spark cluster launched and started with ec2 script, is there the Hadoop cluster ready to go, or just an easy command to start it? The reason I am asking for this is - I would like to enable log and save that in hadoop FS which is accessible for Spark workers?

shivaram commented 7 years ago

Yes - there should be a HDFS installation on /root/ephemeral-hdfs

biolearning commented 7 years ago

Thanks for your reply, @shivaram, is it started by default? I guess not, then what is the command I can start the Hadoop cluster?

shivaram commented 7 years ago

It should be started by default. If you want to reconfigure / restart it you can read the Hadoop docs for the appropriate version.