Closed dm-tran closed 8 years ago
Hi @dm-tran and thank you for the kind words. They are appreciated.
I feel a bit hesitant to add support for other services that are not strictly Spark-related. But coming from spark-ec2 I totally understand why you want Ganglia in Flintrock too.
One thing that Flintrock offers over spark-ec2 is the ability to copy files and run commands against the cluster in parallel. This should make it relatively easy to piece together your own post-launch scripts for setting up other things you need, like Ganglia.
For example:
flintrock launch dm-tran
flintrock run-command dm-tran 'sudo yum install -y ganglia'
Or alternately:
flintrock launch dm-tran
flintrock copy-file dm-tran ganglia-setup.sh
flintrock run-command dm-tran 'chmod u+x ganglia-setup.sh'
flintrock run-command dm-tran './ganglia-setup.sh'
Would an approach like that meet your needs?
Another thing you can do is create your own custom AMIs that have Ganglia pre-installed from the default Amazon Linux AMIs, and use those to launch your Flintrock clusters.
I am open to considering "promoting" certain services into first-class Flintrock features in the future, but I think the bar will be pretty high. If something proves to be extremely valuable in conjunction with Spark and is widely used by Flintrock users, I would consider it.
Thanks @nchammas for the answer. An approach using copy-file
and run-command
(and optionally custom AMIs) would indeed meet our needs, we'll try that.
I'm closing the issue.
I would love to have Ganglia as additional support. Since it is so important to understand what the cluster is doing and to figure where the problem lies.
Thanks for creating flintrock and making it open source, it's really fast.
My team and I are moving from spark-ec2 to flintrock. One thing that is missing is the support for ganglia, to monitor Spark clusters.
A user should be able to specify the version and (de)activate ganglia's installation, e.g. :