search

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
636 stars 116 forks source link

switch all worker to private dns #285

Closed pragnesh closed 4 years ago

pragnesh commented 5 years ago

This PR makes the following changes:

switch all worker to private dns

I tested this PR by... launching cluster where i am seeing slow startup(flintrock needed bend-aid to start cluster). After this change cluster start in first try.

Fixes #157. Fixes #129.

When instance start on aws, aws update public dns host from public ip -> private ip, This create issue with cluster communication. If you switch all worker to private dns name, Cluster does not have any issue in starting which fix band-aid required for starting hdfs cluster and spark cluster.

nchammas commented 4 years ago

Hey @pragnesh! Somehow I missed this PR and am only just seeing it now. I'm sorry about that.

This PR seems to add private VPC support by removing public VPC support. Or are you saying this PR patches Flintrock's behavior even when it's launching into a public VPC?

There is another PR attempting to add private VPC support: #296. If that's what you're interested in, I suggest we close this PR in favor of that one and consolidate our efforts there.

pragnesh commented 4 years ago

Hi @nchammas, This is meant for stability, i am using this change since 1 year and this is fix for issue describe in #157 , #129.

pragnesh commented 4 years ago

This PR has nothing do with private VPC support.

nchammas commented 4 years ago

OK, thanks for clarifying. I will test this out and see if it works for me.

To be clear, do you occasionally run into the issues reported in #129 and #157 when running on a recent version of Flintrock (like 0.9.0 or 1.0.0)? If you haven't tested out a recent version of Flintrock, could you do that and confirm the issue is still there?

pragnesh commented 4 years ago

Hi @nchammas , i was actully facing issue #198, while investing and fixing i found that issue #157 , #129 also same fundamental problem.

I am using this change rebased on flintrock master. I rebased my change regulary after flintrock version change.

pragnesh commented 4 years ago

This PR only change cluster internal communication address. Rest of thing remain as it is.

pragnesh commented 4 years ago

Hi @nchammas, by the way, i have also added alluxio support to flintrock and waiting to raise PR, but unfortunately alluxio related change depend on current PR.

I think alluxio is very useful for spark user.

nchammas commented 4 years ago

Tested this out and it seems to work well. End-to-end tests all pass, too. Nice work @pragnesh.

pragnesh commented 4 years ago

thanks @nchammas

pragnesh commented 4 years ago

@nchammas does flintrock project intrested in Alluxio service ? I am using it sometime with flintrock and if there is interest, I can work on it, here is my Alluxio change

nchammas commented 4 years ago

Thank you for sharing this change. I'd like to keep the services bundled with Flintrock to a bare minimum to keep the maintenance burden low, so I wouldn't want to add Alluxio at this time. I wish I could offer a neat plugin system that lets users contribute new services like Alluxio without having to bake them directly into Flintrock.

Perhaps the one new service I'd consider adding directly into Flintrock is Jupyter (#23) because of how widely used it is, but otherwise I think Flintrock will support only Spark and Hadoop for the foreseeable future.

pragnesh commented 4 years ago

I understand @nchammas, no issue.