search

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
637 stars 116 forks source link

Security and Linux version #282

Closed pferrel closed 5 years ago

pferrel commented 5 years ago
pferrel commented 5 years ago

Oh, and thanks for the tool, saved me loads of time and tedious setup!!!

pferrel commented 5 years ago

I guess my real question is: can Flintrock secure these connections with at least HTTPS if not some kind of authentication?

nchammas commented 5 years ago

is there a discussion group? Do we ask questions via these issues?

Nope. Opening new issues for substantive questions is fine for now.

I seem to have successfully setup a small cluster. But all the URI access is via public ip addresses??? Is this correct? Since all API level access uses ssh that may only be partially bad but all of the GUI seems to be http via IPs or public AWS DNS. Is there setup to use internal IPs and leave the security group with only ssh access? This seems worrisome, no?

I don't know if Spark can be configured to serve its web UIs exclusively over HTTPS. If that's possible I would be open to a PR adjusting Flintrock to enable that configuration by default.

That said, the main way to secure your cluster with Flintrock is via the VPC and security group configuration. Flintrock will respect your existing configuration and only add rules to allow access from your Flintrock client to the Spark cluster.

I seem to have created my cluster with an Ubuntu AMI and user=ubuntu. Are there any problem with this config? It better fits my experience but I could use centos if this is better.

Please see https://github.com/nchammas/flintrock/issues/95#issuecomment-197924531. Flintrock does not support Ubuntu.