nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
637 stars 116 forks source link

"python" binary does not exist on AWS images? #299

Closed anku94 closed 4 years ago

anku94 commented 4 years ago

Most AWS AMIs, by default, don't have a binary named python anymore, causing the tool to crash.

Shouldn't the binary for setup-ephemeral-storage.py be renamed to python3?

nchammas commented 4 years ago

I test Flintrock against the latest Amazon Linux 2 AMIs, which have python available on the path. The current default AMI on master is ami-00b882ac5193044e4, which shows the following:

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
38 package(s) needed for security, out of 77 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-31-7-123 ~]$ which python
/usr/bin/python
[ec2-user@ip-172-31-7-123 ~]$ python --version
Python 2.7.16
[ec2-user@ip-172-31-7-123 ~]$ which python3
/usr/bin/which: no python3 in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin)

What AMIs are you using?

anku94 commented 4 years ago

I've tried Ubuntu 18.04 (ami-04b9e92b5572fa0d1) and Ubuntu 16.04 (ami-04763b3055de4860b).

I agree that the Amazon Linux AMI does have the python binary, but it's also based on an extremely old/stable kernel (4.4, released in 2016) and the packages are also outdated.

nchammas commented 4 years ago

Does Flintrock work with Ubuntu apart from that issue? Last I was aware (#95), there are other binaries that Flintrock expects but which are not available on Ubuntu. You will have an easier time using Flintrock with the Red Hat family of Linux distributions, which include CentOS and Amazon Linux.

anku94 commented 4 years ago

You're right - for one, flintrock uses yum to set up the AMIs, which wouldn't work on the Ubuntu images anyway. Seems like supporting more AMIs is more complicated than I assumed!

I'll make Amazon Linux work and close this issue for now. Thanks for the dev work!