Can't connect localhost to EC2 instance

[timmoti]

I can't seem to run the ec2_create_tunnel.sh script successfully. Here is the output with the ec2.sh script:

`Logging operations to '/tmp/ec2.sh.log' ... tee: illegal option -- / usage: tee [-ai] [file ...] Welcome to Agile Data Science 2.0 :)

I will launch an r3.xlarge instance in the default VPC for you, using a key and security group we will create.

The utility 'jq' is required for this script to detect the hostname of your ec2 instance ... Detecting 'jq' ... 'jq' was detected ... Testing for security group 'agile_data_science' ... parse error: Invalid numeric literal at line 1, column 15 Security group 'agile_data_science' not present ... Creating security group 'agile_data_science' ... sg-351cf94c

Detecting external IP address ... Authorizing port 22 to your external IP (132.147.82.83) in security group 'agile_data_science' ...

Testing for existence of keypair 'agile_data_science' and key 'agile_data_science.pem' ... parse error: Invalid numeric literal at line 1, column 9 Key pair 'agile_data_science' not found ... Generating keypair called 'agile_data_science' ... parse error: Invalid numeric literal at line 1, column 3 Changing permissions of 'agile_data_science.pem' to 0600 ...

Detecting the default region... The default region is 'ap-southeast-1' Determining the image ID to use according to region... The image for region 'ap-southeast-1' is 'ami-81a5f0e2' ...

Initializing EBS optimized r3.xlarge EC2 instance in region 'ap-southeast-1' with security group 'agile_data_science', key name 'agile_data_science' and image id 'ami-81a5f0e2' using the script 'aws/ec2_bootstrap.sh' jq: error (at :1): Cannot index number with string "ReservationId" parse error: Invalid numeric literal at line 2, column 0 Got reservation ID '' ...

Sleeping 10 seconds before inquiring to get the public hostname of the instance we just created ... ... Awake!

Using the reservation ID to get the public hostname ... parse error: Invalid numeric literal at line 1, column 13 The public hostname of the instance we just created is '' ... Writing hostname to '.ec2_hostname' ...

Now we will tag this ec2 instance and name it 'agile_data_science_ec2' ... parse error: Invalid numeric literal at line 1, column 13

An error occurred (MissingParameter) when calling the CreateTags operation: The request must contain the parameter resourceIdSet

After a few minutes (for it to initialize), you may ssh to this machine via the command in red: ssh -i ./agile_data_science.pem ubuntu@ Note: only your IP of '132.147.82.83' is authorized to connect to this machine.

NOTE: IT WILL TAKE SEVERAL MINUTES FOR THIS MACHINE TO INITIALIZE. PLEASE WAIT FIVE MINUTES BEFORE LOGGING IN.

Note: if you ssh to this machine after a few minutes and there is no software in $HOME, please wait a few minutes for the install to finish.

Once you ssh in, the exercise code is in the Agile_Data_Code_2 directory! Run all files from this directory, with the exception of the web applications, which you will run from ex. ch08/web

Note: after a few minutes, now you will need to run ./ec2_create_tunnel.sh to forward ports 5000 and 8888 on the ec2 instance to your local ports 5000 and 8888. This way you can run the example web applications on the ec2 instance and browse them at http://localhost:5000 and you can view Jupyter notebooks at http://localhost:8888 If you tire of the ssh tunnel port forwarding, you may end these connections by executing ./ec2_kill_tunnel.sh

---

Thanks for trying Agile Data Science 2.0!

If you have ANY problems, please file an issue on Github at https://github.com/rjurney/Agile_Data_Code_2/issues and I will resolve them.

If you need help creating your own applications, or with on-site or video training... Check out Data Syndrome at http://datasyndrome.com

Enjoy! Russell Jurney <@rjurney> russell.jurney@gmail.com http://linkedin.com/in/russelljurney`

And here's the output after running the create_tunnel script:

'No hostname detected in '.ec2_hostname' :( Exiting!

The command to create an ssh tunnel to port 5000 of your ec2 instance is: ssh -R 5000:localhost:5000 ubuntu@'

I noticed several parsing errors while running the ec2.sh script but I'm not sure what they mean.

Any help appreciated!

[venclov]

I have similar problem. After ./.ec2.sh i get this: log.txt and after create_tunnel script: 'No hostname detected in '.ec2_hostname' :( Exiting!

[timmoti]

Is the issue with line 35 and 61 in the ec2.sh script?

[35]GROUP_NAME_FILTER=`aws ec2 describe-security-groups | jq '.SecurityGroups[] | select(.GroupName == "agile_data_science") | length'`

[61]KEY_PAIR_RESULTS=`aws ec2 describe-key-pairs | jq '.KeyPairs[] | select(.KeyName == "agile_data_science") | length'`

[IronistM]

I just launched an ec2 image, the problem I had was that running it from the root of the Agile_Data_Code folder the ssh command should be

ssh -i "agile_data_science.pem" ubuntu@

[rjurney]

Ok, sorry for the delay. Let me try this one after the other...

[rjurney]

@timmoti Please paste the output of this command: aws ec2 describe-key-pairs

For some reason jq can't parse this output.

Before you do that run 'aws configure' and make sure the default output format is 'json'. Then run ./ec2.sh again and THEN do the above and report back on all of it.

[rjurney]

@venclov Your log.txt looks fine. Please run this: 'cat .ec2_hostname'

[rjurney]

@IronistM Please paste/attach the entire log output of ./ec2.sh

[timmoti]

@rjurney

Thanks for replying. Changing the default output format to json worked to get the jq parsing correctly (was text previously).

here's the output after entering aws ec2 describe-key-pairs:

{ "KeyPairs": [ { "KeyName": "agile_data_science", "KeyFingerprint": "a0:db:14:e1:c8:7e:22:d4:0a:31:c0:f9:b5:3c:cf:c3:6b:1e:76:5d" } ] }

However, I seem to have some issue with creating tunnels:

`timmoti$ ./ec2_create_tunnel.sh

This script will create an ssh tunnel between the ec2 host's port 5000 and your local port 5000. This will enable you to view web applications you run from ex. Agile_Data_Code_2/ch08/web to be viewed at http://localhost:5000 Note: the tunnel will run in the background, and will die when you terminate the EC2 instance.

First we will forward the port the web appliations use... Forwarding the remote machine's port 5000 to the local port 5000, which you can then access at http://localhost:5000 Executing: ssh -N -i ./agile_data_science.pem -o StrictHostKeyChecking=no -L 5000:localhost:5000 ubuntu@$EC2_HOSTNAME &

Next we will forward the port the Jupyter Notebooks use... Forwarding the remote machine's port 8888 to the local port 8888, which you can then access at http://localhost:8888 Executing: ssh -N -i ./agile_data_science.pem -o StrictHostKeyChecking=no -L 8888:localhost:8888 ubuntu@$EC2_HOSTNAME &

Next we will forward the port that Airflow uses... Forwarding the remote machine's port 8080 to the local port 8080, which you can then access at http://localhost:8080 Executing: ssh -N -i ./agile_data_science.pem -o StrictHostKeyChecking=no -L 8080:localhost:8080 ubuntu@$EC2_HOSTNAME &

timmoti$ key_load_private: invalid format key_load_private: invalid format key_load_private: invalid format Load key "./agile_data_science.pem": invalid format Load key "./agile_data_science.pem": invalid format Load key "./agile_data_science.pem": invalid format ubuntu@ec2-184-73-145-251.compute-1.amazonaws.com: Permission denied (publickey). ubuntu@ec2-184-73-145-251.compute-1.amazonaws.com: Permission denied (publickey). ubuntu@ec2-184-73-145-251.compute-1.amazonaws.com: Permission denied (publickey). `

I've got to hit enter to get the prompt back.

Any idea how to troubleshoot this?

[rjurney]

@timmoti Delete the key in the file system and the AWS interface and then try ec2.sh again.

[sewald101]

Was having trouble ssh'ing in.

First I thought the problem was that the script wasn't detecting my external IP per this suspicious output, which I thought, in turn, might be screwing up the security group:

After a few minutes (for it to initialize), you may ssh to this machine via the command in red: ssh -i ./agile_data_science.pem ubuntu@ec2-52-34-149-8.us-west-2.compute.amazonaws.com Note: only your IP of ';; connection timed out; no servers could be reached' is authorized to connect to this machine.

Fixed that by manually setting EXTERNAL_IP on line 49 with EXTERNAL_IP='echo <my IP here>'. Then re-ran ec2.sh. The output rule from ec2.sh and the security group on AWS now looked good. But still couldn't log in.

SOLUTION: From the AWS Console, I had to manually create and inbound SSH rule listing my IP. Instructions here: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-ssh-troubleshooting/

[rjurney]

@sewald101 Sorry things went wrong, when the first run fails things can get wonky.