search

osrf / cloudsim-widgets

Other
0 stars 1 forks source link

SASC: Rebooting payload machine doesn't work #36

Closed osrf-migration closed 7 years ago

osrf-migration commented 7 years ago

Original report (archived issue) by Hugo Boyer (Bitbucket: hugomatic, GitHub: hugomatic).

The original report had attachments: syslog.zip

After a sudo reboot, and the machine never came back up. we were pinging both locally on our machine over vpn and from the arbiter machine, no response.

Note: AWS machines normally come back online after a reboot, so this may be related to the VPN setup.

osrf-migration commented 7 years ago

Original comment by Hugo Boyer (Bitbucket: hugomatic, GitHub: hugomatic).

Part of Syslog where the vpn is lost:

#!syslog

Jan 12 15:43:48 ip-172-31-2-11 cloud-init[1425]: + cd /home/ubuntu/code/cloudsim-sim/aws/../../vpn
Jan 12 15:43:48 ip-172-31-2-11 cloud-init[1425]: + tar xf bundle.tgz
Jan 12 15:43:50 ip-172-31-2-11 cloud-init[1425]: + echo cd /home/ubuntu/code/cloudsim-sim/aws/../../vpn
Jan 12 15:43:50 ip-172-31-2-11 cloud-init[1425]: + cd /home/ubuntu/code/cloudsim-sim/aws/../../vpn
Jan 12 15:43:50 ip-172-31-2-11 cloud-init[1425]: + echo openvpn --config openvpn.conf --daemon
Jan 12 15:43:50 ip-172-31-2-11 cloud-init[1425]: + openvpn --config openvpn.conf --daemon
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1879]: OpenVPN 2.3.10 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Feb  2 2016
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1879]: library versions: OpenSSL 1.0.2g  1 Mar 2016, LZO 2.08
Jan 12 15:43:50 ip-172-31-2-11 cloud-init[1425]: + cat
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: UDPv4 link local: [undef]
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: UDPv4 link remote: [AF_INET]52.53.245.198:1196
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: TLS: Initial packet from [AF_INET]52.53.245.198:1196, sid=15b82286 95ef6677
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: VERIFY OK: depth=1, C=US, ST=CA, L=Mountain View, O=OSRF, OU=Gazebo, CN=OSRF CA, name=EasyRSA, emailAddress=info@osrfoundation.org
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: VERIFY OK: nsCertType=SERVER
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: VERIFY OK: depth=0, C=US, ST=CA, L=Mountain View, O=OSRF, OU=Gazebo, CN=blue_sascround-098, name=EasyRSA, emailAddress=info@osrfoundation.org
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: Data Channel Encrypt: Cipher 'AES-128-CBC' initialized with 128 bit key
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: Data Channel Encrypt: Using 256 bit message hash 'SHA256' for HMAC authentication
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: Data Channel Decrypt: Cipher 'AES-128-CBC' initialized with 128 bit key
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: Data Channel Decrypt: Using 256 bit message hash 'SHA256' for HMAC authentication
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Jan 12 15:43:50 ip-172-31-2-11 openvpn[1881]: [blue_sascround-098] Peer Connection Initiated with [AF_INET]52.53.245.198:1196
Jan 12 15:43:51 ip-172-31-2-11 ec2:
Jan 12 15:43:51 ip-172-31-2-11 ec2: #############################################################
Jan 12 15:43:51 ip-172-31-2-11 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
Jan 12 15:43:51 ip-172-31-2-11 ec2: 1024 SHA256:mW4PpSJsFQnBe2iLQS2PLLE3vQTXzsrQQvDCu8NmN2E root@ip-172-31-2-11 (DSA)
Jan 12 15:43:51 ip-172-31-2-11 ec2: 256 SHA256:d8SkECfT+Hs0xeIYbImsLn5TbLaiDq/rWQJQ7keQ1Zo root@ip-172-31-2-11 (ECDSA)
Jan 12 15:43:51 ip-172-31-2-11 ec2: 256 SHA256:y8MsKF9qLBCYx0yyACMKAQ/scMbgetl0eQlm5t0TyOY root@ip-172-31-2-11 (ED25519)
Jan 12 15:43:51 ip-172-31-2-11 ec2: 2048 SHA256:PDMplBd0bwzU9lxJgP4XoeLATlHteRDHU+4ZbBmmeyo root@ip-172-31-2-11 (RSA)
Jan 12 15:43:51 ip-172-31-2-11 ec2: -----END SSH HOST KEY FINGERPRINTS-----
Jan 12 15:43:51 ip-172-31-2-11 ec2: #############################################################
Jan 12 15:43:51 ip-172-31-2-11 systemd[1]: Started Execute cloud user/final scripts.
Jan 12 15:43:51 ip-172-31-2-11 systemd[1]: Reached target Cloud-init target.
Jan 12 15:43:51 ip-172-31-2-11 cloud-init[1425]: Cloud-init v. 0.7.8 running 'modules:final' at Thu, 12 Jan 2017 15:43:22 +0000. Up 49.45 seconds.
Jan 12 15:43:51 ip-172-31-2-11 cloud-init[1425]: Cloud-init v. 0.7.8 finished at Thu, 12 Jan 2017 15:43:51 +0000. Datasource DataSourceEc2.  Up 77.82 seconds
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: SENT CONTROL [blue_sascround-098]: 'PUSH_REQUEST' (status=1)
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: PUSH: Received control message: 'PUSH_REPLY,route 192.168.3.1 255.255.255.255,route-gateway 192.168.2.1,ping 10,ping-restart 120,ifconfig 192.168.2.10 255.255.255.0'
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: OPTIONS IMPORT: timers and/or timeouts modified
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: OPTIONS IMPORT: --ifconfig/up options modified
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: OPTIONS IMPORT: route options modified
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: OPTIONS IMPORT: route-related options modified
Jan 12 15:43:52 ip-172-31-2-11 systemd-udevd[1911]: Could not generate persistent MAC address for tap0: No such file or directory
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: ROUTE_GATEWAY 172.31.0.1/255.255.240.0 IFACE=ens3 HWADDR=06:9a:1c:18:c9:f6
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: TUN/TAP device tap0 opened
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: TUN/TAP TX queue length set to 100
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: /sbin/ip link set dev tap0 up mtu 1500
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: /sbin/ip addr add dev tap0 192.168.2.10/24 broadcast 192.168.2.255
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: /sbin/ip route add 192.168.3.1/32 via 192.168.2.1
Jan 12 15:43:52 ip-172-31-2-11 openvpn[1881]: Initialization Sequence Completed
Jan 12 15:45:11 ip-172-31-2-11 systemd[1]: Started Daily apt activities.
Jan 12 15:45:11 ip-172-31-2-11 systemd[1]: apt-daily.timer: Adding 10h 6min 33.108219s random time.
Jan 12 15:45:11 ip-172-31-2-11 systemd[1]: Startup finished in 9.872s (kernel) + 2min 28.608s (userspace) = 2min 38.480s.
Jan 12 15:45:11 ip-172-31-2-11 systemd[1]: apt-daily.timer: Adding 4h 54min 11.607412s random time.
Jan 12 15:50:56 ip-172-31-2-11 systemd[1]: Created slice User Slice of ubuntu.
Jan 12 15:50:56 ip-172-31-2-11 systemd[1]: Starting User Manager for UID 1000...
Jan 12 15:50:56 ip-172-31-2-11 systemd[1]: Started Session 1 of user ubuntu.

...

Jan 12 15:58:11 ip-172-31-2-11 systemd[1]: Started Cleanup of Temporary Directories.
Jan 12 16:12:16 ip-172-31-2-11 dhclient[1060]: DHCPREQUEST of 172.31.2.11 on ens3 to 172.31.0.1 port 67 (xid=0x6b32b910)
Jan 12 16:12:16 ip-172-31-2-11 dhclient[1060]: DHCPACK of 172.31.2.11 from 172.31.0.1
Jan 12 16:12:16 ip-172-31-2-11 dhclient[1060]: bound to 172.31.2.11 -- renewal in 1668 seconds.
Jan 12 16:14:38 ip-172-31-2-11 systemd[1]: Stopping User Manager for UID 1000...
Jan 12 16:14:38 ip-172-31-2-11 systemd[2535]: Stopped target Default.
Jan 12 16:14:38 ip-172-31-2-11 systemd[2535]: Stopped target Basic System.
Jan 12 16:14:38 ip-172-31-2-11 systemd[2535]: Stopped target Paths.
Jan 12 16:14:38 ip-172-31-2-11 systemd[2535]: Stopped target Sockets.
Jan 12 16:14:38 ip-172-31-2-11 systemd[2535]: Stopped target Timers.
Jan 12 16:14:38 ip-172-31-2-11 systemd[2535]: Reached target Shutdown.
Jan 12 16:14:38 ip-172-31-2-11 systemd[2535]: Starting Exit the Session...
Jan 12 16:14:38 ip-172-31-2-11 systemd[2535]: Received SIGRTMIN+24 from PID 10857 (kill).
Jan 12 16:14:38 ip-172-31-2-11 systemd[1]: Stopped User Manager for UID 1000.
Jan 12 16:14:38 ip-172-31-2-11 systemd[1]: Removed slice User Slice of ubuntu.
Jan 12 16:17:01 ip-172-31-2-11 CRON[10866]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jan 12 16:18:25 ip-172-31-2-11 openvpn[1881]: [blue_sascround-098] Inactivity timeout (--ping-restart), restarting
Jan 12 16:18:25 ip-172-31-2-11 openvpn[1881]: SIGUSR1[soft,ping-restart] received, process restarting
Jan 12 16:18:25 ip-172-31-2-11 openvpn[1881]: Restart pause, 2 second(s)
Jan 12 16:18:27 ip-172-31-2-11 openvpn[1881]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Jan 12 16:18:27 ip-172-31-2-11 openvpn[1881]: UDPv4 link local: [undef]
Jan 12 16:18:27 ip-172-31-2-11 openvpn[1881]: UDPv4 link remote: [AF_INET]52.53.245.198:1196
Jan 12 16:19:27 ip-172-31-2-11 openvpn[1881]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
osrf-migration commented 7 years ago

Original comment by Brian Gerkey (Bitbucket: Brian Gerkey, GitHub: gerkey).

Odd. I would have expected https://osrf-migration.github.io/osrf-archived-gh-pages/#!/osrf/cloudsim-sim/pull-requests/27 to make the VPN come back up.

osrf-migration commented 7 years ago

Original comment by Tully Foote (Bitbucket: Tully Foote, GitHub: tfoote).

It looks like there's one bug in the rc.local script ( a trailing " >>/etc/rc.local here (osrf/cloudsim-sim@75c347ee446fbeaa38d12fb33f36c546ebbbeda7)

The contents look like this:

ubuntu@ip-172-31-16-25:~$ cat /etc/rc.local 
#!/bin/bash
cd /home/ubuntu/code/cloudsim-sim/aws/../../vpn && openvpn --config openvpn.conf --daemon" >> /etc/rc.local
exit 0

This appears to be why it didn't run: 

ubuntu@ip-172-31-16-25:~$ sudo /etc/rc.local 
/etc/rc.local: line 2: unexpected EOF while looking for matching `"'
/etc/rc.local: line 4: syntax error: unexpected end of file

Fixed proposed: https://osrf-migration.github.io/osrf-archived-gh-pages/#!/osrf/cloudsim-sim/pull-requests/30/

osrf-migration commented 7 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

It looks like this was an issue on cloudsim-sim which has been resolved? I'll close this, please reopen if this is still an issue, and it is related to cloudsim-widgets.

osrf-migration commented 7 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).