Retry injector restart when network unavailable

[jccurtis]

This morning the Building 50 power went down and when the server regained power it did not have network access. Thus the injector tmux a -t INJECTOR failed:

injector
dosenet@dosenet:~$ . ~/.bashrc

 * keychain 2.7.1 ~ http://www.funtoo.org
 * Starting ssh-agent...
 * Adding 1 ssh key(s): /home/dosenet/.ssh/id_rsa_github
 * ssh-add: Identities added: /home/dosenet/.ssh/id_rsa_github

 * keychain 2.7.1 ~ http://www.funtoo.org
 * Found existing ssh-agent: 1446
 * Adding 1 ssh key(s): /home/dosenet/.ssh/id_rsa_kepler
 * ssh-add: Identities added: /home/dosenet/.ssh/id_rsa_kepler

dosenet:~$ injector
[CONFIG] Setting session timezone to: +00:00
        Private Key: /home/dosenet/.ssh/id_rsa_lbl
        Symmetric Key: /home/dosenet/secret.aes
Traceback (most recent call last):
  File "/home/dosenet/git/dosenet/injector.py", line 830, in <module>
    main(**vars(parser.parse_args()))
  File "/home/dosenet/git/dosenet/injector.py", line 799, in main
    inj = Injector(**kwargs)
  File "/home/dosenet/git/dosenet/injector.py", line 154, in __init__
    self.ip = self.get_external_ip()
  File "/home/dosenet/git/dosenet/injector.py", line 182, in get_external_ip
    s.connect(('8.8.8.8', 80))
  File "/home/dosenet/anaconda/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 101] Network is unreachable
dosenet:~$

A try/except should be added to the injector script to catch socket.error: [Errno 101] Network is unreachable and retry after some time (maybe 1 minute...)

[bplimley]

Yes... the Slacker would have failed too in trying to connect to Slack.