Paramiko - PasswordRequiredException when restarting chiadog

pieterhelsen commented 3 years ago

Describe the bug

I'm running chiadog in a remote harvester setup with a password-protected private key from a tmux session with multiple Windows (one for each remote harvester). After upgrading to 0.4.1 (main) and restarting chiadog I get an error message saying that the private key file is encrypted.

It appears that the Paramiko library is not able to access ssh-agent.

    self._ssh_client.connect(hostname=self._remote_host, username=self._remote_user)
  File "/home/pieter/chiadog/venv/lib/python3.8/site-packages/paramiko/client.py", line 435, in connect
    self._auth(
  File "/home/pieter/chiadog/venv/lib/python3.8/site-packages/paramiko/client.py", line 764, in _auth
    raise saved_exception
  File "/home/pieter/chiadog/venv/lib/python3.8/site-packages/paramiko/client.py", line 734, in _auth
    key = self._key_from_filepath(
  File "/home/pieter/chiadog/venv/lib/python3.8/site-packages/paramiko/client.py", line 586, in _key_from_filepath
    key = klass.from_private_key_file(key_path, password)
  File "/home/pieter/chiadog/venv/lib/python3.8/site-packages/paramiko/pkey.py", line 235, in from_private_key_file
    key = cls(filename=filename, password=password)
  File "/home/pieter/chiadog/venv/lib/python3.8/site-packages/paramiko/rsakey.py", line 55, in __init__
    self._from_private_key_file(filename, password)
  File "/home/pieter/chiadog/venv/lib/python3.8/site-packages/paramiko/rsakey.py", line 175, in _from_private_key_file
    data = self._read_private_key_file("RSA", filename, password)
  File "/home/pieter/chiadog/venv/lib/python3.8/site-packages/paramiko/pkey.py", line 308, in _read_private_key_file
    data = self._read_private_key(tag, f, password)
  File "/home/pieter/chiadog/venv/lib/python3.8/site-packages/paramiko/pkey.py", line 337, in _read_private_key
    data = self._read_private_key_openssh(lines[start:end], password)
  File "/home/pieter/chiadog/venv/lib/python3.8/site-packages/paramiko/pkey.py", line 438, in _read_private_key_openssh
    raise PasswordRequiredException(
paramiko.ssh_exception.PasswordRequiredException: private key file is encrypted

The workaround for this issue is to kill ssh-agent and re-add the key everytime I restart the process. So since I have three remote harvesters (one Windows, two Linux), I need to do:

Run python3 chiadog --config harvester1.yaml to see if the issue occurs
Stop chiadog
Run ps aux | grep ssh-agent to find out the PID
Run kill [PID]
Run ``evalssh-agent -s````
Run ssh-add ~/.ssh/id_rsa
Run python3 chiadog --config harvester1.yaml
Rinse and repeat for each harvester

I am not sure whether it is chiadogs implementation of Paramiko or some inherent bug in the Paramiko library. I did notice that Paramiko use the PasswordRequiredException as a bit of a catch-all, so it may be something else entirely.

My main goal for this ticket:

Check whether other people have the same problem
Come up with potential solutions or explanations. E.g. implement Fabric instead of Paramiko, which is a simplified interface on top of Paramiko and might simplify some of the error handling.

Environment:

OS: Ubuntu 20.04
Python version: 3.8.5
PIP version: 20.3.3
Chia version: 1.1.1

martomi commented 3 years ago

Paramiko's connect method accepts passphrase argument that can be used to unlock private keys protected with passwords. But implementing that and storing the password in config file kinda defeats the purpose of the password in the first place. I think the ssh-agent keeps password protected private keys unlocked for some time after you add them and that's probably why it works with re-adding.

I've looked info Fabric before and it's not a better option.

pieterhelsen commented 3 years ago

Delving deeper into the issue, this appears to be related to tmux, not Paramiko. https://blog.testdouble.com/posts/2016-11-18-reconciling-tmux-and-ssh-agent-forwarding/

martomi / chiadog

Paramiko - PasswordRequiredException when restarting chiadog #57