decryption of secrets fails using batou config of fc.directory.deployment

[mcdonc]

I've been working with Christian Theune on flyingcircus code, and today we ran into this issue and he asked me to put it here. I apologize that I cant give an easily reproducible set of steps to replicate this (it'll be a pretty sketchy bug report here), but when working on fc.directory.deployment, if I use bateau deploy dev on the branch which uses age rather than gpg (I guess it's in master now), it doesn't seem to want to decrypt secrets. When I type my (correct) ssh passphrase upon being asked it gives this traceback. I verified the correctness of the passphrase by doing ssh-keygen -y so I know it works.

 ~/pr/f/fc.directory.deployment$ ./batou deploy dev   
batou/2.4a4.dev0 (cpython 3.10.12-final0, Linux 6.1.31 x86_64)
================================== Preparing ===================================
main: Loading environment `dev`...
main: Verifying repository ...
You are using rsync. This is a non-verifying repository -- continuing on your own risk!
main: Loading secrets ...
Enter passphrase for /home/chrism/.ssh/id_ed25519: 

ERROR: Unexpected exception
        Traceback (most recent call last):
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/deploy.py", line 371, in main
    getattr(deployment, step)()
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/deploy.py", line 148, in load
    self.environment.load_secrets()
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/environment.py", line 248, in load_secrets
    self.secret_provider.inject_secrets()
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/__init__.py", line 116, in inject_secrets
    secret_blob = self.read()
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/__init__.py", line 280, in read
    for section in self.config.sections():
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/__init__.py", line 271, in config
    return ConfigUpdater().read_string(self.config_file.cleartext)
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/encryption.py", line 42, in cleartext
    return self.decrypted.decode("utf-8")
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/encryption.py", line 32, in decrypted
    self._decrypted = self.decrypt()
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/encryption.py", line 341, in decrypt
    chunk = os.read(fd, 1024)
        OSError: [Errno 5] Input/output error
======================= DEPLOYMENT FAILED (during load) ========================
 ~/projects/flyingcircus/fc.directory.deployment  master  ssh-keygen -y                                                                                                1 ✘  5s 
Enter file in which the key is (/home/chrism/.ssh/id_rsa): ^C
 ~/projects/flyingcircus/fc.directory.deployment  master  ssh-keygen -y                                                                                              INT ✘  3s 
Enter file in which the key is (/home/chrism/.ssh/id_rsa): /home/chrism/.ssh/id_ed25519
Enter passphrase: 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOLXUsGqUIEMfcXoIiiItmGNqOucJjx5D6ZEE3KgLKYV ednesia

[elikoga]

Would be great to see this run with the debug flag: ./batou -d ...

[mcdonc]

~/pr/f/fc.directory.deployment  master  ./batou -d deploy dev            ✔ 
Removing expired path: /home/chrism/projects/flyingcircus/fc.directory.deployment/./.appenv/d1491c4c ...
Creating venv ...
/home/chrism/projects/flyingcircus/fc.directory.deployment/./batou:76: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  import distutils.util  # noqa: F401 imported but unused
Ensuring pip ...
Installing ...
batou/2.4a4.dev0 (cpython 3.10.12-final0, Linux 6.1.31 x86_64)
================================== Preparing ===================================
main: Loading environment `dev`...
resolving (v4) `5b0d5312`
resolving (v6) `5b0d5312`
main: Number of jobs: 1
main: Verifying repository ...
You are using rsync. This is a non-verifying repository -- continuing on your own risk!
main: Loading secrets ...
Looking for secrets for environment dev.
Found age secrets for environment dev.
Injecting secrets for environment dev.
Locking `/home/chrism/projects/flyingcircus/fc.directory.deployment/environments/dev/secrets.cfg.age`
Found identities: ['/home/chrism/.ssh/id_rsa', '/home/chrism/.ssh/id_ed25519']
Running `['age', '--version']`
Running `['age', '-d', '-i', '/home/chrism/.ssh/id_rsa', '-o', '/tmp/tmp_ei6y327', '/home/chrism/projects/flyingcircus/fc.directory.deployment/environments/dev/secrets.cfg.age']`
Running `['age', '-d', '-i', '/home/chrism/.ssh/id_ed25519', '-o', '/tmp/tmpnkjf3j4h', '/home/chrism/projects/flyingcircus/fc.directory.deployment/environments/dev/secrets.cfg.age']`
Enter passphrase for /home/chrism/.ssh/id_ed25519: 
Unlocking `/home/chrism/projects/flyingcircus/fc.directory.deployment/environments/dev/secrets.cfg.age`

ERROR: Unexpected exception
        Traceback (most recent call last):
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/deploy.py", line 371, in main
    getattr(deployment, step)()
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/deploy.py", line 148, in load
    self.environment.load_secrets()
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/environment.py", line 248, in load_secrets
    self.secret_provider.inject_secrets()
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/__init__.py", line 116, in inject_secrets
    secret_blob = self.read()
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/__init__.py", line 280, in read
    for section in self.config.sections():
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/__init__.py", line 271, in config
    return ConfigUpdater().read_string(self.config_file.cleartext)
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/encryption.py", line 42, in cleartext
    return self.decrypted.decode("utf-8")
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/encryption.py", line 32, in decrypted
    self._decrypted = self.decrypt()
        File "/home/chrism/projects/flyingcircus/fc.directory.deployment/.appenv/0b623105/lib/python3.10/site-packages/batou/secrets/encryption.py", line 341, in decrypt
    chunk = os.read(fd, 1024)
        OSError: [Errno 5] Input/output error
======================= DEPLOYMENT FAILED (during load) ========================
main: Disconnecting from nodes ...

[ctheune]

I tried to reproduce with my setup (both on plain batou and the Frankenstein one we use in this project) and it worked afaict.

I think we might need even more debugging output, but I can do that with Chris together and throw in a pdb in the virtualenv.

[elikoga]

I didn't have any luck replicating this, maybe it's yet another age build and a different behaviour for the binary?

[ctheune]

It turns out that this is another instance of the PTY dance that we already did see in the expect function. We're fixing that now.

[mcdonc]

Closed by PR #382