Cannot run any playbook due to a `ValueError("Invalid padding bytes.")`

[powerpaul17]

Describe the bug It is not possible to run any playbook (including the built-in ones as ping, etc.) because there seems to be a problem with vault decryption. I tried with a very plain password and also with the previous version (0.1.14) and it also did not work.

Logs

Starting...
ansible-playbook [core 2.17.0]
config file = /server/src/ansible/ansible.cfg
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.12/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible-playbook
python version = 3.12.3 (main, Apr 18 2024, 07:52:31) [GCC 13.2.1 20240309] (/usr/bin/python3)
jinja version = 3.1.4
libyaml = True
Using /server/src/ansible/ansible.cfg as config file
Reading vault password file: /server/src/ansible/ssm-ansible-vault-password-client.py
The vault password file /server/src/ansible/ssm-ansible-vault-password-client.py is a client script.
Executing vault password client script: /server/src/ansible/ssm-ansible-vault-password-client.py --vault-id default
setting up inventory plugins
Loading collection ansible.builtin from
host_list declined parsing /server/src/ansible/inventory/hosts.json as it did not pass its verify_file() method
script declined parsing /server/src/ansible/inventory/hosts.json as it did not pass its verify_file() method
auto declined parsing /server/src/ansible/inventory/hosts.json as it did not pass its verify_file() method
Parsed /server/src/ansible/inventory/hosts.json inventory source with yaml plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3.12/site-packages/ansible/plugins/callback/default.py
Loading callback plugin awx_display of type stdout, v2.0 from /usr/lib/python3.12/site-packages/ansible_runner/display_callback/callback/awx_display.py
Attempting to use 'awx_display' callback.
Skipping callback 'awx_display', as we already have a stdout callback.
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.
PLAYBOOK: _ping.yml ************************************************************
Positional arguments: /server/src/ansible/00000000-0000-0000-0000-000000000000/device/_ping.yml
verbosity: 5
connection: ssh
become_method: sudo
tags: ('all',)
inventory: ('/server/src/ansible/inventory/hosts.json',)
vault_password_files: ('/server/src/ansible/ssm-ansible-vault-password-client.py',)
forks: 5
1 plays in /server/src/ansible/00000000-0000-0000-0000-000000000000/device/_ping.yml
PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
task path: /server/src/ansible/00000000-0000-0000-0000-000000000000/device/_ping.yml:1
Found a vault_id (ssm) in the vaulttext
Found a vault_id (ssm) in the vault text, but we do not have a associated secret (--vault-id)
Trying to use vault secret=(ClientScriptVaultSecret(filename='/server/src/ansible/ssm-ansible-vault-password-client.py', vault_id='default')) id=default to decrypt None
Trying secret ClientScriptVaultSecret(filename='/server/src/ansible/ssm-ansible-vault-password-client.py', vault_id='default') for vault_id=default
Found a vault_id (ssm) in the vaulttext
Found a vault_id (ssm) in the vault text, but we do not have a associated secret (--vault-id)
Trying to use vault secret=(ClientScriptVaultSecret(filename='/server/src/ansible/ssm-ansible-vault-password-client.py', vault_id='default')) id=default to decrypt None
Trying secret ClientScriptVaultSecret(filename='/server/src/ansible/ssm-ansible-vault-password-client.py', vault_id='default') for vault_id=default
The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/ansible/playbook/base.py", line 551, in post_validate
if omit_value is not None and value == omit_value:
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/yaml/objects.py", line 116, in __eq__
return other == self.data
^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/yaml/objects.py", line 105, in data
return to_text(self.vault.decrypt(self._ciphertext, obj=self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/vault/__init__.py", line 642, in decrypt
plaintext, vault_id, vault_secret = self.decrypt_and_get_vault_id(vaulttext, filename=filename, obj=obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/vault/__init__.py", line 722, in decrypt_and_get_vault_id
b_plaintext = this_cipher.decrypt(b_vaulttext, vault_secret)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/vault/__init__.py", line 1288, in decrypt
b_plaintext = cls._decrypt_cryptography(b_ciphertext, b_crypted_hmac, b_key1, b_key2, b_iv)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/vault/__init__.py", line 1251, in _decrypt_cryptography
) + unpadder.finalize()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/cryptography/hazmat/primitives/padding.py", line 160, in finalize
result = _byte_unpadding_check(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/cryptography/hazmat/primitives/padding.py", line 97, in _byte_unpadding_check
raise ValueError("Invalid padding bytes.")
ValueError: Invalid padding bytes.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/ansible/executor/task_executor.py", line 164, in run
res = self._execute()
^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/executor/task_executor.py", line 449, in _execute
self._play_context.post_validate(templar=templar)
File "/usr/lib/python3.12/site-packages/ansible/playbook/base.py", line 564, in post_validate
"The error was: %s" % (name, value, attribute.isa, e), obj=self.get_ds(), orig_exc=e)
^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/yaml/objects.py", line 130, in __str__
return to_native(self.data, errors='surrogate_or_strict')
^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/yaml/objects.py", line 105, in data
return to_text(self.vault.decrypt(self._ciphertext, obj=self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/vault/__init__.py", line 642, in decrypt
plaintext, vault_id, vault_secret = self.decrypt_and_get_vault_id(vaulttext, filename=filename, obj=obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/vault/__init__.py", line 722, in decrypt_and_get_vault_id
b_plaintext = this_cipher.decrypt(b_vaulttext, vault_secret)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/vault/__init__.py", line 1288, in decrypt
b_plaintext = cls._decrypt_cryptography(b_ciphertext, b_crypted_hmac, b_key1, b_key2, b_iv)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/ansible/parsing/vault/__init__.py", line 1251, in _decrypt_cryptography
) + unpadder.finalize()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/cryptography/hazmat/primitives/padding.py", line 160, in finalize
result = _byte_unpadding_check(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/cryptography/hazmat/primitives/padding.py", line 97, in _byte_unpadding_check
raise ValueError("Invalid padding bytes.")
ValueError: Invalid padding bytes.
 fatal: [152.53.2.64]: FAILED! => {
  "msg": "Unexpected failure during module execution: Invalid padding bytes.",
  "stdout": ""
}
PLAY RECAP *********************************************************************
152.53.2.64 : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 

To Reproduce Steps to reproduce the behavior:

1. Inventory
2. Click on "Ping" from the actions menu of a device
3. See error

Expected behavior The playbook should run through without an error.

(please complete the following information):

• OS: Linux (Debian) on ARM64
• Browser: Firefox
• Version 0.1.15

[SquirrelDeveloper]

Hi @powerpaul17 I am not able to reproduce. Did you change the VAULT_PWD along the way ? It seems more like the underlying values could be wrongly encrypted. Could you try to set again all the authentication info of the device in Inventory, your device , Configuration so the values are saved and encrypted again.

[powerpaul17]

I already tried to change the password to a very simple one and updated the configuration of the device. I also removed the data directories and recreated the compose stack. Both times the same errors occurred.

[powerpaul17]

Also I tried the same on a machine with x86_64 architecture and it also didn't work.. 😔

[SquirrelDeveloper]

@powerpaul17 Could you tell me if you what kind of ssh authentication are you using? Key based or user/password? Also, what kind of sudo method? Do you have a sudo user AND a sudo password? Could you also tell me if any point you tried to unset one of those parameters?

[SquirrelDeveloper]

BTW, checking Ansible Vault code, there is a comment

           # In VaultAES, ValueError: invalid padding bytes can mean bad
            # password was given

[powerpaul17]

@powerpaul17 Could you tell me if you what kind of ssh authentication are you using? Key based or user/password? Also, what kind of sudo method? Do you have a sudo user AND a sudo password? Could you also tell me if any point you tried to unset one of those parameters?

I used user/password and 'sudo' sudo method. At first I did not set the sudo password but then I tried with it but it didn't work in both cases. No, I didn't explicitly unset these parameters but I reset them after changing the vault secret (like you told me before)

[powerpaul17]

BTW, checking Ansible Vault code, there is a comment

           # In VaultAES, ValueError: invalid padding bytes can mean bad
            # password was given

I think this refers to the vault password. As this is set through the environment variable it should not be wrong?

[SquirrelDeveloper]

Do you have the ability to install Squirrel on another device? To test if there is no underlying cryptographic magic.

This error, at this step, means something is wrong with either the SSH password (or key) or the Sudo password, those are the only ones vaulted, hence, triggering vault decryption.

[powerpaul17]

I tried already on another machine and got the same error. Or do you mean I should try adding another machine in Squirrel?

[powerpaul17]

SSH & Sudo password are working on the client machine, I tested it explicitly again.

[SquirrelDeveloper]

Well @powerpaul17, I can't understand the issue. Will you be willing to send me a zip file of both the whole content of/.data.prod/ and your .env file to squirrelserversmanager[...at...]gmail.com ? I dont any other way so I can directly reproduce this bug

[powerpaul17]

Ok, thanks for the effort. I tried again putting wrong passwords in SSH/Sudo input fields. When I put the wrong password for SSH, I get a message that the connection was not successful, so I suppose it has to do with the value of the passwords. I'll run some more checks.

[powerpaul17]

FYI, it worked with a SSH key, I don't want to send you my current password but I suppose there are some problematic characters in it.

[SquirrelDeveloper]

@powerpaul17 Good clue, I will test with that in mind

[SquirrelDeveloper]

@powerpaul17 I am now reproducing the error!

[SquirrelDeveloper]

Fix in latest version