AWX Credential Create: ssh_key_data fails to detect/sanitize newlines in keys and 500's with no useable output

danxg87 commented 1 year ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
[X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

Hey all,

When creating a new Source Control credential in AWX, we were receiving a 500 error from AWX:

"msg": "The host sent back a server error (HTTP Error 500: Internal Server Error): /api/v2/credentials/. Please check the logs and try again later"

The AWX server logs showed a decoding error:

2023-07-10 14:59:30,131 ERROR    ..... django.request Internal Server Error: /api/v2/credentials/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/api/generics.py", line 334, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/rest_framework/generics.py", line 242, in post
    return self.create(request, *args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/rest_framework/mixins.py", line 18, in create
    serializer.is_valid(raise_exception=True)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/rest_framework/serializers.py", line 227, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/api/serializers.py", line 595, in run_validation
    return super(BaseSerializer, self).run_validation(data)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/rest_framework/serializers.py", line 429, in run_validation
    value = self.validate(value)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/api/serializers.py", line 2738, in validate
    return super(CredentialSerializerCreate, self).validate(attrs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/api/serializers.py", line 2645, in validate
    return super(CredentialSerializer, self).validate(attrs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/api/serializers.py", line 629, in validate
    obj.full_clean(exclude=exclusions)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/base.py", line 1229, in full_clean
    self.clean_fields(exclude=exclude)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/models/base.py", line 121, in clean_fields
    super(BaseModel, self).clean_fields(exclude)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/base.py", line 1271, in clean_fields
    setattr(self, f.attname, f.clean(raw_value, self))
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/fields/__init__.py", line 670, in clean
    self.validate(value, model_instance)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/fields.py", line 594, in validate
    for error in Draft4Validator(self.schema(model_instance), format_checker=self.format_checker).iter_errors(decrypted_values):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/jsonschema/validators.py", line 288, in iter_errors
    for error in errors:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/jsonschema/_validators.py", line 332, in properties
    yield from validator.descend(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/jsonschema/validators.py", line 305, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/jsonschema/validators.py", line 288, in iter_errors
    for error in errors:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/jsonschema/_validators.py", line 238, in format
    validator.format_checker.check(instance, format)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/jsonschema/_format.py", line 135, in check
    result = func(instance)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/fields.py", line 444, in format_ssh_private_key
    validate_ssh_private_key(value)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/validators.py", line 184, in validate_ssh_private_key
    return validate_pem(data, min_keys=1)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/validators.py", line 104, in validate_pem
    decoded_data = base64.b64decode(base64_data)
  File "/usr/lib64/python3.9/base64.py", line 87, in b64decode
    return binascii.a2b_base64(s)
binascii.Error: Incorrect padding

This was not very helpful and led me down multiple unrelated rabbit holes/issues/etc.

Ultimately, it turns out the user was supplying a VALID ssh_key, but there appear to have been newlines in it, causing us to continually get 500 errors as above every time (UI/API accepted the ssh_key_data as valid, but failed as above if there were newlines somewhere).

By trimming the ssh_key_data in our playbook and rerunning, things succeeded just fine.

It appears we have a bug either:

not exposing to the user there is an unknown formatting issue (invisible/newline chars in our case)
not sanitizing the user inputs to account for/trim out these invisible chars server-side when accepting/using the data

If someone can, please confirm the behavior as I've outlined and if it seems appropriate, add some capability to "sanitize" the ssh_key_data and/or message on detected issues with the content so end users can address that.

AWX version

21.11.0

Select the relevant components

[X] UI
[ ] UI (tech preview)
[X] API
[ ] Docs
[ ] Collection
[ ] CLI
[ ] Other

Installation method

kubernetes

Modifications

yes

Ansible version

2.14.5

Operating system

Linux/Ubuntu

Web browser

Chrome

Steps to reproduce

Generate SSH key w/ passphrase
Set ssh key to env var GH_KEY_DATA
Set ssh passphrase to env var GH_UNLOCK_DATA
Use the following playbook to try to deploy a cred using the generated key:
```
---
```
hosts: localhost connection: local gather_facts: false

tasks:
- name: Deploy test SC credential awx.awx.credential: name: test SC credential description: Testing credential to highlight unexpected AWX 500 errors controller_host: "{{ lookup('ansible.builtin.env', 'TOWER_HOST') }}" controller_username: "{{ lookup('ansible.builtin.env', 'TOWER_USERNAME') }}" controller_password: "{{ lookup('ansible.builtin.env', 'TOWER_PASSWORD') }}" organization: test org credential_type: Source Control inputs: username: git ssh_key_data: "{{ lookup('ansible.builtin.env', 'GH_KEY_DATA') }}" ssh_key_unlock: "{{ lookup('ansible.builtin.env', 'GH_UNLOCK_DATA') }}"

Expected results

A successfully created Source Control credential with the generated ssh_key data/passphrase.

Actual results

We get a 500 error back from AWX. AWX server logs show a padding error/no clear reason for the actual failure here, which was the key data string had "hidden characters" in it.

Additional information

No response

fosterseth commented 1 year ago

@danxg87 thanks for the details. can you please provide a (fake) dummy ssh key with the newline you had? we'll use this to fix the bug / validation issue

also, if you want to put up a PR to address this, we'll certainly welcome it!

KamilBlaz commented 10 months ago

@danxg87 if you will provide dummy ssh key then I would like to put up a PR

baseely commented 5 months ago

Hello @danxg87 and @KamilBlaz I'm interested to work on this task. Thanks to let me know of we have further progress :blush:

ansible / awx