Communication with receptor fails: could not load signing key file: unknown block type PRIVATE KEY

edudobay commented 2 years ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that AWX is open source software provided for free and that I might not receive a timely response.

Summary

I've installed AWX in a test server using the instructions from https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md, and the application fails to communicate with the receptor instances — messages like these show up in the log and in the instance status page: Remote error: ERROR: could not load signing key file: unknown block type PRIVATE KEY.

AWX version

21.0.0

Select the relevant components

[ ] UI
[ ] API
[ ] Docs

Installation method

docker on linux

Modifications

no

Ansible version

2.12.5 (ansible-core) 5.8.0 (ansible)

Operating system

Ubuntu 22.04

Web browser

No response

Steps to reproduce

What I did for the setup:

Installed Docker (official installation steps for Ubuntu), Docker Compose and Ansible (sudo pip install docker-compose ansible)
git clone -b 21.0.0 https://github.com/ansible/awx.git; cd awx
make docker-compose-build
make docker-compose COMPOSE_UP_OPTS=-d
build the UI with docker exec tools_awx_1 make clean-ui ui-devel
configured the user with docker exec -ti tools_awx_1 awx-manage createsuperuser

Expected results

All instances should be healthy.

Actual results

Both receptor-1 and receptor-2 have an error state in the Instances page: https://localhost:8043/#/instances

Additional information

Log output from docker logs tools_awx_1:

2022-05-23 05:10:45,397 WARNING  [f6876450] awx.main.tasks.system Hop node receptor-hop, has rejoined the receptor mesh
2022-05-23 05:10:45,439 WARNING  [f6876450] awx.main.tasks.system Execution node attempting to rejoin as instance receptor-1.
2022-05-23 05:10:45,499 WARNING  [f6876450] awx.main.tasks.system Execution node attempting to rejoin as instance receptor-2.
ERROR 2022/05/23 05:10:45 could not load signing key file: unknown block type PRIVATE KEY
2022-05-23 05:10:45,633 INFO     [f6876450] awx.main.tasks.system Failed to find capacity of new or lost execution node receptor-1, errors:
Remote error: ERROR: could not load signing key file: unknown block type PRIVATE KEY
ERROR 2022/05/23 05:10:45 could not load signing key file: unknown block type PRIVATE KEY
2022-05-23 05:10:45,663 INFO     [f6876450] awx.main.tasks.system Failed to find capacity of new or lost execution node receptor-2, errors:
Remote error: ERROR: could not load signing key file: unknown block type PRIVATE KEY

justbyBen commented 2 years ago

Any updates?

justbyBen commented 2 years ago

@edudobay FYI: I fixed by generating a new cert with open ssl and replaced the default pub and priv key with the new ones. (On AWX: /opt/awx/tools/docker-compose/_sources/receptor//)

Ashish1981 commented 2 years ago

@justbyBen I tried to create as mentioned on task with 4096 bit, but still not worked for me. which exact type key to be created?

justbyBen commented 2 years ago

OpenSSL rsa 4096 and then converting to pem format

Ashish1981 commented 2 years ago

OpenSSL rsa 4096 and then converting to pem format

Can you please provide me the commands, I tried various as you mentioned, still no success.

edudobay commented 2 years ago

Unfortunately that didn't work for me. I ran the following commands and then restarted the awx and receptor services:

cd tools/docker-compose/_sources/receptor/
openssl genrsa -out work_private_key.pem 4096
openssl rsa -in work_private_key.pem -pubout -out work_public_key.pem

Ashish1981 commented 2 years ago

@justbyBen clould you please provide the commands for us, by how you created the keys that worked for you.

fosterseth commented 2 years ago

docker-compose runs the following command to generate the work signing keys

openssl genrsa -out {{ work_sign_private_keyfile }} {{ receptor_rsa_bits }}

openssl rsa -in {{ work_sign_private_keyfile }} -out {{ work_sign_public_keyfile }} -outform PEM -pubout

should mostly work out of the box, so it is strange you are getting bad PEM files that receptor cannot read.

which OS and openssl version are you using?

[sbf@fedora ~]$ openssl version
OpenSSL 1.1.1n  FIPS 15 Mar 2022

Ashish1981 commented 2 years ago

I am trying with both x86_64(wsl2) and aarch64 with OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

edudobay commented 2 years ago

I can't tell you the exact version because I had launched a temporary instance to experiment with. But it was Ubuntu 22.04 with the latest package upgrades by that date.

Which component/library reads the signing key? I tried to google some parts of this error message, but couldn't find anything:

could not load signing key file: unknown block type PRIVATE KEY

fosterseth commented 2 years ago

@edudobay this is surfacing up from receptor itself https://github.com/ansible/receptor/blob/dba716a031f8ce7547d8e967f57a02f9ce27a197/pkg/workceptor/workceptor.go#L163

edudobay commented 2 years ago

I have the exact same OpenSSL version. The default key output format for openssl genrsa was changed. Running openssl genrsa -traditional reverts to the old format, and the health check passes.

OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

Should this still be fixed or documented somehow?

fosterseth commented 2 years ago

@edudobay that is helpful info

@Zokormazo @shanemcd do you have a suggestions on solving this, may have downstream implications as well. Wondering if using ansible module would help here https://docs.ansible.com/ansible/2.5/modules/openssl_privatekey_module.html

Ashish1981 commented 2 years ago

I installed OpenSSL 1.1.1p 21 Jun 2022 as described on Open SSL. Then it worked.

Zokormazo commented 2 years ago

openssl genrsa -traditional -out key.key 4096

This creates traditional formated keys with openssl 3+

Zokormazo commented 2 years ago

@edudobay that is helpful info

@Zokormazo @shanemcd do you have a suggestions on solving this, may have downstream implications as well. Wondering if using ansible module would help here https://docs.ansible.com/ansible/2.5/modules/openssl_privatekey_module.html

openssl_privatekey:
  format: pkcs1

Same effect as -traditional

SckyzO commented 2 years ago

-traditional
openssl genrsa -traditional -out work_private_key.pem 4096
it solved the same issue, after rebuild awx docker-compose
make docker-compose-sources
make docker-compose-build
make docker-compose

gainskills commented 2 years ago

the solution for me is (awx version: 21.7.0)

edit tools/docker-compose/ansible/roles/sources/tasks/main.yml
search openssl
add -traditional param to the command

chen1i commented 2 years ago

adding -traditional is no use , I am deploying on Ubuntu 22.04

ii  libgnutls-openssl27:amd64             3.7.3-4ubuntu1.1                        amd64        GNU TLS library - OpenSSL wrapper
ii  libxmlsec1-openssl:amd64              1.2.33-1build2                          amd64        Openssl engine for the XML security library
ii  openssl                               3.0.2-0ubuntu1.7                        amd64        Secure Sockets Layer toolkit - cryptographic utility
ii  python3-openssl                       21.0.0-1                                all          Python 3 wrapper around the OpenSSL library

ansible / awx