search

cloudera-labs / cloudera.cluster

An Ansible collection for lifecycle and management of Cloudera CDP Private Cloud resources on bare metal, IaaS, and PaaS.
Apache License 2.0
32 stars 46 forks source link

AES Encryption Issue with YARN Service in Cloudera CDP Private Cloud 7.1.9.3 & freeIPA #217

Open bacikerik opened 3 months ago

bacikerik commented 3 months ago

I recently deployed a new test cluster using Cloudera CDP Private Cloud version 7.1.9.3 (later with hotfix 7.1.9.4) with Cloudera Manager version 7.11.3. The deployment includes a configuration using freeIPA for secure authentication:

$ ansible-galaxy collection list freeipa.ansible_freeipa

# /home/ebacik/.ansible/collections/ansible_collections
Collection              Version
----------------------- -------
freeipa.ansible_freeipa 1.12.1

However, I was encountering an issue with the YARN service related to AES encryption, specifically 'aes256-cts-hmac-sha384-192' encryption type, which caused problems with service principals:

WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - AES256 CTS mode with HMAC SHA384-192)

Based on the log message, I found the following article and tried setting permitted_enctypes = aes256-cts aes128-cts in /etc/krb5.conf, which didn't help at all.

After further investigation, I found out that the configuration is managed in the following file:

[root@ipa1 ~]# readlink -f /etc/krb5.conf.d/crypto-policies
/usr/share/crypto-policies/DEFAULT/krb5.txt
[root@ipa1 ~]# cat /usr/share/crypto-policies/DEFAULT/krb5.txt
[libdefaults]
# permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac
permitted_enctypes = aes256-cts aes128-cts

Solution/Executed steps:

  1. Stop Cluster

  2. Stop CMS

  3. modify /usr/share/crypto-policies/DEFAULT/krb5.txt (contains the same enctypes as hdfs.keytab), comment out the original value and set 'aes256-cts aes128-cts' only:

    [root@ipa ~]# readlink -f /etc/krb5.conf.d/crypto-policies
/usr/share/crypto-policies/DEFAULT/krb5.txt
[root@ipa ~]# cat /usr/share/crypto-policies/DEFAULT/krb5.txt
[libdefaults]
# permitted_enctypes = aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 camellia256-cts-cmac camellia128-cts-cmac
permitted_enctypes = aes256-cts aes128-cts

  4. ipactl restart

  5. CM -> Administration -> Security -> Select all principals -> Regenerate Selected

  6. Start CMS

  7. Start Cluster

Conclusion

The configuration using freeIPA playbooks manage permitted_enctypes in /usr/share/crypto-policies/DEFAULT/krb5.txt instead of /etc/krb5.conf as assumed.

Environment Details:

clevesque commented 3 months ago

What version of RHEL? I have seen this issue in RHEL 8.9 (did not see this in RHEL8.8...where the change in crypto policies & SHA1 hash seems caused kerberos issues I moved on to RHEL 9.1 and did NOT see this side effect

bacikerik commented 3 months ago

RHEL 8.9, as you mentioned:

# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.9 (Ootpa)
clevesque commented 3 months ago

As of today, Cloudera has not yet done any product testing/certification on RHEL 8.9 Usually RHEL minor upgrades dont impact us so much, but this one def breaks YARN and I don't have any workaround.

See: https://supportmatrix.cloudera.com/ for the official matrix