配置Hadoop KMS服务

[liusheng]

参考文档： Hadoop官方文档 这里我配置用的SM4作为KMS provider，因为这里主要记录调试Hadoop SM4支持特性的代码

1. 配置etc/hadoop/kms-site.xml文件：

<configuration>
  <property>
     <name>hadoop.kms.key.provider.uri</name>
     <value>jceks://file@/${user.home}/kms.keystore</value>
  </property>
  <property>
    <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
    <value>kms.keystore.password</value>
  </property>
  <property>
     <name>dfs.encryption.key.provider.uri</name>
     <value>kms://http@localhost:9600/kms</value>
  </property>
  <property>
     <name>hadoop.kms.authentication.type</name>
     <value>simple</value>
  </property>
</configuration>

2. 配置etc/hadoop/core-site.xml文件：

    <property>
      <name>hadoop.security.key.provider.path</name>
      <value>kms://http@localhost:9600/kms</value>
    </property>
    <property>
      <name>hadoop.security.crypto.cipher.suite</name>
      <value>SM4/CTR/NoPadding</value>
    </property>
    <property>
      <name>hadoop.security.key.default.cipher</name>
      <value>SM4/CTR/NoPadding</value>
    </property>
    <property>
      <name>hadoop.security.crypto.jce.provider</name>
      <value>BC</value>
    </property>

3. 创建keystore文件：

keytool -genkey -alias 'kmskey' -keystore ~/kms.keystore -dname "CN=localhost, OU=localhost, O=localhost, L=SH, ST=SH, C=CN" -keypass 123456 -storepass 123456

4. 创建加密key的密码文件：

echo 123456 > etc/hadoop/kms.keystore.password

这里要注意的是，文件的名称需要和上面的配置文件中配置项对应起来。

5. 启动KMS服务：

hadoop [--daemon start|status|stop] kms

6. 验证

hadoop key create key1 -cipher 'SM4/CTR/NoPadding'
hdfs dfs -mkdir /testkms
hdfs crypto -createZone -keyName key1 -path /testkms
touch aaa
hadoop fs -put aaa /testkms

[liusheng]

问题记录：

2020-07-23 10:57:31,872 INFO  Server - jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 1.8.0_252-8u252-b09-1~18.04-b09
2020-07-23 10:57:31,899 INFO  session - DefaultSessionIdManager workerName=node0
2020-07-23 10:57:31,899 INFO  session - No SessionScavenger set, using defaults
2020-07-23 10:57:31,901 INFO  session - node0 Scavenging every 660000ms
2020-07-23 10:57:31,912 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@5bf0d49{logs,/logs,file:///opt/hadoop-3.4.0-SNAPSHOT/logs/,AVAILABLE}
2020-07-23 10:57:31,913 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@7c7a06ec{static,/static,jar:file:/opt/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/hadoop-kms-3.4.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
2020-07-23 10:57:31,986 INFO  TypeUtil - JVM Runtime does not support Modules
2020-07-23 10:57:32,015 INFO  KMSWebApp - -------------------------------------------------------------
2020-07-23 10:57:32,015 INFO  KMSWebApp -   Java runtime version : 1.8.0_252-8u252-b09-1~18.04-b09
2020-07-23 10:57:32,015 INFO  KMSWebApp -   User: hadoop
2020-07-23 10:57:32,015 INFO  KMSWebApp -   KMS Hadoop Version: 3.4.0-SNAPSHOT
2020-07-23 10:57:32,015 INFO  KMSWebApp - -------------------------------------------------------------
2020-07-23 10:57:32,023 INFO  KMSACLs - 'CREATE' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'DELETE' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'ROLLOVER' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'GET' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'GET_KEYS' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'GET_METADATA' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'SET_KEY_MATERIAL' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'GENERATE_EEK' ACL '*'
2020-07-23 10:57:32,024 INFO  KMSACLs - 'DECRYPT_EEK' ACL '*'
2020-07-23 10:57:32,025 INFO  KMSACLs - default.key.acl. for KEY_OP 'READ' is set to '*'
2020-07-23 10:57:32,025 INFO  KMSACLs - default.key.acl. for KEY_OP 'MANAGEMENT' is set to '*'
2020-07-23 10:57:32,025 INFO  KMSACLs - default.key.acl. for KEY_OP 'GENERATE_EEK' is set to '*'
2020-07-23 10:57:32,025 INFO  KMSACLs - default.key.acl. for KEY_OP 'DECRYPT_EEK' is set to '*'
2020-07-23 10:57:32,080 INFO  KMSAudit - Initializing audit logger class org.apache.hadoop.crypto.key.kms.server.SimpleKMSAuditLogger
2020-07-23 10:57:32,537 INFO  KMSWebServer - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down KMSWebServer at hadoop-benchmark/172.17.0.2

https://github.com/eclipse/jetty.project/issues/4064

升级Hadoop中jetty的依赖版本:

    <jetty.version>9.4.20.v20190813</jetty.version>

[FlowerBirds]

使用keytool生成的jks，查看key：

[root@tianhe-space-station hadoop-3.3.1]# hadoop key list
kmskey

默认会有kmskey，如果查询，metadata的话，会报错：

[root@tianhe-space-station hadoop-3.3.1]# hadoop key list -metadata
Listing keys for KeyProvider: org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@fa36558
2021-06-23 14:49:01,989 WARN kms.LoadBalancingKMSClientProvider: KMS provider at [http://localhost:9600/kms/v1/] threw an IOException:
java.io.IOException: Can't cast key for mykey in keystore file:/opt/data/hadoop-3.3.1/kms.jks to a KeyMetadata. Key may have been added using  keytool or some other non-Hadoop method.

kms后端报错：

java.io.IOException: Can't cast key for mykey in keystore file:/opt/data/hadoop-3.3.1/kms.jks to a KeyMetadata. Key may have been added using  keytool or some other non-Hadoop method.
        at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.getMetadata(JavaKeyStoreProvider.java:415)
        at org.apache.hadoop.crypto.key.CachingKeyProvider$CacheExtension$2.load(CachingKeyProvider.java:65)
.....
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException: sun.security.provider.DSAPrivateKey cannot be cast to org.apache.hadoop.crypto.key.JavaKeyStoreProvider$KeyMetadata
        at org.apache.hadoop.crypto.key.JavaKeyStoreProvider.getMetadata(JavaKeyStoreProvider.java:411)
        ... 93 more

看意思，keytool生成的和hadoop生成的key格式不一样，无法解析

[FlowerBirds]

如果在kms-site.xml中啥也不配置，启动kms后，查看key为空，创建key后去查看metadata，是可以的