leopaul36
commented
1 year ago We have yet to implement the automatic creation of the policy allowing tdp_user to use the Knox topology. This needs to be done via the ranger_policies variable at inventory/group_vars/all.yml feel free to submit a PR :)
In the meantime, you can manually add tdp_user to the Ranger Knox policy via the Ranger UI.
The Permission denied: user=knox, access=WRITE, inode="/ranger/audit":hdfs:supergroup:drwxr-xr-x logs are not related to the issue. I created https://github.com/TOSIT-IO/tdp-collection/issues/539 for tracking of this issue.
gboutry
After entire installation, the knox access is not working. Connecting on https://edge-01.tdp:8443/gateway/tdpldap/hdfs The user/pwd is asked : tdp_user / tdp_user123
And the result is :
There are plenty of following error in the Knox log file :
2022-11-10 15:49:53,499 INFO destination.HDFSAuditDestination: Flushing HDFS audit. Event Size:1 1792022-11-10 15:49:53,499 ERROR queue.AuditFileSpool: Error sending logs to consumer. provider=knox.async.multi_dest.batch, consumer=knox.async.multi_dest.batch.hdfs 7092022-11-10 15:49:53,499 INFO queue.AuditFileSpool: Destination is down. sleeping for 30000 milli seconds. indexQueue=0, queueName=knox.async.multi_dest.batch, consumer=knox.async.multi_dest.batch.hdfs 7692022-11-10 15:50:53,500 INFO provider.BaseAuditHandler: Audit Status Log: name=knox.async.multi_dest.batch.hdfs, interval=01:00.018 minutes, events=1, deferredCount=1, totalEvents=47, totalDeferredCount=47 3122022-11-10 15:50:53,504 INFO destination.HDFSAuditDestination: Adding property to HDFS config: fs.azure.account.key.REPLACE_AZURE_ACCOUNT_NAME.blob.core.windows.net => __REPLACE_AZURE_ACCOUNT_KEY 3222022-11-10 15:50:53,504 INFO destination.HDFSAuditDestination: Adding property to HDFS config: fs.azure.account.keyprovider.REPLACE_AZURE_ACCOUNT_NAME.blob.core.windows.net => REPLACE_AZURE_ACCOUNT_KEY_PROVIDER 3222022-11-10 15:50:53,504 INFO destination.HDFSAuditDestination: Adding property to HDFS config: fs.azure.shellkeyprovider.script => REPLACE_AZURE_SHELL_KEY_PROVIDER 3222022-11-10 15:50:53,504 INFO destination.HDFSAuditDestination: Returning HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml 3252022-11-10 15:50:53,504 INFO destination.HDFSAuditDestination: Checking whether log file exists. hdfPath=hdfs://mycluster/ranger/audit/knox/20221110/knox/20221110/knox_ranger_audit_edge-01.log, UGI=knox/edge-01.tdp@REALM.TDP (auth:KERBEROS) 2872022-11-10 15:50:53,521 INFO destination.HDFSAuditDestination: Log file doesn't exists. Will create and use it. hdfPath=hdfs://mycluster/ranger/audit/knox/20221110/knox/20221110/knox_ranger_audit_edge-01.log 2992022-11-10 15:50:53,521 INFO destination.HDFSAuditDestination: Creating parent folder for hdfs://mycluster/ranger/audit/knox/20221110/knox/20221110/knox_ranger_audit_edge-01.log 3312022-11-10 15:50:53,524 ERROR provider.BaseAuditHandler: Error writing to log file. 329org.apache.hadoop.security.AccessControlException: Permission denied: user=knox, access=WRITE, inode="/ranger/audit":hdfs:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:255) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:588) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:350) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1834) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1793) at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:59) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3135) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1126) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:707) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=knox, access=WRITE, inode="/ranger/audit":hdfs:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:255) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:588) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:350) at org.apache.hadoop.hdfs.server.nameno