search

prestodb / presto

The official home of the Presto distributed SQL query engine for big data
http://prestodb.io
Apache License 2.0
15.76k stars 5.29k forks source link

prestodb ranger integration error #17388

Open DivyasriDaka opened 2 years ago

DivyasriDaka commented 2 years ago

Currently we are trying to integrate ranger 2.1.0 with emr 5.33.1, But ended up with below error

2022-02-21T14:09:52.979Z INFO main com.facebook.presto.security.AccessControlManager -- Loading system access control -- 2022-02-21T14:09:52.979Z ERROR main com.facebook.presto.server.PrestoServer Access control ranger is not registered java.lang.IllegalStateException: Access control ranger is not registered at com.google.common.base.Preconditions.checkState(Preconditions.java:588) at com.facebook.presto.security.AccessControlManager.setSystemAccessControl(AccessControlManager.java:148) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:134) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:121) at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:155) at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:79)

rohanpednekar commented 2 years ago

@DivyasriDaka What version of presto are you using? cc: @agrawalreetika

agrawalreetika commented 2 years ago

@DivyasriDaka There is an ongoing PR for Presto and Ranger security integration with Hive Plugin. Please take a look - https://github.com/prestodb/presto/pull/16999 Let me know if you face any issues while integration

turtoise commented 2 years ago

Hi,I meet same problem, did you resolve this problem?

drummerglen commented 1 year ago

Same problem on version 0.276.1 2022-09-18T23:10:37.102+0800 INFO main com.facebook.presto.security.AccessControlManager -- Loading system access control -- 2022-09-18T23:10:37.103+0800 ERROR main com.facebook.presto.server.PrestoServer Access control ranger is not registered java.lang.IllegalStateException: Access control ranger is not registered at com.google.common.base.Preconditions.checkState(Preconditions.java:588) at com.facebook.presto.security.AccessControlManager.setSystemAccessControl(AccessControlManager.java:150) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:136) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:123) at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:169) at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:85)

agrawalreetika commented 1 year ago

Hi @drummerglen, Currently, prestodb doesn't support Ranger-Based Authorization at the catalog level. If you are looking for authorization in hive connector, You can use Ranger-Based Authorization for Hive connector by using the required configuration in your hive catalog configuration - https://prestodb.io/docs/current/connector/hive-security.html#hive-ranger-based-authorization This is supported in presto-0.275 or later. Let me know if you have any questions.

pratyakshsharma commented 1 year ago

@DivyasriDaka @drummerglen Are you still facing issues, now that the mentioned PR is merged?

drummerglen commented 1 year ago

Hi @drummerglen, Currently, prestodb doesn't support Ranger-Based Authorization at the catalog level. If you are looking for authorization in hive connector, You can use Ranger-Based Authorization for Hive connector by using the required configuration in your hive catalog configuration - https://prestodb.io/docs/current/connector/hive-security.html#hive-ranger-based-authorization This is supported in presto-0.275 or later. Let me know if you have any questions.

Hi @agrawalreetika Sorry for missing your message so long time. I have used Ranger-Based Authorization for Hive connector. It works well. But for Presto, I'm going to try it in later weeks, and I will reply to you. Thanks. BTW, does version 0.278 supported now?

agrawalreetika commented 1 year ago

YES, @drummergle presto-0.278 is supported now. https://github.com/prestodb/presto/tree/0.278

drummerglen commented 1 year ago

@agrawalreetika OOPS, I try to run presto by bin/launcher run with enabling ranger-presto-plugin, and I have configured the catalog/hive.properties file, but it still can not work. doc ref:https://prestodb.io/docs/current/connector/hive-security.html#ranger-based-authorization

prestodb version:0.278.1 hive version:3.1.3 hdfs version:3.2.4 ranger version:2.3.0

startup log

2022-12-26T18:33:06.911+0800 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Loading catalog properties etc/catalog/hive.properties -- 2022-12-26T18:33:06.913+0800 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Loading catalog hive -- 2022-12-26T18:33:10.875+0800 WARN main org.apache.hadoop.util.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2022-12-26T18:33:11.018+0800 INFO main org.apache.hadoop.io.compress.bzip2.Bzip2Factory Successfully loaded & initialized native-bzip2 library system-native 2022-12-26T18:33:11.023+0800 INFO main org.apache.hadoop.io.compress.zlib.ZlibFactory Successfully loaded & initialized native-zlib library 2022-12-26T18:33:12.034+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-audit.xml): couldn't find resource file location 2022-12-26T18:33:12.036+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-security.xml): couldn't find resource file location 2022-12-26T18:33:12.037+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-policymgr-ssl.xml): couldn't find resource file location 2022-12-26T18:33:12.037+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-prestodev-audit.xml): couldn't find resource file location 2022-12-26T18:33:12.037+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-prestodev-security.xml): couldn't find resource file location 2022-12-26T18:33:12.038+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-prestodev-policymgr-ssl.xml): couldn't find resource file location 2022-12-26T18:33:12.039+0800 INFO main org.apache.ranger.authorization.hadoop.config.RangerPluginConfig PolicyEngineOptions: { evaluatorType: auto, evaluateDelegateAdminOnly: false, disableContextEnrichers: true, disableCustomConditions: true, disableTagPolicyEvaluation: true, enableTagEnricherWithLocalRefresher: false, disableTrieLookupPrefilter: false, optimizeTrieForRetrieval: false, cacheAuditResult: false } 2022-12-26T18:33:12.053+0800 INFO main org.apache.ranger.audit.provider.AuditProviderFactory AuditProviderFactory: creating.. 2022-12-26T18:33:12.054+0800 INFO main org.apache.ranger.audit.provider.AuditProviderFactory AuditProviderFactory: initializing.. 2022-12-26T18:33:12.099+0800 INFO main org.apache.ranger.audit.provider.AuditProviderFactory No v3 audit configuration found. Trying v2 audit configurations 2022-12-26T18:33:12.101+0800 INFO Ranger async Audit cleanup org.apache.ranger.audit.provider.AuditProviderFactory RangerAsyncAuditCleanup: Waiting to audit cleanup start signal 2022-12-26T18:33:12.815+0800 INFO main com.facebook.airlift.bootstrap.LifeCycleManager Life cycle starting... 2022-12-26T18:33:12.817+0800 INFO main com.facebook.airlift.bootstrap.LifeCycleManager Life cycle startup complete. System ready. 2022-12-26T18:33:12.839+0800 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Added catalog hive using connector hive-hadoop2 -- 2022-12-26T18:33:12.880+0800 INFO main com.facebook.presto.security.AccessControlManager -- Loading system access control -- 2022-12-26T18:33:12.880+0800 ERROR main com.facebook.presto.server.PrestoServer Access control ranger is not registered java.lang.IllegalStateException: Access control ranger is not registered at com.google.common.base.Preconditions.checkState(Preconditions.java:588) at com.facebook.presto.security.AccessControlManager.setSystemAccessControl(AccessControlManager.java:153) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:139) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:126) at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:169) at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:85)

hive.properties configuration

cat etc/catalog/hive.properties

connector.name=hive-hadoop2 hive.metastore.uri=thrift://localhost:9083 hive.config.resources=/data/hadoop/hadoop-3.2.4/etc/hadoop/core-site.xml,/data/hadoop/hadoop-3.2.4/etc/hadoop/hdfs-site.xml hive.security=ranger hive.ranger.rest-endpoint=http://localhost:6080/ hive.ranger.policy.hive-servicename=prestodev hive.ranger.service.basic-auth-username=root hive.ranger.service.basic-auth-password=xxxxxxx

ENV processes

$ jps

22611 EmbeddedServer --- ranger 55316 DataNode 2724 RunJar --- hive metastore 57974 NameNode 18344 Jps
2589 RunJar --- hiveserver2

screenshots

ranger hive-plugin status

image

Access Policy

image image

agrawalreetika commented 1 year ago

Hi @drummerglen, Current Ranger Implementation is for Presto Hive Catalog, which honors the policies under the hive service in Ranger. So here in your case, you can use hive.ranger.policy.hive-servicename=hivedev in the hive.properties file.

And few question here, do you have any ranger-related config in etc/access-control.properties? If yes, then you can take it out. Since you just need the configuration given in here.

drummerglen commented 1 year ago

Hi @agrawalreetika ,

It still can not work after changing hive.ranger.policy.hive-servicename=hivedev in the hive.properties file. Ref log 1.

YES, there's only one line ranger-related config in etc/access-control.properties. you can check it below. If I take it out, it also crash in the end. Ref log 2.

log 1

2022-12-26T22:54:19.390+0800 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Loading catalog properties etc/catalog/hive.properties -- 2022-12-26T22:54:19.392+0800 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Loading catalog hive -- 2022-12-26T22:54:24.218+0800 WARN main org.apache.hadoop.util.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2022-12-26T22:54:24.475+0800 INFO main org.apache.hadoop.io.compress.bzip2.Bzip2Factory Successfully loaded & initialized native-bzip2 library system-native 2022-12-26T22:54:24.481+0800 INFO main org.apache.hadoop.io.compress.zlib.ZlibFactory Successfully loaded & initialized native-zlib library 2022-12-26T22:54:26.897+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-audit.xml): couldn't find resource file location 2022-12-26T22:54:26.904+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-security.xml): couldn't find resource file location 2022-12-26T22:54:26.905+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-policymgr-ssl.xml): couldn't find resource file location 2022-12-26T22:54:26.907+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-hivedev-audit.xml): couldn't find resource file location 2022-12-26T22:54:26.907+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-hivedev-security.xml): couldn't find resource file location 2022-12-26T22:54:26.908+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-hivedev-policymgr-ssl.xml): couldn't find resource file location 2022-12-26T22:54:26.913+0800 INFO main org.apache.ranger.authorization.hadoop.config.RangerPluginConfig PolicyEngineOptions: { evaluatorType: auto, evaluateDelegateAdminOnly: false, disableContextEnrichers: true, disableCustomConditions: true, disableTagPolicyEvaluation: true, enableTagEnricherWithLocalRefresher: false, disableTrieLookupPrefilter: false, optimizeTrieForRetrieval: false, cacheAuditResult: false } 2022-12-26T22:54:26.963+0800 INFO main org.apache.ranger.audit.provider.AuditProviderFactory AuditProviderFactory: creating.. 2022-12-26T22:54:26.966+0800 INFO main org.apache.ranger.audit.provider.AuditProviderFactory AuditProviderFactory: initializing.. 2022-12-26T22:54:27.112+0800 INFO main org.apache.ranger.audit.provider.AuditProviderFactory No v3 audit configuration found. Trying v2 audit configurations 2022-12-26T22:54:27.117+0800 INFO Ranger async Audit cleanup org.apache.ranger.audit.provider.AuditProviderFactory RangerAsyncAuditCleanup: Waiting to audit cleanup start signal 2022-12-26T22:54:28.372+0800 INFO main com.facebook.airlift.bootstrap.LifeCycleManager Life cycle starting... 2022-12-26T22:54:28.373+0800 INFO main com.facebook.airlift.bootstrap.LifeCycleManager Life cycle startup complete. System ready. 2022-12-26T22:54:28.406+0800 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Added catalog hive using connector hive-hadoop2 -- 2022-12-26T22:54:28.465+0800 INFO main com.facebook.presto.security.AccessControlManager -- Loading system access control -- 2022-12-26T22:54:28.465+0800 ERROR main com.facebook.presto.server.PrestoServer Access control ranger is not registered java.lang.IllegalStateException: Access control ranger is not registered at com.google.common.base.Preconditions.checkState(Preconditions.java:588) at com.facebook.presto.security.AccessControlManager.setSystemAccessControl(AccessControlManager.java:153) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:139) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:126) at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:169) at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:85)

config files check

[root@harbor01 presto]# ll etc/ -rw-r--r-- 1 root root 27 12月 26 18:04 access-control.properties drwxr-xr-x 2 root root 29 12月 26 22:53 catalog -rw-r--r-- 1 root root 232 12月 26 17:42 config.properties lrwxrwxrwx 1 root root 38 12月 26 17:42 etc -> /data/presto/presto-server-0.276.1/etc -rw-r--r-- 1 root root 170 12月 26 17:42 jvm.config -rw-r--r-- 1 root root 25 12月 26 17:42 log.properties -rw-r--r-- 1 root root 104 12月 26 18:30 node.properties -rwxr--r-- 1 root root 2065 12月 26 18:04 ranger-policymgr-ssl.xml -rwxr--r-- 1 root root 10852 12月 26 18:04 ranger-presto-audit.xml -rwxr--r-- 1 root root 2663 12月 26 18:04 ranger-presto-security.xml -rw-r--r-- 1 root root 83 12月 26 18:04 ranger-security.xml [root@localhost presto]# cat etc/access-control.properties access-control.name=ranger [root@localhost presto]# cat etc/catalog/hive.properties connector.name=hive-hadoop2 hive.metastore.uri=thrift://localhost:9083 hive.config.resources=/data/hadoop/hadoop-3.2.4/etc/hadoop/core-site.xml,/data/hadoop/hadoop-3.2.4/etc/hadoop/hdfs-site.xml hive.security=ranger hive.ranger.rest-endpoint=http://localhost:6080/ hive.ranger.policy.hive-servicename=hivedev [root@harbor01 presto]#

log 2

2022-12-26T23:02:54.321+0800 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Loading catalog properties etc/catalog/hive.properties -- 2022-12-26T23:02:54.324+0800 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Loading catalog hive -- 2022-12-26T23:02:58.023+0800 WARN main org.apache.hadoop.util.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2022-12-26T23:02:58.194+0800 INFO main org.apache.hadoop.io.compress.bzip2.Bzip2Factory Successfully loaded & initialized native-bzip2 library system-native 2022-12-26T23:02:58.197+0800 INFO main org.apache.hadoop.io.compress.zlib.ZlibFactory Successfully loaded & initialized native-zlib library 2022-12-26T23:02:59.023+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-audit.xml): couldn't find resource file location 2022-12-26T23:02:59.027+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-security.xml): couldn't find resource file location 2022-12-26T23:02:59.028+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-policymgr-ssl.xml): couldn't find resource file location 2022-12-26T23:02:59.028+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-hivedev-audit.xml): couldn't find resource file location 2022-12-26T23:02:59.028+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-hivedev-security.xml): couldn't find resource file location 2022-12-26T23:02:59.029+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-hivedev-policymgr-ssl.xml): couldn't find resource file location 2022-12-26T23:02:59.031+0800 INFO main org.apache.ranger.authorization.hadoop.config.RangerPluginConfig PolicyEngineOptions: { evaluatorType: auto, evaluateDelegateAdminOnly: false, disableContextEnrichers: true, disableCustomConditions: true, disableTagPolicyEvaluation: true, enableTagEnricherWithLocalRefresher: false, disableTrieLookupPrefilter: false, optimizeTrieForRetrieval: false, cacheAuditResult: false } 2022-12-26T23:02:59.052+0800 INFO main org.apache.ranger.audit.provider.AuditProviderFactory AuditProviderFactory: creating.. 2022-12-26T23:02:59.053+0800 INFO main org.apache.ranger.audit.provider.AuditProviderFactory AuditProviderFactory: initializing.. 2022-12-26T23:02:59.103+0800 INFO main org.apache.ranger.audit.provider.AuditProviderFactory No v3 audit configuration found. Trying v2 audit configurations 2022-12-26T23:02:59.105+0800 INFO Ranger async Audit cleanup org.apache.ranger.audit.provider.AuditProviderFactory RangerAsyncAuditCleanup: Waiting to audit cleanup start signal 2022-12-26T23:02:59.643+0800 INFO main com.facebook.airlift.bootstrap.LifeCycleManager Life cycle starting... 2022-12-26T23:02:59.644+0800 INFO main com.facebook.airlift.bootstrap.LifeCycleManager Life cycle startup complete. System ready. 2022-12-26T23:02:59.703+0800 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Added catalog hive using connector hive-hadoop2 -- 2022-12-26T23:02:59.748+0800 ERROR main com.facebook.presto.server.PrestoServer Access control configuration /data/presto/data/etc/access-control.properties does not contain access-control.name java.lang.IllegalArgumentException: Access control configuration /data/presto/data/etc/access-control.properties does not contain access-control.name at com.google.common.base.Preconditions.checkArgument(Preconditions.java:440) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:121) at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:169) at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:85)

It seems that there are some config files did not generate after executed enable-presto-plugin.sh by read following log

2022-12-26T22:54:26.897+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-audit.xml): couldn't find resource file location 2022-12-26T22:54:26.904+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-security.xml): couldn't find resource file location 2022-12-26T22:54:26.905+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-policymgr-ssl.xml): couldn't find resource file location 2022-12-26T22:54:26.907+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-hivedev-audit.xml): couldn't find resource file location 2022-12-26T22:54:26.907+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-hivedev-security.xml): couldn't find resource file location 2022-12-26T22:54:26.908+0800 ERROR main org.apache.ranger.authorization.hadoop.config.RangerConfiguration addResourceIfReadable(ranger-hive-hivedev-policymgr-ssl.xml): couldn't find resource file location

agrawalreetika commented 1 year ago

@drummerglen Thanks for sharing the details. Please remove etc/access-control.properties from your config. That's not required. YOu can ignore audit-related ERROR messages from the Ranger side when looking for audit config files.

drummerglen commented 1 year ago

@agrawalreetika Thank you for your guidance! Now presto can start successfully. But it still can't access data and prompt an error below. I did the following steps:

  1. disable ranger-presto-plugin
  2. config etc/catalog/hive.properties using hivedev
  3. start presto using bin/launch run (for Observing log)
  4. Using the DBeaver client connect the presto server without inputting a password cause if I enter a password it will prompt Authentication using username/password requires SSL to be enabled image

Here is the log on server:

2023-01-13T18:06:03.974+0800 INFO Ranger async Audit cleanup org.apache.ranger.audit.provider.AuditProviderFactory RangerAsyncAuditCleanup: Waiting to audit cleanup start signal 2023-01-13T18:06:05.374+0800 INFO main com.facebook.airlift.bootstrap.LifeCycleManager Life cycle starting... 2023-01-13T18:06:05.376+0800 INFO main com.facebook.airlift.bootstrap.LifeCycleManager Life cycle startup complete. System ready. 2023-01-13T18:06:05.397+0800 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Added catalog hive using connector hive-hadoop2 -- 2023-01-13T18:06:05.448+0800 INFO main com.facebook.presto.security.AccessControlManager -- Loading system access control -- 2023-01-13T18:06:05.451+0800 INFO main com.facebook.presto.security.AccessControlManager -- Loaded system access control allow-all -- 2023-01-13T18:06:05.457+0800 INFO main com.facebook.presto.storage.TempStorageManager -- Loading temp storage local -- 2023-01-13T18:06:05.477+0800 INFO main com.facebook.presto.storage.TempStorageManager -- Loaded temp storage local -- 2023-01-13T18:06:05.537+0800 INFO main com.facebook.presto.server.PrestoServer ======== SERVER STARTED ======== 2023-01-13T18:06:23.868+0800 INFO dispatcher-query-2 com.facebook.presto.event.QueryMonitor TIMELINE: Query 20230113_100618_00000_7jzvq :: Transaction:[fcbfa3c0-553a-4efc-a27b-9437d5784f92] :: elapsed 3787ms :: planning 1152ms :: scheduling 1947ms :: running 323ms :: finishing 365ms :: begin 2023-01-13T18:06:19.375+08:00 :: end 2023-01-13T18:06:23.162+08:00 2023-01-13T18:06:24.045+0800 INFO dispatcher-query-1 com.facebook.presto.event.QueryMonitor TIMELINE: Query 20230113_100623_00001_7jzvq :: Transaction:[2363b229-6a1d-4ba0-ad07-478b005d4627] :: elapsed 679ms :: planning 90ms :: scheduling 142ms :: running 186ms :: finishing 261ms :: begin 2023-01-13T18:06:23.278+08:00 :: end 2023-01-13T18:06:23.957+08:00 2023-01-13T18:06:28.600+0800 ERROR SplitRunner-13-120 com.facebook.presto.execution.executor.TaskExecutor Error processing Split 20230113_100625_00002_7jzvq.2.0.0-0 SystemSplit{connectorId=$system@system, tableHandle=$system@system:jdbc.schemas, addresses=[localhost:8080]} (start = 1.923611779217307E9, wall = 1032 ms, cpu = 0 ms, wait = 3 ms, calls = 1): HIVE_RANGER_SERVER_ERROR: Unable to fetch user groups information from ranger 2023-01-13T18:06:28.666+0800 ERROR remote-task-callback-10 com.facebook.presto.execution.StageExecutionStateMachine Stage execution 20230113_100625_00002_7jzvq.2.0 failed com.facebook.presto.spi.PrestoException: Unable to fetch user groups information from ranger at com.facebook.presto.hive.security.ranger.RangerBasedAccessControl.getGroupsForUser(RangerBasedAccessControl.java:195) at com.facebook.presto.hive.security.ranger.RangerBasedAccessControl.filterSchemas(RangerBasedAccessControl.java:269) at com.facebook.presto.hive.security.SystemTableAwareAccessControl.filterSchemas(SystemTableAwareAccessControl.java:77) at com.facebook.presto.security.AccessControlManager.filterSchemas(AccessControlManager.java:285) at com.facebook.presto.metadata.MetadataListing.listSchemas(MetadataListing.java:59) at com.facebook.presto.connector.system.jdbc.SchemaJdbcTable.cursor(SchemaJdbcTable.java:74) at com.facebook.presto.connector.system.SystemPageSourceProvider$1.cursor(SystemPageSourceProvider.java:130) at com.facebook.presto.split.MappedRecordSet.cursor(MappedRecordSet.java:53) at com.facebook.presto.spi.RecordPageSource.(RecordPageSource.java:40) at com.facebook.presto.connector.system.SystemPageSourceProvider.createPageSource(SystemPageSourceProvider.java:109) at com.facebook.presto.spi.connector.ConnectorPageSourceProvider.createPageSource(ConnectorPageSourceProvider.java:52) at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:80) at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:250) at com.facebook.presto.operator.Driver.processInternal(Driver.java:426) at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:309) at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:730) at com.facebook.presto.operator.Driver.processFor(Driver.java:302) at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1079) at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:166) at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:599) at com.facebook.presto.$gen.Presto_0_278_1_ec67ba1____20230113_100542_1.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.facebook.airlift.http.client.UnexpectedResponseException: Expected response code to be [200, 201, 202, 203, 204, 205, 206], but was 401: Unauthorized at com.facebook.airlift.http.client.JsonResponseHandler.handle(JsonResponseHandler.java:71) at com.facebook.airlift.http.client.jetty.JettyHttpClient.execute(JettyHttpClient.java:529) at com.facebook.presto.hive.security.ranger.RangerBasedAccessControl.getUsers(RangerBasedAccessControl.java:146) at com.facebook.presto.hive.security.ranger.RangerBasedAccessControl.getUserGroupsMappings(RangerBasedAccessControl.java:179) at com.facebook.presto.hive.security.ranger.RangerBasedAccessControl.lambda$new$1(RangerBasedAccessControl.java:105) at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:241) at com.facebook.presto.hive.security.ranger.RangerBasedAccessControl.getGroupsForUser(RangerBasedAccessControl.java:192) ... 23 more`

and here is the screenshot on DBeaver client:

image
330570902 commented 1 year ago

any progress? I have the same issues.

BigDataDZ commented 9 months ago

2023-09-20T15:16:48.892+0800 INFO main com.facebook.presto.security.AccessControlManager -- Loading system access control -- 2023-09-20T15:16:48.892+0800 ERROR main com.facebook.presto.server.PrestoServer Access control ranger is not registered java.lang.IllegalStateException: Access control ranger1 is not registered at com.google.common.base.Preconditions.checkState(Preconditions.java:588) at com.facebook.presto.security.AccessControlManager.setSystemAccessControl(AccessControlManager.java:154) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:140) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:127) at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:170) at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:86)

What is the general reason for the above error?

ranger_version:2.1.0 prestodb_version:0.282

[root@ddp1 etc]# cat access-control.properties access-control.name=ranger

Is it because Ranger does not support prestodb?

agrawalreetika commented 9 months ago

Hi @BigDataDZ , Currently, prestodb doesn't support Ranger-Based Authorization at the catalog level. The one you are trying to configure is at System Access Control level, which is not supported. If you are looking for authorization in the hive connector, You can use Ranger-Based Authorization for the Hive connector by using the required configuration in your hive catalog configuration - https://prestodb.io/docs/current/connector/hive-security.html#hive-ranger-based-authorization

Let me know if you have any questions.