prestodb / presto

The official home of the Presto distributed SQL query engine for big data
http://prestodb.io
Apache License 2.0
15.89k stars 5.32k forks source link

Feature Request: Ranger authorization integration #8980

Open HariSekhon opened 6 years ago

HariSekhon commented 6 years ago

Feature Request to add Apache Ranger integration support for Presto.

This will solve authorization very nicely for most people who are running the widely used standard open source Hortonworks Hadoop platform and provide granular access controls down to the column level, as well as integrating to the single-pane-of-glass for security configuration and auditing across all data access components on a Hadoop cluster.

Also, is there any chance this project could go in to the Apache Foundation? I think this combined with Ranger integration would massively increase Presto's market, especially if Hortonworks were to adopt it (they usually require their integrated components to be in the Apache Foundation).

Great work so far on this distributed SQL engine btw! :)

HariSekhon commented 6 years ago

This could perhaps piggy back to use Hive policies in Ranger or else copy what Hive has to a separately managed policy for Presto. There are trade-offs to both styles, so perhaps make this a configurable user choice.

illion20 commented 6 years ago

I am trying to implement authorization for any connector. For that to work I need to access the resources (columns etc) for the connector. I have been trying to find where in the source you have access to the connector metadata (tables, columns, connection detail etc) and the query information before execution and it seems all of this is available to you in the SQLQueryExecution class in the start function.

https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/execution/SqlQueryExecution.java

After you have analyzed the query and gotten all the connector handles.

Would this be a good place to perform an authorization check if we can access the necessary resources (column granularity). It feels like its too late in the Presto process to reject a request but who knows. Anyone who can point me in the right direction?

Thank you

mwacc commented 6 years ago

any updates or plans where integration will be completed?

RameshByndoor commented 6 years ago

Hi, I could add a plugin to presto with Ranger(Registered presto as new ranger service with catalog->schema->table->column as resources) and all those methods implemented from SystemAccessControl are working fine. I would need to refractor and would like to bring it to here.

Need some help with addressing column level security: Does com.facebook.presto.sql.analyzer.StatementAnalyzer has any placeholder for identifying column resources used in query to apply column level security check.? Currently i am looking at this method analyzeSelect( https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/sql/analyzer/StatementAnalyzer.java#L1710

Any hint around this area would be much appreciated.

kokosing commented 6 years ago

What about using https://github.com/prestodb/presto/blob/master/presto-spi/src/main/java/com/facebook/presto/spi/security/SystemAccessControl.java#L221

RameshByndoor commented 6 years ago

Thanks @kokosing. that helps. My bad I was checking with v201. Need a clarification on Set<String> columnNames passed to checkCanSelectFromColumns method.
For the below query, I'm expecting object_name alone in columnNames. but as of now it's object_name_alias and object_name. Can this be filtered with the help of metadata in AccessControlManager before making call.

select object_name as object_name_alias from "java.lang:type=memory" mem;

https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/security/AccessControlManager.java#L598

kokosing commented 6 years ago

To me it sounds like a bug?

CC: @rschlussel2 Is this expected?

tooptoop4 commented 6 years ago

@RameshByndoor can u share the code?

kokosing commented 6 years ago

@RameshByndoor Are you going to base your work on top of https://github.com/prestodb/presto/pull/10904?

rschlussel commented 6 years ago

@RameshByndoor can you give an example query where you're seeing the alias and object name. I can take a look.

RameshByndoor commented 6 years ago

@rschlussel2 you can use simple command as select object_name as object_name_alias from "jmx.current.java.lang:type=memory"; & you can catch this getting called from here. https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/security/AccessControlManager.java#L598

RameshByndoor commented 6 years ago

@rschlussel can you help me with the above.? Is it expected or how to patch it.?

InfyNord commented 6 years ago

I want to implement ranger plugin for presto .. could you please point to the branch which I can refer and clone and test

shubhamtagra commented 6 years ago

@RameshByndoor are you working on this actively? We have a need to add this support and were planning to take up this work up unless someone else is already working on it. Please let me know.

shubhamtagra commented 6 years ago

@RameshByndoor the issue you mentioned does seem like a bug and is easily reproducible. While @rschlussel is looking at it you can unblock by filtering columns which are not identifiers, something like this in Analyzer.analyze:

+        List<String> identifiers = analysis
+                .getColumnReferences()
+                .stream()
+                .filter(nodeRef -> nodeRef.getNode() instanceof Identifier)
+                .map(nodeRef -> ((Identifier) nodeRef.getNode()).getValue())
+                .collect(Collectors.toList());
         analysis.getTableColumnReferences().forEach((accessControlInfo, tableColumnReferences) ->
                 tableColumnReferences.forEach((tableName, columns) ->
                         accessControlInfo.getAccessControl().checkCanSelectFromColumns(
                                 session.getRequiredTransactionId(),
                                 accessControlInfo.getIdentity(),
                                 tableName,
-                                columns)));
+                                columns.stream().filter(column -> identifiers.contains(column)).collect(Collectors.toSet()))));
rschlussel commented 6 years ago

Thanks for the reminder- I'd forgotten about this. I just put up a PR to fix it #11295

RameshByndoor commented 6 years ago

@stagraqubole The code is similar in both the cases. Its a matter of where we plug it. According to me the trade off between SystemAccessControl and ConnectorAccessControl is .

SystemAccessControl would work regardless of all the underlying connectors of presto. Policies will be maintained under presto service in Ranger making presto as a main query engine. Using ConnectorAccessControl can make use of the existing policies defined. This helps presto be a parallel query engine which points to existing policies. For eg hive policies defined in ranger. Another point about using ConnectorAccessControl is, with multiple connectors enabled then multiple Policy engines of ranger will be created, which are inherently heavy(As per https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741207).

shubhamtagra commented 6 years ago

Taking example of Hive connector which would be the biggest consumer of Ranger support, doing it in SystemAccessControl would mean that you will maintain two Ranger policies now: one for Hive and one for Presto which would get difficult to maintain. Doing it in ConnectorAccessControl allows a single Policy holder in Ranger and that would be the one created for Hive, very similar to having SqlStandardAccessControl as ConnectorAccessControl rather than SystemAccessControl.

nmadhire commented 6 years ago

@rschlussel Does your PR fix #11295 has a support for Ranger in Presto now?

rschlussel commented 6 years ago

no. it unblocks @RameshByndoor who is working on adding support.

saravsathyamoorthy commented 6 years ago

im interested in this ranger plugin for presto.. can you give some branch which i can take and try to test

RameshByndoor commented 5 years ago

We are on test cases nd other sanity checks, Not yet ready to PR.

gray-eb commented 5 years ago

Any update on timeline? I appreciate the work you've done for this plugin. Would love to get my hands on it.

cryptoe commented 5 years ago

https://docs.google.com/document/d/1Jtapmwkp1Up_w6w_3dUeOXfLPLRAsglbqoIWNRG-NJM/edit#

This is what me and @RameshByndoor have done. We are testing the code at this moment. I can't commit any hard timelines but we should be in a state to release it next week.

We have also handled https://github.com/prestodb/presto/issues/10996 as part of the ranger presto integration.

Features supported :

  1. Multiple catalog support
  2. Able to reuse existing policies
  3. Row based + Column based filters supported
gray-eb commented 5 years ago

This may not be the best place for a question, but given the lack of support for this specific feature, I figured I'll give it a shot.

@cquptEthan or any one else? Have you gotten SSL working with this custom plugin? We don't have the CredentialProvider API setup for keystore passwords, and I'm at a loss on how to get this working without it.

From what I can tell, the properties I need to have available with an https Ranger endpoint are: xasecure.policymgr.clientssl.keystore= xasecure.policymgr.clientssl.truststore= xasecure.policymgr.clientssl.keystore.credential.file= xasecure.policymgr.clientssl.truststore.credential.file=

The plugin then expects a value for hadoop.security.credential.provider.path to interrogate the CredentialProvider API for a password of the keystore/truststore based on the last two properties above. Is this understanding correct? How can I get around using the CredentialProvider API?

cquptEthan commented 5 years ago

@gray-eb It seems that you want to add a ranger-policymgr-ssl.xml file in presto ranger plugin, but the presto plugin can't find the xml file, am i right? In my implements, I read the ranger configs from access-control.properties that begin with 'ranger.'. Like this.

 @Override
    public SystemAccessControl create(Map<String, String> config)
    {
        RangerConfiguration rangerConfig = RangerConfiguration.getInstance();
        try {
            handleKerberos(rangerConfig, config);
        }
        catch (IOException e) {
            throw new PrestoException(StandardErrorCode.GENERIC_INTERNAL_ERROR, "Failed to do kerberos right", e);
        }
        for (final Map.Entry<String, String> configEntry : config.entrySet()) {
            if (configEntry.getKey().startsWith("ranger.")) {
                rangerConfig.set(configEntry.getKey(), configEntry.getValue());
                log.info("Setting: " + configEntry.getKey() + " to: " + configEntry.getValue());
            }
        }

        PrestoAuthorizer authorizer = getPrestoAuthorizer(config);
        return new RangerSystemAccessControl(authorizer, config);
    }

Then you can add this config in access-control.properties to define your ssl config xml. ranger.plugin.\<ServiceType>.policy.rest.ssl.config.file=path/to/ranger-policymgr-ssl.xml

tooptoop4 commented 5 years ago

@cquptEthan is this ranger authorization being built into presto only going to work for kerberos?

cquptEthan commented 5 years ago

@tooptoop4 No. LDAP Authentication and other implement of PasswordAuthenticator will use username to do authorization.

sajjoseph commented 5 years ago

@cquptEthan or @RameshByndoor

Can you confirm whether this PR will support existing ranger rules defined for hive as @stagraqubole mentioned earlier? I saw the statement - "Able to reuse existing policies". Can you elaborate on how this is done if you are using SystemAccessControl ?

cryptoe commented 5 years ago

https://github.com/prestodb/presto/pull/11640 one does. It reuses the existing policies.

sajjoseph commented 5 years ago

@cquptEthan or @RameshByndoor

Looks like Ranger policy might contain HQL or native DB supported SQL extension based queries. Any thoughts around how to address those as they could fail in Presto? (I haven't tried to see how it behaves. But asking based on the information found here - https://cwiki.apache.org/confluence/display/RANGER/Row-level+filtering+and+column-masking+using+Apache+Ranger+policies+in+Apache+Hive)

cryptoe commented 5 years ago

@sajjoseph Yes the ranger policy might contain HQL or native DB supported SQL externsion based queries. As of now, this patch just assumes the row level filter is ANSI SQL and tries to parse it. If it fails, query is not launched else its launched.

We tried using Apache clacite for conversion of HQL to ANSI SQL, but it was failing in complex q's and was not solving our internal use case.

sajjoseph commented 5 years ago

Thanks Karan. I will try it out and share the results.

On Wed, Oct 31, 2018 at 1:40 AM Karan Kumar notifications@github.com wrote:

@sajjoseph https://github.com/sajjoseph Yes the ranger policy might contain HQL or native DB supported SQL externsion based queries. As of now, this patch just assumes the row level filter is ANSI SQL and tries to parse it. If it fails, query is not launched else its launched.

We tried using Apache clacite for conversion of HQL to ANSI SQL, but it was failing in complex q's and was not solving our internal use case.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/prestodb/presto/issues/8980#issuecomment-434589999, or mute the thread https://github.com/notifications/unsubscribe-auth/AQhEuP46l0-mt6JrQqe5olI5yI3PMOzeks5uqVPlgaJpZM4PYokw .

-- Sajumon Joseph 303-378-0593 sajjoseph@gmail.com

tooptoop4 commented 5 years ago

gentle ping

tooptoop4 commented 5 years ago

Ship it!

bolkedebruin commented 5 years ago

Support for Presto has been merged into Ranger (no row level security yet, as Presto lacks support at the moment).

https://github.com/apache/ranger/commit/43757e798751ffab99dbe15ab3f9ae0773ae69f7

johnwhumphreys commented 5 years ago

@bolkedebruin - Naive questions - (1) how would one configure presto to hit ranger now that this is present, and (2) any idea at a release timeline?

bolkedebruin commented 5 years ago

See here https://cwiki.apache.org/confluence/display/RANGER/Presto+Plugin @johnwhumphreys

Release date is really up to the ranger committers.

x90004942 commented 5 years ago

See here https://cwiki.apache.org/confluence/display/RANGER/Presto+Plugin @johnwhumphreys

Release date is really up to the ranger committers.

but I do not find the presto-plugin in the newest apache-ranger-1.2.0. do you find it? @bolkedebruin

johnwhumphreys commented 5 years ago

It’s in master. 1.2 was released a long time ago, the plugin was only added about a month ago though.

bradhill99 commented 5 years ago

Which Presto version support this ranger plugin? I am trying to install this plug in AWS EMR, the latest version of Presto in EMR 5.24 is 0.219, want to make sure it can fit.

x90004942 commented 5 years ago

I downloaded the newest code of prestosql, which is presto316. I tryed install ranger-admin and ranger-presto-plugin. but when I restart presto server ,i got an Error .can anyone help me ? thankyou

Error:

2019-06-25T20:22:32.299+0800 INFO main org.apache.ranger.plugin.service.RangerBasePlugin PolicyEngineOptions: { evaluatorType: auto, evaluateDelegateAdminOnly: false, disableContextEnrichers: false, disableCustomConditions: false, disableTagPolicyEvaluation: false, enableTagEnricherWithLocalRefresher: false, disableTrieLookupPrefilter: false, optimizeTrieForRetrieval: false, cacheAuditResult: false } 2019-06-25T20:22:32.327+0800 ERROR main io.prestosql.server.PrestoServer Unable to create injector, see the following errors:

1) Error injecting constructor, java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.(RangerSystemAccessControl.java:40) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControlFactory.lambda$create$0(RangerSystemAccessControlFactory.java:45) while locating org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl

1 error com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error injecting constructor, java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.(RangerSystemAccessControl.java:40) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControlFactory.lambda$create$0(RangerSystemAccessControlFactory.java:45) while locating org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl

1 error at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) at com.google.inject.Guice.createInjector(Guice.java:87) at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:240) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControlFactory.create(RangerSystemAccessControlFactory.java:53) at io.prestosql.security.AccessControlManager.setSystemAccessControl(AccessControlManager.java:142) at io.prestosql.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:122) at io.prestosql.server.PrestoServer.run(PrestoServer.java:138) at io.prestosql.server.PrestoServer.main(PrestoServer.java:70) Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.(RangerSystemAccessControl.java:56) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl$$FastClassByGuice$$ec9f475b.newInstance() at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.(RangerSystemAccessControl.java:54) ... 18 more Caused by: java.lang.NoClassDefFoundError: com/kstruct/gethostname4j/Hostname at org.apache.ranger.plugin.util.RangerRESTUtils.(RangerRESTUtils.java:74) at org.apache.ranger.admin.client.RangerAdminRESTClient.(RangerAdminRESTClient.java:58) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at org.apache.ranger.plugin.service.RangerBasePlugin.createAdminClient(RangerBasePlugin.java:597) at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:233) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.(RangerSystemAccessControl.java:84) ... 23 more Caused by: java.lang.ClassNotFoundException: com.kstruct.gethostname4j.Hostname at java.lang.ClassLoader.findClass(ClassLoader.java:530) at org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:285) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:127) ... 33 more

findepi commented 5 years ago

I downloaded the newest code of prestosql, which is presto316. I tryed install ranger-admin and ranger-presto-plugin. but when I restart presto server ,i got an Error .can anyone help me ? thankyou

@x90004942 for help with prestosql please join the #troubleshooting channel on Presto Community Slack (https://prestosql.io/community.html)

bolkedebruin commented 5 years ago

@x90004942 I apologize it seems that with a different invocation some extra jars are needed in the plugin directory (plugin/ranger/ranger-impl). You can just pick those up and add them and it will start to work:

commons-codec commons-codec-1.12 com.kstruct gethostname4j-0.0.3 com.sun jna-3.0.9.jar

I will update the plugin in Ranger to add those.

x90004942 commented 5 years ago

@x90004942 I apologize it seems that with a different invocation some extra jars are needed in the plugin directory (plugin/ranger/ranger-impl). You can just pick those up and add them and it will start to work:

commons-codec commons-codec-1.12 com.kstruct gethostname4j-0.0.3 com.sun jna-3.0.9.jar

I will update the plugin in Ranger to add those.

thank you I have put these jar into directory ranger-impl, like below. but I also got the same error. is there anything I did wrong?

ll direcotry:

root@slave3:/opt/presto316/presto-server-316-SNAPSHOT/plugin/ranger/ranger-presto-plugin-impl# ll total 30284 drwxr-xr-x 3 root root 4096 Jun 27 10:13 ./ drwxr-xr-x 3 root root 4096 Jun 25 11:30 ../ -rwsrwsrwt 1 1003 1002 41123 Nov 26 2018 commons-cli-1.2.jar -rwsrwsrwt 1 root root 2107981 Jun 27 09:59 commons-codec-1.12-bin.tar.gz -rwsrwsrwt 1 root root 339669 Jun 27 10:13 commons-codec-1.12.jar -rwsrwsrwt 1 root root 489179 Jun 27 10:13 commons-codec-1.12-javadoc.jar -rwsrwsrwt 1 root root 344285 Jun 27 10:13 commons-codec-1.12-sources.jar -rwsrwsrwt 1 root root 290302 Jun 27 10:13 commons-codec-1.12-tests.jar -rwsrwsrwt 1 root root 194732 Jun 27 10:13 commons-codec-1.12-test-sources.jar -rwsrwsrwt 1 1003 1002 588337 Nov 30 2018 commons-collections-3.2.2.jar -rwsrwsrwt 1 1003 1002 616888 Nov 30 2018 commons-configuration2-2.1.1.jar -rwsrwsrwt 1 1003 1002 208700 Nov 27 2018 commons-io-2.5.jar -rwsrwsrwt 1 1003 1002 284220 Nov 26 2018 commons-lang-2.6.jar -rwsrwsrwt 1 1003 1002 61829 Nov 30 2018 commons-logging-1.2.jar drwxr-xr-x 2 root root 4096 Jun 25 16:08 conf/ -rwsrwsrwt 1 1003 1002 8632395 Nov 30 2018 eclipselink-2.5.2.jar -rwsrwsrwt 1 root root 4202 Jun 27 09:59 gethostname4j-0.0.3.jar -rwsrwsrwt 1 1003 1002 190432 Nov 30 2018 gson-2.2.4.jar -rwsrwsrwt 1 root root 138309 Jun 20 14:29 hadoop-auth-3.1.1.jar -rwsrwsrwt 1 root root 4034318 Jun 20 14:29 hadoop-common-3.1.1.jar -rwsrwsrwt 1 1003 1002 1502280 Nov 30 2018 htrace-core4-4.1.0-incubating.jar -rwsrwsrwt 1 1003 1002 747794 Nov 30 2018 httpclient-4.5.3.jar -rwsrwsrwt 1 1003 1002 323824 Nov 30 2018 httpcore-4.4.6.jar -rwsrwsrwt 1 1003 1002 41029 Nov 30 2018 httpmime-4.5.3.jar -rwsrwsrwt 1 1003 1002 232248 Nov 26 2018 jackson-core-asl-1.9.13.jar -rwsrwsrwt 1 1003 1002 18336 Nov 30 2018 jackson-jaxrs-1.9.13.jar -rwsrwsrwt 1 1003 1002 780664 Nov 26 2018 jackson-mapper-asl-1.9.13.jar -rwsrwsrwt 1 1003 1002 27075 Dec 7 2018 jackson-xc-1.9.2.jar -rwsrwsrwt 1 1003 1002 162126 Nov 30 2018 javax.persistence-2.1.0.jar -rwsrwsrwt 1 1003 1002 1621691 Nov 30 2018 jersey-bundle-1.19.3.jar -rwsrwsrwt 1 1003 1002 436689 Nov 30 2018 jersey-core-1.19.jar -rwsrwsrwt 1 root root 165345 Jun 20 14:29 jersey-json-1.19.jar -rwsrwsrwt 1 root root 702882 Jun 20 14:29 jersey-server-1.19.jar -rwsrwsrwt 1 root root 795871 Jun 27 09:59 jna-3.0.9.jar -rwsrwsrwt 1 root root 27948 Jun 20 14:30 noggit-0.8.jar -rw-r--r-- 1 root root 196377 Jun 21 11:56 ranger-plugins-audit-2.0.0-SNAPSHOT.jar -rw-r--r-- 1 root root 768440 Jun 21 11:56 ranger-plugins-common-2.0.0-SNAPSHOT.jar -rw-r--r-- 1 root root 12157 Jun 21 11:56 ranger-plugins-cred-2.0.0-SNAPSHOT.jar -rwxr--r-- 1 root root 10476 Jun 25 16:07 ranger-presto-audit.xml -rw-r--r-- 1 root root 46186 Jun 21 11:58 ranger-presto-plugin-2.0.0-SNAPSHOT.jar -rwxr--r-- 1 root root 2657 Jun 25 16:07 ranger-presto-security.xml -rwsrwsrwt 1 root root 2001986 Jun 20 14:30 solr-solrj-7.7.1.jar -rwsrwsrwt 1 1003 1002 161867 Nov 30 2018 stax2-api-3.1.4.jar -rwsrwsrwt 1 1003 1002 512742 Nov 30 2018 woodstox-core-5.0.3.jar -rwsrwsrwt 1 root root 911603 Jun 20 14:30 zookeeper-3.4.14.jar

error: 2019-06-27T10:17:42.393+0800 INFO Ranger async Audit cleanup org.apache.ranger.audit.provider.AuditProviderFactory RangerAsyncAuditCleanup: Waiting to audit cleanup start signal 2019-06-27T10:17:42.396+0800 INFO main org.apache.ranger.plugin.service.RangerBasePlugin PolicyEngineOptions: { evaluatorType: auto, evaluateDelegateAdminOnly: false, disableContextEnrichers: false, disableCustomConditions: false, disableTagPolicyEvaluation: false, enableTagEnricherWithLocalRefresher: false, disableTrieLookupPrefilter: false, optimizeTrieForRetrieval: false, cacheAuditResult: false } 2019-06-27T10:17:42.455+0800 ERROR main io.prestosql.server.PrestoServer Unable to create injector, see the following errors:

1) Error injecting constructor, java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.(RangerSystemAccessControl.java:40) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControlFactory.lambda$create$0(RangerSystemAccessControlFactory.java:45) while locating org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl

1 error com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error injecting constructor, java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.(RangerSystemAccessControl.java:40) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControlFactory.lambda$create$0(RangerSystemAccessControlFactory.java:45) while locating org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl

1 error at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) at com.google.inject.Guice.createInjector(Guice.java:87) at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:240) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControlFactory.create(RangerSystemAccessControlFactory.java:53) at io.prestosql.security.AccessControlManager.setSystemAccessControl(AccessControlManager.java:142) at io.prestosql.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:122) at io.prestosql.server.PrestoServer.run(PrestoServer.java:138) at io.prestosql.server.PrestoServer.main(PrestoServer.java:70) Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.(RangerSystemAccessControl.java:56) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl$$FastClassByGuice$$ec9f475b.newInstance() at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211) at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.(RangerSystemAccessControl.java:54) ... 18 more Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'c': /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:145) at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:188) at com.sun.jna.Library$Handler.(Library.java:123) at com.sun.jna.Native.loadLibrary(Native.java:255) at com.sun.jna.Native.loadLibrary(Native.java:241) at com.kstruct.gethostname4j.Hostname$UnixCLibrary.(Hostname.java:12) at com.kstruct.gethostname4j.Hostname.getHostname(Hostname.java:30) at org.apache.ranger.plugin.util.RangerRESTUtils.(RangerRESTUtils.java:74) at org.apache.ranger.admin.client.RangerAdminRESTClient.(RangerAdminRESTClient.java:58) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at org.apache.ranger.plugin.service.RangerBasePlugin.createAdminClient(RangerBasePlugin.java:597) at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:233) at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.(RangerSystemAccessControl.java:84) ... 23 more

bolkedebruin commented 5 years ago

Its not the same: Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'c': /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header

This you need to fix yourself as that is not an issue with the plug-in but an is errot

x90004942 commented 5 years ago

thank you very much . i have solved this problem by putting A right libc.so into this directory. now I can see my presto plugin in the Ranger web -- Audit--Plugins page , whose status is 200:)

then i add a service prestodev, using jdbc:presto://10.183.243.83:8090. when I test the connection , i got another error :

Connection Failed.
Unable to retrieve any files using given parameters, You can still save the repository and start creating policies, but you would not be able to use autocomplete for resource names. Check ranger_admin.log for more info.

org.apache.ranger.plugin.client.HadoopException: Unable to connect to Presto instance.. 
Unable to connect to Presto instance.. 
Authentication using username/password requires SSL to be enabled.

my presto server is configured with SSL. and i can access presto by ./presto_cli --server https://slave3.example.com:9090 --catalog mysql --schema population --truststore-path /opt/presto-public.store --truststore-password XXX123 --user root --password

so how can I configure Ranger with SSL? and also i can not find the file ranger_admin.log

dsjoegeo commented 5 years ago

Could you please tell us which version of horton on which it worked.we get a facebook spi not found error when we add ranger plugin in prest

dsjoegeo commented 5 years ago

We are getting errors on the same lines as the one posted above by x90004942 commented on Jul 2 Connection Failed. Unable to retrieve any files using given parameters, You can still save the repository and start creating policies, but you would not be able to use autocomplete for resource names. Check ranger_admin.log for more info. We got test connection failed. We also tried to search the ranger_admin.log without any sucess. Any help would be really appreciated.

andy12383 commented 5 years ago

@dsjoegeo in my environment , I configured presto wtih ranger and LDAP with https enabled. you can refer to the official website. the problem I encountered was that , in the Config Properties in ranger admin webpage, I input a wrong port into jdbc.url (jdbc:presto://10.183.163.138:9090). 9090 is the secure port for https. then I change the port to 8080 which is my unsecured port. it works .test connection successfully