Open HariSekhon opened 6 years ago
This could perhaps piggy back to use Hive policies in Ranger or else copy what Hive has to a separately managed policy for Presto. There are trade-offs to both styles, so perhaps make this a configurable user choice.
I am trying to implement authorization for any connector. For that to work I need to access the resources (columns etc) for the connector. I have been trying to find where in the source you have access to the connector metadata (tables, columns, connection detail etc) and the query information before execution and it seems all of this is available to you in the SQLQueryExecution class in the start function.
After you have analyzed the query and gotten all the connector handles.
Would this be a good place to perform an authorization check if we can access the necessary resources (column granularity). It feels like its too late in the Presto process to reject a request but who knows. Anyone who can point me in the right direction?
Thank you
any updates or plans where integration will be completed?
Hi, I could add a plugin to presto with Ranger(Registered presto as new ranger service with catalog->schema->table->column as resources) and all those methods implemented from SystemAccessControl are working fine. I would need to refractor and would like to bring it to here.
Need some help with addressing column level security:
Does com.facebook.presto.sql.analyzer.StatementAnalyzer
has any placeholder for identifying column resources used in query to apply column level security check.? Currently i am looking at this method
analyzeSelect(
https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/sql/analyzer/StatementAnalyzer.java#L1710
Any hint around this area would be much appreciated.
What about using https://github.com/prestodb/presto/blob/master/presto-spi/src/main/java/com/facebook/presto/spi/security/SystemAccessControl.java#L221
Thanks @kokosing. that helps. My bad I was checking with v201.
Need a clarification on Set<String> columnNames
passed to checkCanSelectFromColumns
method.
For the below query, I'm expecting object_name
alone in columnNames
. but as of now it's object_name_alias
and object_name
. Can this be filtered with the help of metadata in AccessControlManager
before making call.
select object_name as object_name_alias from "java.lang:type=memory" mem;
To me it sounds like a bug?
CC: @rschlussel2 Is this expected?
@RameshByndoor can u share the code?
@RameshByndoor Are you going to base your work on top of https://github.com/prestodb/presto/pull/10904?
@RameshByndoor can you give an example query where you're seeing the alias and object name. I can take a look.
@rschlussel2 you can use simple command as select object_name as object_name_alias from "jmx.current.java.lang:type=memory";
& you can catch this getting called from here. https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/security/AccessControlManager.java#L598
@rschlussel can you help me with the above.? Is it expected or how to patch it.?
I want to implement ranger plugin for presto .. could you please point to the branch which I can refer and clone and test
@RameshByndoor are you working on this actively? We have a need to add this support and were planning to take up this work up unless someone else is already working on it. Please let me know.
@RameshByndoor the issue you mentioned does seem like a bug and is easily reproducible. While @rschlussel is looking at it you can unblock by filtering columns which are not identifiers, something like this in Analyzer.analyze:
+ List<String> identifiers = analysis
+ .getColumnReferences()
+ .stream()
+ .filter(nodeRef -> nodeRef.getNode() instanceof Identifier)
+ .map(nodeRef -> ((Identifier) nodeRef.getNode()).getValue())
+ .collect(Collectors.toList());
analysis.getTableColumnReferences().forEach((accessControlInfo, tableColumnReferences) ->
tableColumnReferences.forEach((tableName, columns) ->
accessControlInfo.getAccessControl().checkCanSelectFromColumns(
session.getRequiredTransactionId(),
accessControlInfo.getIdentity(),
tableName,
- columns)));
+ columns.stream().filter(column -> identifiers.contains(column)).collect(Collectors.toSet()))));
Thanks for the reminder- I'd forgotten about this. I just put up a PR to fix it #11295
@stagraqubole The code is similar in both the cases. Its a matter of where we plug it. According to me the trade off between SystemAccessControl and ConnectorAccessControl is .
SystemAccessControl would work regardless of all the underlying connectors of presto. Policies will be maintained under presto service in Ranger making presto as a main query engine. Using ConnectorAccessControl can make use of the existing policies defined. This helps presto be a parallel query engine which points to existing policies. For eg hive policies defined in ranger. Another point about using ConnectorAccessControl is, with multiple connectors enabled then multiple Policy engines of ranger will be created, which are inherently heavy(As per https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741207).
Taking example of Hive connector which would be the biggest consumer of Ranger support, doing it in SystemAccessControl would mean that you will maintain two Ranger policies now: one for Hive and one for Presto which would get difficult to maintain. Doing it in ConnectorAccessControl allows a single Policy holder in Ranger and that would be the one created for Hive, very similar to having SqlStandardAccessControl as ConnectorAccessControl rather than SystemAccessControl.
@rschlussel Does your PR fix #11295 has a support for Ranger in Presto now?
no. it unblocks @RameshByndoor who is working on adding support.
im interested in this ranger plugin for presto.. can you give some branch which i can take and try to test
We are on test cases nd other sanity checks, Not yet ready to PR.
Any update on timeline? I appreciate the work you've done for this plugin. Would love to get my hands on it.
https://docs.google.com/document/d/1Jtapmwkp1Up_w6w_3dUeOXfLPLRAsglbqoIWNRG-NJM/edit#
This is what me and @RameshByndoor have done. We are testing the code at this moment. I can't commit any hard timelines but we should be in a state to release it next week.
We have also handled https://github.com/prestodb/presto/issues/10996 as part of the ranger presto integration.
Features supported :
This may not be the best place for a question, but given the lack of support for this specific feature, I figured I'll give it a shot.
@cquptEthan or any one else? Have you gotten SSL working with this custom plugin? We don't have the CredentialProvider API setup for keystore passwords, and I'm at a loss on how to get this working without it.
From what I can tell, the properties I need to have available with an https Ranger endpoint are: xasecure.policymgr.clientssl.keystore= xasecure.policymgr.clientssl.truststore= xasecure.policymgr.clientssl.keystore.credential.file= xasecure.policymgr.clientssl.truststore.credential.file=
The plugin then expects a value for hadoop.security.credential.provider.path to interrogate the CredentialProvider API for a password of the keystore/truststore based on the last two properties above. Is this understanding correct? How can I get around using the CredentialProvider API?
@gray-eb It seems that you want to add a ranger-policymgr-ssl.xml file in presto ranger plugin, but the presto plugin can't find the xml file, am i right? In my implements, I read the ranger configs from access-control.properties that begin with 'ranger.'. Like this.
@Override
public SystemAccessControl create(Map<String, String> config)
{
RangerConfiguration rangerConfig = RangerConfiguration.getInstance();
try {
handleKerberos(rangerConfig, config);
}
catch (IOException e) {
throw new PrestoException(StandardErrorCode.GENERIC_INTERNAL_ERROR, "Failed to do kerberos right", e);
}
for (final Map.Entry<String, String> configEntry : config.entrySet()) {
if (configEntry.getKey().startsWith("ranger.")) {
rangerConfig.set(configEntry.getKey(), configEntry.getValue());
log.info("Setting: " + configEntry.getKey() + " to: " + configEntry.getValue());
}
}
PrestoAuthorizer authorizer = getPrestoAuthorizer(config);
return new RangerSystemAccessControl(authorizer, config);
}
Then you can add this config in access-control.properties to define your ssl config xml. ranger.plugin.\<ServiceType>.policy.rest.ssl.config.file=path/to/ranger-policymgr-ssl.xml
@cquptEthan is this ranger authorization being built into presto only going to work for kerberos?
@tooptoop4 No. LDAP Authentication and other implement of PasswordAuthenticator will use username to do authorization.
@cquptEthan or @RameshByndoor
Can you confirm whether this PR will support existing ranger rules defined for hive as @stagraqubole mentioned earlier? I saw the statement - "Able to reuse existing policies". Can you elaborate on how this is done if you are using SystemAccessControl ?
https://github.com/prestodb/presto/pull/11640 one does. It reuses the existing policies.
@cquptEthan or @RameshByndoor
Looks like Ranger policy might contain HQL or native DB supported SQL extension based queries. Any thoughts around how to address those as they could fail in Presto? (I haven't tried to see how it behaves. But asking based on the information found here - https://cwiki.apache.org/confluence/display/RANGER/Row-level+filtering+and+column-masking+using+Apache+Ranger+policies+in+Apache+Hive)
@sajjoseph Yes the ranger policy might contain HQL or native DB supported SQL externsion based queries. As of now, this patch just assumes the row level filter is ANSI SQL and tries to parse it. If it fails, query is not launched else its launched.
We tried using Apache clacite for conversion of HQL to ANSI SQL, but it was failing in complex q's and was not solving our internal use case.
Thanks Karan. I will try it out and share the results.
On Wed, Oct 31, 2018 at 1:40 AM Karan Kumar notifications@github.com wrote:
@sajjoseph https://github.com/sajjoseph Yes the ranger policy might contain HQL or native DB supported SQL externsion based queries. As of now, this patch just assumes the row level filter is ANSI SQL and tries to parse it. If it fails, query is not launched else its launched.
We tried using Apache clacite for conversion of HQL to ANSI SQL, but it was failing in complex q's and was not solving our internal use case.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/prestodb/presto/issues/8980#issuecomment-434589999, or mute the thread https://github.com/notifications/unsubscribe-auth/AQhEuP46l0-mt6JrQqe5olI5yI3PMOzeks5uqVPlgaJpZM4PYokw .
-- Sajumon Joseph 303-378-0593 sajjoseph@gmail.com
gentle ping
Ship it!
Support for Presto has been merged into Ranger (no row level security yet, as Presto lacks support at the moment).
https://github.com/apache/ranger/commit/43757e798751ffab99dbe15ab3f9ae0773ae69f7
@bolkedebruin - Naive questions - (1) how would one configure presto to hit ranger now that this is present, and (2) any idea at a release timeline?
See here https://cwiki.apache.org/confluence/display/RANGER/Presto+Plugin @johnwhumphreys
Release date is really up to the ranger committers.
See here https://cwiki.apache.org/confluence/display/RANGER/Presto+Plugin @johnwhumphreys
Release date is really up to the ranger committers.
but I do not find the presto-plugin in the newest apache-ranger-1.2.0. do you find it? @bolkedebruin
It’s in master. 1.2 was released a long time ago, the plugin was only added about a month ago though.
Which Presto version support this ranger plugin? I am trying to install this plug in AWS EMR, the latest version of Presto in EMR 5.24 is 0.219, want to make sure it can fit.
I downloaded the newest code of prestosql, which is presto316. I tryed install ranger-admin and ranger-presto-plugin. but when I restart presto server ,i got an Error .can anyone help me ? thankyou
Error:
2019-06-25T20:22:32.299+0800 INFO main org.apache.ranger.plugin.service.RangerBasePlugin PolicyEngineOptions: { evaluatorType: auto, evaluateDelegateAdminOnly: false, disableContextEnrichers: false, disableCustomConditions: false, disableTagPolicyEvaluation: false, enableTagEnricherWithLocalRefresher: false, disableTrieLookupPrefilter: false, optimizeTrieForRetrieval: false, cacheAuditResult: false } 2019-06-25T20:22:32.327+0800 ERROR main io.prestosql.server.PrestoServer Unable to create injector, see the following errors:
1) Error injecting constructor, java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.
1 error com.google.inject.CreationException: Unable to create injector, see the following errors:
1) Error injecting constructor, java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.
1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
at com.google.inject.Guice.createInjector(Guice.java:87)
at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:240)
at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControlFactory.create(RangerSystemAccessControlFactory.java:53)
at io.prestosql.security.AccessControlManager.setSystemAccessControl(AccessControlManager.java:142)
at io.prestosql.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:122)
at io.prestosql.server.PrestoServer.run(PrestoServer.java:138)
at io.prestosql.server.PrestoServer.main(PrestoServer.java:70)
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.
I downloaded the newest code of prestosql, which is presto316. I tryed install ranger-admin and ranger-presto-plugin. but when I restart presto server ,i got an Error .can anyone help me ? thankyou
@x90004942 for help with prestosql please join the #troubleshooting
channel on Presto Community Slack (https://prestosql.io/community.html)
@x90004942 I apologize it seems that with a different invocation some extra jars are needed in the plugin directory (plugin/ranger/ranger-impl). You can just pick those up and add them and it will start to work:
commons-codec commons-codec-1.12 com.kstruct gethostname4j-0.0.3 com.sun jna-3.0.9.jar
I will update the plugin in Ranger to add those.
@x90004942 I apologize it seems that with a different invocation some extra jars are needed in the plugin directory (plugin/ranger/ranger-impl). You can just pick those up and add them and it will start to work:
commons-codec commons-codec-1.12 com.kstruct gethostname4j-0.0.3 com.sun jna-3.0.9.jar
I will update the plugin in Ranger to add those.
thank you I have put these jar into directory ranger-impl, like below. but I also got the same error. is there anything I did wrong?
ll direcotry:
root@slave3:/opt/presto316/presto-server-316-SNAPSHOT/plugin/ranger/ranger-presto-plugin-impl# ll total 30284 drwxr-xr-x 3 root root 4096 Jun 27 10:13 ./ drwxr-xr-x 3 root root 4096 Jun 25 11:30 ../ -rwsrwsrwt 1 1003 1002 41123 Nov 26 2018 commons-cli-1.2.jar -rwsrwsrwt 1 root root 2107981 Jun 27 09:59 commons-codec-1.12-bin.tar.gz -rwsrwsrwt 1 root root 339669 Jun 27 10:13 commons-codec-1.12.jar -rwsrwsrwt 1 root root 489179 Jun 27 10:13 commons-codec-1.12-javadoc.jar -rwsrwsrwt 1 root root 344285 Jun 27 10:13 commons-codec-1.12-sources.jar -rwsrwsrwt 1 root root 290302 Jun 27 10:13 commons-codec-1.12-tests.jar -rwsrwsrwt 1 root root 194732 Jun 27 10:13 commons-codec-1.12-test-sources.jar -rwsrwsrwt 1 1003 1002 588337 Nov 30 2018 commons-collections-3.2.2.jar -rwsrwsrwt 1 1003 1002 616888 Nov 30 2018 commons-configuration2-2.1.1.jar -rwsrwsrwt 1 1003 1002 208700 Nov 27 2018 commons-io-2.5.jar -rwsrwsrwt 1 1003 1002 284220 Nov 26 2018 commons-lang-2.6.jar -rwsrwsrwt 1 1003 1002 61829 Nov 30 2018 commons-logging-1.2.jar drwxr-xr-x 2 root root 4096 Jun 25 16:08 conf/ -rwsrwsrwt 1 1003 1002 8632395 Nov 30 2018 eclipselink-2.5.2.jar -rwsrwsrwt 1 root root 4202 Jun 27 09:59 gethostname4j-0.0.3.jar -rwsrwsrwt 1 1003 1002 190432 Nov 30 2018 gson-2.2.4.jar -rwsrwsrwt 1 root root 138309 Jun 20 14:29 hadoop-auth-3.1.1.jar -rwsrwsrwt 1 root root 4034318 Jun 20 14:29 hadoop-common-3.1.1.jar -rwsrwsrwt 1 1003 1002 1502280 Nov 30 2018 htrace-core4-4.1.0-incubating.jar -rwsrwsrwt 1 1003 1002 747794 Nov 30 2018 httpclient-4.5.3.jar -rwsrwsrwt 1 1003 1002 323824 Nov 30 2018 httpcore-4.4.6.jar -rwsrwsrwt 1 1003 1002 41029 Nov 30 2018 httpmime-4.5.3.jar -rwsrwsrwt 1 1003 1002 232248 Nov 26 2018 jackson-core-asl-1.9.13.jar -rwsrwsrwt 1 1003 1002 18336 Nov 30 2018 jackson-jaxrs-1.9.13.jar -rwsrwsrwt 1 1003 1002 780664 Nov 26 2018 jackson-mapper-asl-1.9.13.jar -rwsrwsrwt 1 1003 1002 27075 Dec 7 2018 jackson-xc-1.9.2.jar -rwsrwsrwt 1 1003 1002 162126 Nov 30 2018 javax.persistence-2.1.0.jar -rwsrwsrwt 1 1003 1002 1621691 Nov 30 2018 jersey-bundle-1.19.3.jar -rwsrwsrwt 1 1003 1002 436689 Nov 30 2018 jersey-core-1.19.jar -rwsrwsrwt 1 root root 165345 Jun 20 14:29 jersey-json-1.19.jar -rwsrwsrwt 1 root root 702882 Jun 20 14:29 jersey-server-1.19.jar -rwsrwsrwt 1 root root 795871 Jun 27 09:59 jna-3.0.9.jar -rwsrwsrwt 1 root root 27948 Jun 20 14:30 noggit-0.8.jar -rw-r--r-- 1 root root 196377 Jun 21 11:56 ranger-plugins-audit-2.0.0-SNAPSHOT.jar -rw-r--r-- 1 root root 768440 Jun 21 11:56 ranger-plugins-common-2.0.0-SNAPSHOT.jar -rw-r--r-- 1 root root 12157 Jun 21 11:56 ranger-plugins-cred-2.0.0-SNAPSHOT.jar -rwxr--r-- 1 root root 10476 Jun 25 16:07 ranger-presto-audit.xml -rw-r--r-- 1 root root 46186 Jun 21 11:58 ranger-presto-plugin-2.0.0-SNAPSHOT.jar -rwxr--r-- 1 root root 2657 Jun 25 16:07 ranger-presto-security.xml -rwsrwsrwt 1 root root 2001986 Jun 20 14:30 solr-solrj-7.7.1.jar -rwsrwsrwt 1 1003 1002 161867 Nov 30 2018 stax2-api-3.1.4.jar -rwsrwsrwt 1 1003 1002 512742 Nov 30 2018 woodstox-core-5.0.3.jar -rwsrwsrwt 1 root root 911603 Jun 20 14:30 zookeeper-3.4.14.jar
error: 2019-06-27T10:17:42.393+0800 INFO Ranger async Audit cleanup org.apache.ranger.audit.provider.AuditProviderFactory RangerAsyncAuditCleanup: Waiting to audit cleanup start signal 2019-06-27T10:17:42.396+0800 INFO main org.apache.ranger.plugin.service.RangerBasePlugin PolicyEngineOptions: { evaluatorType: auto, evaluateDelegateAdminOnly: false, disableContextEnrichers: false, disableCustomConditions: false, disableTagPolicyEvaluation: false, enableTagEnricherWithLocalRefresher: false, disableTrieLookupPrefilter: false, optimizeTrieForRetrieval: false, cacheAuditResult: false } 2019-06-27T10:17:42.455+0800 ERROR main io.prestosql.server.PrestoServer Unable to create injector, see the following errors:
1) Error injecting constructor, java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.
1 error com.google.inject.CreationException: Unable to create injector, see the following errors:
1) Error injecting constructor, java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.
1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
at com.google.inject.Guice.createInjector(Guice.java:87)
at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:240)
at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControlFactory.create(RangerSystemAccessControlFactory.java:53)
at io.prestosql.security.AccessControlManager.setSystemAccessControl(AccessControlManager.java:142)
at io.prestosql.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:122)
at io.prestosql.server.PrestoServer.run(PrestoServer.java:138)
at io.prestosql.server.PrestoServer.main(PrestoServer.java:70)
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.ranger.authorization.presto.authorizer.RangerSystemAccessControl.
Its not the same: Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'c': /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header
This you need to fix yourself as that is not an issue with the plug-in but an is errot
thank you very much . i have solved this problem by putting A right libc.so into this directory. now I can see my presto plugin in the Ranger web -- Audit--Plugins page , whose status is 200:)
then i add a service prestodev, using jdbc:presto://10.183.243.83:8090. when I test the connection , i got another error :
Connection Failed.
Unable to retrieve any files using given parameters, You can still save the repository and start creating policies, but you would not be able to use autocomplete for resource names. Check ranger_admin.log for more info.
org.apache.ranger.plugin.client.HadoopException: Unable to connect to Presto instance..
Unable to connect to Presto instance..
Authentication using username/password requires SSL to be enabled.
my presto server is configured with SSL. and i can access presto by ./presto_cli --server https://slave3.example.com:9090 --catalog mysql --schema population --truststore-path /opt/presto-public.store --truststore-password XXX123 --user root --password
so how can I configure Ranger with SSL? and also i can not find the file ranger_admin.log
Could you please tell us which version of horton on which it worked.we get a facebook spi not found error when we add ranger plugin in prest
We are getting errors on the same lines as the one posted above by x90004942 commented on Jul 2 Connection Failed. Unable to retrieve any files using given parameters, You can still save the repository and start creating policies, but you would not be able to use autocomplete for resource names. Check ranger_admin.log for more info. We got test connection failed. We also tried to search the ranger_admin.log without any sucess. Any help would be really appreciated.
@dsjoegeo in my environment , I configured presto wtih ranger and LDAP with https enabled. you can refer to the official website. the problem I encountered was that , in the Config Properties in ranger admin webpage, I input a wrong port into jdbc.url (jdbc:presto://10.183.163.138:9090). 9090 is the secure port for https. then I change the port to 8080 which is my unsecured port. it works .test connection successfully
Feature Request to add Apache Ranger integration support for Presto.
This will solve authorization very nicely for most people who are running the widely used standard open source Hortonworks Hadoop platform and provide granular access controls down to the column level, as well as integrating to the single-pane-of-glass for security configuration and auditing across all data access components on a Hadoop cluster.
Also, is there any chance this project could go in to the Apache Foundation? I think this combined with Ranger integration would massively increase Presto's market, especially if Hortonworks were to adopt it (they usually require their integrated components to be in the Apache Foundation).
Great work so far on this distributed SQL engine btw! :)