[FEATURE] Migrate Spark security module from submarine to kyuubi

apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

https://kyuubi.apache.org/

Apache License 2.0

2.09k stars 913 forks source link

[FEATURE] Migrate Spark security module from submarine to kyuubi #1451

Closed anumee closed 2 years ago

anumee commented 2 years ago

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the feature

Spark security module was retired from submarine pull- 796 and it is still missing here. This is needed with Spark 3.2 support.

Motivation

Spark 3.2 support

Describe the solution

Migrate Submarine security module here - https://github.com/apache/submarine/pull/796 and,
Update Spark version to 3.2

Additional context

No response

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

lordk911 commented 2 years ago

Wonder to know when will this Spark security module to support Spark 3.2

pan3793 commented 2 years ago

This task is of medium priority and we don't have ETA now, volunteers are welcome :)

RamakrishnaChilaka commented 2 years ago

@pan3793 ,can you please check this PR ? https://github.com/apache/submarine/pull/861, This is a rough PR, After your review i am willing to change this.

pan3793 commented 2 years ago

Thanks @RamakrishnaChilaka for working on this feature, but ranger spark module is deprecated since https://github.com/apache/submarine/pull/796, the test infrastructure become not runable in apache/submarine, would you please port this module to kyuubi firstly?

RamakrishnaChilaka commented 2 years ago

sure @pan3793 , Kindly, Please check whether the current code is valid for spark 3.2 ?

pan3793 commented 2 years ago

Please check whether the current code is valid for Spark 3.2?

We need CI to verify that.

RamakrishnaChilaka commented 2 years ago

Ok @pan3793, I am not aware of kyuubi code base.. can you please guide me in porting submarine spark security module into kyuubi ? I tried searching for spark security files in the codebase..

minyk commented 2 years ago

@RamakrishnaChilaka I've done almost same thing for our product. I don't have the right to open the code, but this is commit list I made:

parseRawDataType() removed by [SPARK-33641]
child renamed to plan by [SPARK-34701]
AlterTableRecoverPartitionsCommand renamed to RepairTableCommand by [SPARK-34518]
CacheTableCommand renamed to CacheTalbe by [SPARK-33654]
Change RunnableCommand to LeafRunnableCommand by [SPARK-34989]
implements a withNewChildInteral method created by [SPARK-34989]

pan3793 commented 2 years ago

cc @yaooqinn

RamakrishnaChilaka commented 2 years ago

thanks @minyk , your list was truly helpful, I didn't knew about SPARK-34989, I have incorporated it, one small change to your list is UnCacheTableCommand is renamed to UnCacheTable.

RamakrishnaChilaka commented 2 years ago

You can find code here

Please note that, I have taken the code from submarine project.. But I couldn't find the code of spark-security inside Kyuubi.

yaooqinn commented 2 years ago

The spark security module currently contains 4 main features, including authorization, data masking, row-level filtering, and DCL(incomplete).

It contains a lot of legacy code for very old spark releases including spark 2.0, so I suggest we add them step by step starting with authorization.

FYI, https://issues.apache.org/jira/browse/SUBMARINE-409

lordk911 commented 2 years ago

@yaooqinn will [WIP] [KYUUBI #1451] Introduce Kyuubi Spark Authz Module #2160 support ranger1.2 ?

yaooqinn commented 2 years ago

@yaooqinn will [WIP] [KYUUBI #1451] Introduce Kyuubi Spark Authz Module #2160 support ranger1.2 ?

I don't know