search

exacaster / lighter

REST API for Apache Spark on K8S or YARN
MIT License
91 stars 21 forks source link

Enhance Security: Mask/Remove Reduct sensitive spark properties from session info #1071

Closed katymufler closed 2 weeks ago

katymufler commented 3 months ago

Description:

Currently, Lighter session info page may inadvertently expose sensitive information (e.g., database credentials, API keys) as part of spark properties.

I think it would be useful to create a comprehensive list of sensitive Spark properties that should be masked or removed from session info. This could include: spark.hadoop.hive.metastore.uris spark.datasource.jdbc.url Any property containing password, secret, key, etc.

Also Allow users to customize the list of sensitive properties and the masking/removal behavior via configuration options.

Regards, Katy

katymufler commented 2 weeks ago

Thank you! In which release it will be included and when? Thanks again :)

pdambrauskas commented 2 weeks ago

Hey, it will be included on the next release. We will release it as soon as @Minutis will test our latest changes on his cluster.