Currently, Lighter session info page may inadvertently expose sensitive information (e.g., database credentials, API keys) as part of spark properties.
I think it would be useful to create a comprehensive list of sensitive Spark properties that should be masked or removed from session info. This could include:
spark.hadoop.hive.metastore.uris
spark.datasource.jdbc.url
Any property containing password, secret, key, etc.
Also Allow users to customize the list of sensitive properties and the masking/removal behavior via configuration options.
Description:
Currently, Lighter session info page may inadvertently expose sensitive information (e.g., database credentials, API keys) as part of spark properties.
I think it would be useful to create a comprehensive list of sensitive Spark properties that should be masked or removed from session info. This could include: spark.hadoop.hive.metastore.uris spark.datasource.jdbc.url Any property containing password, secret, key, etc.
Also Allow users to customize the list of sensitive properties and the masking/removal behavior via configuration options.
Regards, Katy