Encryption/Decryption mechanism for Shared Access Policy Key

Shared Access Policy Key is shown in plain text in application logs. I'm using eventhub-spark connector for reading from eventhubs in spark structured streaming mode. If i change the log level from INFO to DEBUG in log4j.properties, i see the physical plan getting dump in target/unit-tests.log. Physical plan contains sensitive information SharedAccessKey . It the security wise concern as is it coming as plain text.

`=== Applying Rule org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences === Before:

'Project icast(icast(null as int) as int) AS 0#30, icast(icast(null as string) as string) AS 1#31, icast(icast(null as string) as string) AS 2#32, icast('0 as binary) AS 3#33 [] +- AnalysisBarrier [] +- Project body#6 AS 0#18, offset#7 AS 1#19, sequenceNumber#8L AS 2#20L, enqueuedTime#9 AS 3#21, publisher#10 AS 4#22, partitionKey#11 AS 5#23 [] +- StreamingRelation DataSource(org.apache.spark.sql.SparkSession@669edc32,eventhubs,List(),None,List(),None,Map(eventhubs.connectionstring -> Endpoint=sb://iisdevhub.servicebus.windows.net;EntityPath=jsonevents;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=1/ypdHvviYjiiCouol6Gx/***=, eventhubs.name -> jsonevents, eventhubs.consumergroup -> $Default, eventhubs.policykey -> 1/ypdHvviY*jiiCouol6Gx/****=, eventhubs.namespace -> iisdevhub, eventhubs.policyname -> RootManageSharedAccessKey),None), eventhubs, body#6, offset#7, sequenceNumber#8L, enqueuedTime#9, publisher#10, partitionKey#11 []`

Drivers pass the SharedAccessKey in plain text to spark, so it is getting dump in physical plan of spark logs. Ideally drivers should take care of encrypting and decrypting the passwords, which is not done . Can encryption/decryption mechanism be added for SharedAccessKey before passing to Spark ? Is there an alternate way to not log it?

Thanks, Shreesh Sinha

Azure / azure-webjobs-sdk

Encryption/Decryption mechanism for Shared Access Policy Key #2030