turtlemonvh / ionic-spark-utils

Utilities for working with Ionic encryption via Spark.
MIT License
0 stars 0 forks source link

Add configurable options for setting attributes+metadata on encrypt and metadata on decrypt #6

Open turtlemonvh opened 4 years ago

turtlemonvh commented 4 years ago

This is likely best configured via lambda functions.

With all these options, it's also time to change the transform function from taking a simple parameter list to taking a case class options object (e.g. IonicEncryptTranformOptions).

turtlemonvh commented 4 years ago

In addition to these options, I need to extend the parameters for the transformer to take more complex configuration for each action, so we can write transformed columns to custom named columns instead of requiring a rename every time.

A nice interface to provide here would be

// Before
def Encrypt(encryptCols: List[String], decryptCols: List[String], agentFactory: () => KeyServices)(df: DataFrame): DataFrame
// After
def Encrypt(actions: List[IonicAction], agentFactory: () => KeyServices)(df: DataFrame): DataFrame

Where IonicAction is a type we define that defines the action to perform, which gives us flexibility to extend capabilities to deterministic encryption (see #7 ) and FPE (see #11 ), and also makes it easy to add custom metadata, attributes, attributes derived from data, column renaming, and even data types (e.g. binary vs string).

The IonicAction also gives us a good interface for testing, since we'll be able to test

turtlemonvh commented 4 years ago

I also plan to support nested columns here.