S3 Connector - Add ability for Client Side encryption

[git-josip]

Hi,

Is there a way to create s3Client with EncryptionMaterials for supporting client side encryption like with AWS:

  private val encryptionMaterials = new EncryptionMaterials(this.loadKeyPair)
  val s3EncryptionClient: AmazonS3 = AmazonS3EncryptionClientBuilder.standard()
    .withCredentials(this.awsCredentialsProvider)
    .withEncryptionMaterials(new StaticEncryptionMaterialsProvider(this.encryptionMaterials))
    .withRegion(region)
    .build()

[fmsbeekmans]

Started implementing but encryption/decryption flows might be more generally useful than just S3.

[markschaake]

Client-side encryption is often coupled with some sort of encryption key rotation strategy. For AWS S3, this is facilitated by putting an object along with an InstructionFile, which provides a mechanism for re-encrypting S3 objects without having to fetch them to the client at all (thus saving bandwidth and processing costs). You can imagine that key rotation can become prohibitive as your usage of S3 scales up (imagine both data transfer time and costs). Here's a simplification of the S3 client wrapper that we have for example:

import javax.crypto.spec.SecretKeySpec
import com.amazonaws.AmazonServiceException
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain
import com.amazonaws.services.s3.AmazonS3
import com.amazonaws.services.s3.model.{ CryptoConfiguration, CryptoStorageMode, EncryptionMaterials, SimpleMaterialProvider }
import com.amazonaws.services.s3.AmazonS3EncryptionClientBuilder

class AwsS3(secretKey: SecretKeySpec, version: Option[String]) {
  val encryptionMaterials = new SimpleMaterialProvider().withLatest(version match {
    case Some(v) => new EncryptionMaterials(secretKey).addDescription("version", v)
    case None => new EncryptionMaterials(secretKey)
  })

  def mkS3Client = AmazonS3EncryptionClientBuilder
    .standard()
    .withCredentials(DefaultAWSCredentialsProviderChain.getInstance)
    .withEncryptionMaterials(encryptionMaterials)
    .withCryptoConfiguration(new CryptoConfiguration()
      .withStorageMode(CryptoStorageMode.InstructionFile)  // this enables efficient key rotation
      .withIgnoreMissingInstructionFile(false))
    .build()

  // ...
}

In my opinion, a general encryption/decryption flow solution is not appropriate for AWS S3, since the (strictly?) better practice is to use InstructionFiles.