search

leo-project / leofs

The LeoFS Storage System
https://leo-project.net/leofs/
Apache License 2.0
1.54k stars 154 forks source link

Support server side encryption (SSE) #114

Open kentarosasaki opened 10 years ago

kentarosasaki commented 10 years ago

Is there a plan for server side encryption support? http://aws.typepad.com/aws/2011/10/new-amazon-s3-server-side-encryption.html http://googlecloudplatform.blogspot.jp/2013/08/google-cloud-storage-now-provides.html

yosukehara commented 10 years ago

We have not planned its function because we need to realize the following functions:

So after realized them we will consider it. Thank you.

mocchira commented 7 years ago

http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html

yosukehara commented 7 years ago

It is listed on LeoFS Milestones. We'll implement SSE with v1.4.0.

yosukehara commented 7 years ago

In the first step, we're going to implement Server-Side Encryption with Customer-Provided Keys (SSE-C) .

mocchira commented 7 years ago

Spec:

Docs for implement:

windkit commented 6 years ago

I propose to add several fields in metadata for SSE-C

  1. Original Object ETag
  2. Original Object Size
  3. Key Salted Hash (for verification)

So most operations could be handled without disk read from AVS and minimize code change at leo_object_storage, object encryption and decryption happens in leo_storage

@mocchira @yosukehara

mocchira commented 6 years ago

@windkit

Can you elaborate

Original Object ETag Original Object Size

how those values contribute to minimizing the code change and disk read from AVS on each operation (GET/PUT/DELETE/HEAD/MULTIPART UPLOAD)?

and also please share the gross sequence (how it works) on each operation for the review.

Key Salted Hash (for verification)

Let me confirm that it's meant to be hmac for Key + Salt? then where does the Salt come from? (I think the Salt also should be stored)

Also I've just noticed, while checking the SSEC spec https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html, a sentence "When you upload an object, Amazon S3 uses the encryption key you provide to apply AES-256 encryption to your data and removes the encryption key from memory.". "removes the encryption key from memory" part can be tough to implement for high-level language including erlang other than low-level ones like C/C++/Rust as there is no way to control the memory (memset should be called to overwrite the encryption key on memory). To comply with the spec above, we might have to implement the encryption/decryption logic in NIF. what do you think? > @yosukehara @windkit

windkit commented 6 years ago

To respond to GET, HEAD, we need to know the ETag and Object Size of the original file. Both checksum and dsize are now referred to that on-disk (after encryption), if we are not recording those, we need to decrypt, aka, disk read.

windkit commented 6 years ago

I was thinking about salted hashing the key and store it as $salt$hash, salt is randomly generated.

In Amazon S3 documentation, they are storing randomly salted HMAC value (probably HMAC with a master rotating key)

windkit commented 6 years ago

@mocchira

As the key is transferred to leo_gateway it is already in Erlang control somewhere, implementing the logic NIF does not seem to solve the problem at all.

And if there is a way to force Erlang GC particular binary, i think that solves the issue already. We keep the key until we finish replying should be sufficient enough.

mocchira commented 6 years ago

@windkit

To respond to GET, HEAD, we need to know the ETag and Object Size of the original file. Both checksum and dsize are now referred to that on-disk (after encryption), if we are not recording those, we need to decrypt, aka, disk read.

In case of GET, the decryption must happen so that only HEAD can benefit from the original fields right?

I was thinking about salted hashing the key and store it as $salt$hash, salt is randomly generated.

OK, I see.

As the key is transferred to leo_gateway it is already in Erlang control somewhere, implementing the logic NIF does not seem to solve the problem at all.

Right. (or implementing http handling on leo_gateway in NIF but that make no sense)

And if there is a way to force Erlang GC particular binary, i think that solves the issue already. We keep the key until we finish replying should be sufficient enough.

AFAIK, no way (only process heap can be GCed) so for now all we can do without any cost is just explicitly documenting that "removes the encryption key from memory" is not implemented yet. Thoughts?

mocchira commented 6 years ago

Note: Even GCed the memory segment including the key, since no memset is invoked at erlang runtime, the key keep residing in somewhere on memory until the same segment will be reused and overwritten. so it's impossible to achieve this spec in any language without having ability to directly control memory.

windkit commented 6 years ago

@mocchira

In case of GET, the decryption must happen so that only HEAD can benefit from the original fields right?

True.

AFAIK, no way (only process heap can be GCed) so for now all we can do without any cost is just explicitly documenting that "removes the encryption key from memory" is not implemented yet. Thoughts?

As Amazon S3 does not give a strong promise like "removes the encryption key right away / immediately", I don't think it is good to say that, it gives an impression that we are storing customers' keys somewhere. To me, we are still discarding the key after we are serving the data.

mocchira commented 6 years ago

@windkit

In case of GET, the decryption must happen so that only HEAD can benefit from the original fields right?

True.

then we have to compare the pros/cons on each whether storing the original fields or not

One way to compare the different unit (disk space used, disk iops/bandwidth, network bandwidth, cpu time) is using the cost in public clouds that enable us to normalize those metrics into cost so that how about making a decision based on this? > @windkit @yosukehara

mocchira commented 6 years ago

@windkit

As Amazon S3 does not give a strong promise like "removes the encryption key right away / immediately", I don't think it is good to say that, it gives impression that we are storing customers' keys somewhere.

Yes we have to take care not to worry users more than necessary so the actual sentence should be considered as much cares as possible.

To me, we are still discarding the key after we are serving the data.

As stated above comment, from the actual memory POV, it's true the raw key keep residing in memory so if the system was taken over by exploiting other vulnerability and also which allow hackers to peek the user land memory on any offset then there is possibility the key being leaked. so at very least we have to tell users this kind of risk is existing even if it's almost unlikely to happen.

mocchira commented 6 years ago

Reference SSE-C Impl on minio: https://github.com/minio/minio/blob/master/cmd/encryption-v1.go

yosukehara commented 6 years ago

@mocchira Thanks for sharing your comment. Let me ask you to share the draft design of the implementation after surveying Minio's implementation.

mocchira commented 6 years ago

The rational behind the impl on minio can be found on https://github.com/minio/sio/blob/master/DARE.md. Now I'm checking its correctness.

yosukehara commented 6 years ago

@mocchira

then we have to compare the pros/cons on each whether storing the original fields or not

Even if metadata's size little bit increases, I recommend storing original fields (size and checksum) because LeoFS' implementation keeps simplicity, and we can expect HEAD performance.

yosukehara commented 6 years ago

@mocchira Thanks for sharing minio/sio/blob/master/DARE.md

I've read Data At Rest Encryption (DARE) - Version 1.0, and then recognized that it is a suitable implementation for LeoFS' object encryption feature:

"DARE provides confidentiality and integrity of the encrypted data as long as the encryption key is never reused. This means that a different encryption key must be used for every data stream. See Appendix A for recommendations.

DARE needs a unique encryption key per data stream. The best approach to ensure that the keys are unique is to derive every encryption key from a master key."

mocchira commented 6 years ago

As for whether storing the original size and checksum, Since we've decided to adopt the requirement that "An attacker is defined as somebody who has full access to the encrypted data but not to the encryption key. An attacker can also act as storage provider." cited from DARE spec, storing the original checksum(md5) and its size could allow attackers to derive the original content by simply brute force attack based on the size and checksum (however ONLY in case its size is very small). so we have to prevent from storing those values on metadata for security.

The rational behind the impl on minio can be found on https://github.com/minio/sio/blob/master/DARE.md. Now I'm checking its correctness.

It looks great however I feel it's a little overkill (which is designed for the large data streams) to LeoFS as we split the large object into small chunks by ourselves. so I'd like to propose to reject it, instead we can only mimic https://github.com/minio/minio/blob/master/cmd/encryption-v1.go part while using raw AES through https://github.com/leo-project/leo_commons/blob/develop/src/leo_ssec.erl instead of DARE (sio).

mocchira commented 6 years ago

Design Draft and TODO for SSE-C on LeoFS

Spec

@yosukehara @windkit let me know if you have any question.

windkit commented 6 years ago

@mocchira if you are thinking about brute force with original size and original md5, I would rather brute force the encryption key, which is only 256-bit...... with the same rationale, we need to think about how to securely store the key checksums for verification.

mocchira commented 6 years ago

@windkit it's impossible to BFA against 256bit (check https://crypto.stackexchange.com/questions/1145/how-much-would-it-cost-in-u-s-dollars-to-brute-force-a-256-bit-key-in-a-year) OTOH, the size of an object which is supposed to encrypt could be more smaller ranging from 1 byte to theoretical limit around 10 bytes according to https://en.wikipedia.org/wiki/Brute-force_attack#Theoretical_limits.

yosukehara commented 6 years ago

@mocchira I've reviewed this comment. I agree with your proposal. We can finally start its implementation.

windkit commented 6 years ago

Status Update

Basic Implementation of SSEC PUT/GET (small object only, key verification skipped) https://github.com/windkit/leofs/tree/ssec

Related PR

https://github.com/leo-project/leo_object_storage/pull/28 https://github.com/leo-project/leo_commons/pull/13

windkit commented 6 years ago

TODO

SSE-C Testing

Wrapper around s3cmd

#!/bin/bash

secret="12345678901234567890123456789012"
key=$(echo -n $secret | base64)
key_md5=$(echo -n $secret | openssl dgst -md5 -binary | base64)

./s3cmd \
  --add-header=x-amz-server-side-encryption-customer-algorithm:AES256 \
  --add-header=x-amz-server-side-encryption-customer-key:"$key" \
  --add-header=x-amz-server-side-encryption-customer-key-MD5:"$key_md5" \
  $@

But it would complain about MD5 unmatch during PUT...

Modification on s3cmd-2.0.0

Need to modify the S3/S3.py for object get, which we need to pass the extra_headers (it does it in PUT only...)

--- a/s3cmd
+++ b/home/wilson/s3cmd-2.0.0/s3cmd
@@ -471,6 +471,7 @@ def cmd_object_get(args):
     ## Each item will be a dict with the following attributes
     # {'remote_uri', 'local_filename'}
     download_list = []
+    extra_headers = copy(cfg.extra_headers)

     if len(args) == 0:
         raise ParameterError("Nothing to download. Expecting S3 URI.")
@@ -570,7 +571,7 @@ def cmd_object_get(args):
                 continue
         try:
             try:
-                response = s3.object_get(uri, dst_stream, destination, start_position = start_position, extra_label = seq_label)
+                response = s3.object_get(uri, dst_stream, destination, extra_headers, start_position = start_position, extra_label = seq_label)
             finally:
                 dst_stream.close()
         except S3DownloadError as e:
--- a/S3/S3.py
+++ b/home/wilson/s3cmd-2.0.0/S3/S3.py
@@ -704,10 +704,11 @@ class S3(object):
         response = self.send_file(request, src_stream, labels)
         return response

-    def object_get(self, uri, stream, dest_name, start_position = 0, extra_label = ""):
+    def object_get(self, uri, stream, dest_name, extra_headers = None, start_position = 0, extra_label = ""):
         if uri.type != "s3":
             raise ValueError("Expected URI type 's3', got '%s'" % uri.type)
-        request = self.create_request("OBJECT_GET", uri = uri)
+        request = self.create_request("OBJECT_GET", uri = uri, headers =
+            extra_headers)
         labels = { 'source' : uri.uri(), 'destination' : dest_name, 'extra' : extra_label }
         response = self.recv_file(request, stream, labels, start_position)
         return response

Error Messages

I could not find document about the error messages in SSE-C, tried to use Google to deduce some. We may need to test it on S3 actually

Cache Handling

I think we would cache encrypted one for data privacy

Range Get Handling

Handle at leo_gateway

Pros:

  1. Do not need to send the key around

Cons:

  1. Increased Traffic
yosukehara commented 6 years ago

We've reached the consensus that Encrypt/Decrypt data is implemented on LeoStorage.

windkit commented 6 years ago

I would just copy to here for discussion then

yes, I have also thought about it, decrypt at leo_storage could be better for performance.

But decrypting at leo_storage means we have to send the key around, is the rpc traffic in Erlang encrypted?
Sending keys around is one issue, sending them in plain text is a big one.
windkit commented 6 years ago

looks like we can do over TLS https://www.erlang-solutions.com/blog/erlang-distribution-over-tls.html

do you have any idea about the performance overhead?

yosukehara commented 6 years ago

@windkit

I would just copy to here for discussion then

yes, I have also thought about it, decrypt at leo_storage could be better for performance.

But decrypting at leo_storage means we have to send the key around, is the rpc traffic in Erlang encrypted? Sending keys around is one issue, sending them in plain text is a big one.

looks like we can do over TLS https://www.erlang-solutions.com/blog/erlang-distribution-over-tls.html

do you have any idea about the performance overhead?

We need to consider its implementation way which includes your concern. Today, I’ve recognized this design is not mature because there is no detailed design. We need to implement its function in a reliable way, I believe. We need to write a design documentation especially new features. We're going to share its design later.

Links

mocchira commented 5 years ago

Reference: https://issues.apache.org/jira/browse/CASSANDRA-7922

It's a little bit different from the goal of the issue but we may have to encrypt not only files uploaded by users but also any other files used by LeoFS at rest in order to cover more strict compliance needs as described on the above link.