search

TritonDataCenter / java-manta

Java Manta Client SDK
Mozilla Public License 2.0
16 stars 26 forks source link

MANTA-5083 JAVA client is taking too long to get file when range start is big. #562

Closed kellymclaughlin closed 4 years ago

kellymclaughlin commented 4 years ago

The heart of this problem is that we need a way when using AES/CTR mode to increment the counter that's used along with the nonce or initialization vector (IV) as input to the decryption process based on the block that the bytes returned from the Range header request belong to. Presently, our solution to that is the calls to Cipher.update, but this is an expensive method when really all we need is to increment a counter to reflect the current block being decrypted. So instead what we can do is to take the IV and increment that value manually (I could not find any built-in Java API for this purpose) and then we can proceed to decrypt the block just like before. Calculating this block-targeted IV value is very cheap compared to the update calls.

I created a test program that riffs on the Range header example already in the repo. It uploads an object to Manta and then downloads 1024 bytes chunks from the beggining, middle, and end of the object and reports the time it took and if the downloaded data matched what we expected. Here is the full code I used:

/*
 * Copyright 2020 Joyent, Inc. All rights reserved.
 *
 * This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/.
 */

import com.joyent.manta.client.MantaClient;
import com.joyent.manta.client.MantaObjectResponse;
import com.joyent.manta.config.ChainedConfigContext;
import com.joyent.manta.config.ConfigContext;
import com.joyent.manta.config.DefaultsConfigContext;
import com.joyent.manta.config.EncryptionAuthenticationMode;
import com.joyent.manta.config.EnvVarConfigContext;
import com.joyent.manta.config.MapConfigContext;
import com.joyent.manta.http.MantaHttpHeaders;

import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import java.util.Scanner;
import java.util.Arrays;

/*
* Usage: set the mantaUserName, privateKeyPath, and publicKeyId with your own values.
 */
public class ClientEncryptionRangeDownload {

    public static void main(String... args) throws IOException {
        String mantaUserName = "kelly";
        String privateKeyPath = "/home/kelly/.ssh/id_rsa";
        String publicKeyId = "39:98:02:54:3f:d5:28:df:09:76:39:62:94:5f:c6:49";

        ConfigContext config = new ChainedConfigContext(
                new DefaultsConfigContext(),
                new EnvVarConfigContext(),
                new MapConfigContext(System.getProperties()))
                .setMantaURL("https://manta.coal.joyent.us")
                // If there is no subuser, then just use the account name
                .setMantaUser(mantaUserName)
                .setMantaKeyPath(privateKeyPath)
                .setMantaKeyId(publicKeyId)
                .setClientEncryptionEnabled(true)
                .setEncryptionAlgorithm("AES256/CTR/NoPadding")
                .setEncryptionAuthenticationMode(EncryptionAuthenticationMode.Optional)
                .setPermitUnencryptedDownloads(false)
                .setEncryptionKeyId("simple/example")
                .setEncryptionPrivateKeyBytes(Base64.getDecoder().decode("RkZGRkZGRkJEOTY3ODNDNkM5MUUyMjIyMTExMTIyMjI="));

        try (MantaClient client = new MantaClient(config)) {
            String mantaPath = MantaClient.SEPARATOR + mantaUserName + "/stor/foo";

            byte[] buf = new byte[100000000];
            Arrays.fill(buf, 0, 55555000, (byte)'a');
            Arrays.fill(buf, 55555000, 99998976, (byte)'b');
            Arrays.fill(buf, 99998976, 100000000, (byte)'c');

            String plaintext = new String(buf, StandardCharsets.UTF_8.name());

            MantaObjectResponse response = client.put(mantaPath, plaintext);

            // Read the first chunk of bytes from the uploaded file
            MantaHttpHeaders headers = new MantaHttpHeaders();
            headers.setByteRange(0L, 1023L);

            long start1 = System.currentTimeMillis();

            byte[] readBuf = new byte[1024];

            byte[] expectedBytes1 = new byte[1024];
            Arrays.fill(expectedBytes1, (byte)'a');
            byte[] expectedBytes2 = new byte[1024];
            Arrays.fill(expectedBytes2, (byte)'b');
            byte[] expectedBytes3 = new byte[1024];
            Arrays.fill(expectedBytes3, (byte)'c');

            String readData = new String("");

            try (InputStream is = client.getAsInputStream(mantaPath, headers);
                 Scanner scanner = new Scanner(is, StandardCharsets.UTF_8.name())) {

                while (scanner.hasNextLine()) {
                    readData += scanner.nextLine();
                }
            }

            long end1 = System.currentTimeMillis();
            long timeElapsed = end1 - start1;
            System.out.println("Time to fetch first chunk: " + timeElapsed);

            readBuf = readData.getBytes();

            if (Arrays.equals(expectedBytes1, readBuf)) {
                System.out.println("First chunk of data matched expected data");
            }

            readData = "";

            // Read a middle chunk of the file
            headers.setByteRange(55555000L, 55556023L);

            long start2 = System.currentTimeMillis();

            try (InputStream is = client.getAsInputStream(mantaPath, headers);
                 Scanner scanner = new Scanner(is, StandardCharsets.UTF_8.name())) {

                while (scanner.hasNextLine()) {
                    readData += scanner.nextLine();
                }
            }

            long end2 = System.currentTimeMillis();
            long timeElapsed2 = end2 - start2;
            System.out.println("Time to fetch middle chunk: " + timeElapsed2);

            readBuf = readData.getBytes();

            if (Arrays.equals(expectedBytes2, readBuf)) {
                System.out.println("Middle chunk of data matched expected data");
            }

            readData = "";

            //Read last chunk of the file
            headers.setByteRange(99998976L, 99999999L);

            long start3 = System.currentTimeMillis();

            try (InputStream is = client.getAsInputStream(mantaPath, headers);
                 Scanner scanner = new Scanner(is, StandardCharsets.UTF_8.name())) {

                while (scanner.hasNextLine()) {
                    readData += scanner.nextLine();
                }
            }

            long end3 = System.currentTimeMillis();
            long timeElapsed3 = end3 - start3;
            System.out.println("Time to fetch last chunk: " + timeElapsed3);

            readBuf = readData.getBytes();

            if (Arrays.equals(expectedBytes3, readBuf)) {
                System.out.println("Last chunk of data matched expected data");
            }
        }
    }
}

Here is a run of the test program without the changes for this PR (i.e. It is using the Cipher.update method to increment the AES/CTR counter):

[kelly@mantadev java-manta-cse-test]$ java -cp ./target/cse-test-1.0-SNAPSHOT.jar ClientEncryptionRangeDownload
16:19:20.903 [main] DEBUG com.joyent.manta.client.crypto.ExternalSecurityProviderLoader - Bouncy Castle provider was not loaded, adding to providers
16:19:20.919 [main] INFO com.joyent.manta.client.crypto.ExternalSecurityProviderLoader - Security provider chosen for CSE: BC version 1.61 
16:19:20.920 [main] DEBUG com.joyent.manta.client.MantaClient - Preferred Security Provider: BC version 1.61
16:19:20.993 [main] WARN com.joyent.manta.http.MantaSSLConnectionSocketFactory - Configuration: tlsInsecure is true.  ALL TLS VERIFICATION IS DISABLED!
16:19:21.599 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Secret key id: simple/example
16:19:21.600 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encryption type: client/1
16:19:21.600 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encryption cipher: AES256/CTR/NoPadding
16:19:21.601 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - IV: e0c84bb6823ea2fc0fa44622ba073086
16:19:21.601 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - HMAC algorithm: HmacMD5
16:19:21.601 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Plaintext content-length: 100000000
16:19:21.602 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata IV: 39911fce41ae93ad5b1d92a85ec8bac6
16:19:21.604 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata plaintext:
e-content-type: text/plain; charset=UTF-8

16:19:21.604 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata ciphertext: OZEfzkGuk61bHZKoXsi6xg==
16:19:21.606 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata HMAC: Rq81Kksb9ObxyZ9lmUlBDg==
16:19:21.606 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - PUT    /kelly/stor/foo
16:19:21.737 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Enabled TLS protocols: TLSv1.2
16:19:21.737 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Enabled cipher suites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256
16:19:21.737 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Supported TLS protocols: TLSv1.2
16:19:21.737 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Supported cipher suites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256
16:19:30.026 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - PUT    /kelly/stor/foo response [204] No Content 
16:19:30.182 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch first chunk: 166
First chunk of data matched expected data
16:19:30.341 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch middle chunk: 1331
Middle chunk of data matched expected data
16:19:31.668 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch last chunk: 2208
Last chunk of data matched expected data

As has been reported the time complete the request for each file chunk increases as the offset into the object increases.

Now here is a run of the program with the changes from this PR:

[kelly@mantadev java-manta-cse-test]$ java -cp ./target/cse-test-1.0-SNAPSHOT.jar ClientEncryptionRangeDownload
16:17:48.319 [main] DEBUG com.joyent.manta.client.crypto.ExternalSecurityProviderLoader - Bouncy Castle provider was not loaded, adding to providers
16:17:48.338 [main] INFO com.joyent.manta.client.crypto.ExternalSecurityProviderLoader - Security provider chosen for CSE: BC version 1.61 
16:17:48.339 [main] DEBUG com.joyent.manta.client.MantaClient - Preferred Security Provider: BC version 1.61
16:17:48.417 [main] WARN com.joyent.manta.http.MantaSSLConnectionSocketFactory - Configuration: tlsInsecure is true.  ALL TLS VERIFICATION IS DISABLED!
16:17:49.032 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Secret key id: simple/example
16:17:49.033 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encryption type: client/1
16:17:49.033 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encryption cipher: AES256/CTR/NoPadding
16:17:49.034 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - IV: 13ac4a9d39378fc7f16c2ef8458db5d8
16:17:49.034 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - HMAC algorithm: HmacMD5
16:17:49.034 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Plaintext content-length: 100000000
16:17:49.035 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata IV: 888b6ee6ed4f95526da61bc07a8911ed
16:17:49.036 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata plaintext:
e-content-type: text/plain; charset=UTF-8

16:17:49.036 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata ciphertext: iItu5u1PlVJtphvAeokR7Q==
16:17:49.036 [main] DEBUG com.joyent.manta.http.EncryptionHttpHelper - Encrypted metadata HMAC: LiGvRuT8fCf7OYL4zj08TA==
16:17:49.036 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - PUT    /kelly/stor/foo
16:17:49.146 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Enabled TLS protocols: TLSv1.2
16:17:49.147 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Enabled cipher suites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256
16:17:49.147 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Supported TLS protocols: TLSv1.2
16:17:49.147 [main] DEBUG com.joyent.manta.http.MantaSSLConnectionSocketFactory - Supported cipher suites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256
16:17:55.535 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - PUT    /kelly/stor/foo response [204] No Content 
16:17:55.697 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch first chunk: 172
First chunk of data matched expected data
16:17:55.856 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch middle chunk: 149
Middle chunk of data matched expected data
16:17:55.996 [main] DEBUG com.joyent.manta.http.StandardHttpHelper - GET    [] response [206] Partial Content 
Time to fetch last chunk: 142
Last chunk of data matched expected data

The reported times for each chunk request are now within the same order of magnitude which is exactly what we want while the data for each chunk still matches our expected data indicating successful decryption.

indianwhocodes commented 4 years ago

All tests passed:

[INFO] Reactor Summary:
[INFO] 
[INFO] java-manta 3.4.2-SNAPSHOT .......................... SUCCESS [  5.250 s]
[INFO] java-manta-client-unshaded ......................... SUCCESS [06:48 min]
[INFO] java-manta-client .................................. SUCCESS [  8.782 s]
[INFO] java-manta-client-kryo-serialization ............... SUCCESS [ 39.553 s]
[INFO] java-manta-cli ..................................... SUCCESS [02:16 min]
[INFO] java-manta-it ...................................... SUCCESS [19:45 min]
[INFO] java-manta-benchmark ............................... SUCCESS [02:12 min]
[INFO] java-manta-examples 3.4.2-SNAPSHOT ................. SUCCESS [  0.528 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 31:57 min
[INFO] Finished at: 2020-03-20T15:21:01-07:00
[INFO] ------------------------------------------------------------------------