Google Clould KMS call hangs

[ShahNewazKhan]

Hello,

I am extending this plugin to include a GCP KMS decryption procedure inside the pubsub subscribe handler by creating a decrypt.java jar file that stalls when I call the decrpytWrappedKey method as described below:

public byte[] decryptWrappedKey(String wrappedKey, String cid)
      throws IOException {

    if (this.testMode) { logger.log(Level.INFO, "Decrypting wrapped key");}

    byte[] wrappedKeyBytes = Base64.decodeBase64(wrappedKey);

    // Create the KeyManagementServiceClient using try-with-resources to manage client cleanup.
    if (this.testMode) { logger.log(Level.INFO, "Creating KMS Client ...");}
    try {
      KeyManagementServiceClient client = KeyManagementServiceClient.create();
      logger.log(Level.INFO, "ENTERED KEY CREATION");
      String keyResourceName = CryptoKeyName.format(
        this.projectId,
        this.locationId,
        this.keyRingId,
        cid);

      // Decrypt the ciphertext with Cloud KMS.
      if (this.testMode) { logger.log(Level.INFO, "Decrypting KMS response"); }
      DecryptResponse response = client.decrypt(keyResourceName, ByteString.copyFrom(wrappedKeyBytes));
      if (this.testMode) { logger.log(Level.INFO, "Returning decrypted wraped key"); }
      client.close();
      logger.log(Level.INFO, "**************CLOSING KMS CLIENT ***********");
      return response.getPlaintext().toByteArray();
    }catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

The unit tests for this method runs fine, however when I jar it up and call it from within the subscribe handler in this plugin, it hangs at the client create portion.

I can see the ENTERED KEY CREATION log and then it just hangs, any ideas how I can go about debugging this?

[josephlewis42]

Hi @ShahNewazKhan,

My best guess is to look in the Google Cloud audit log to see if there are failing events. It looks like the service client is being created with whatever the default credentials are in your environment (the KeyManagementServiceClient.create portion) rather than using the same credential logic that the plugin has https://github.com/logstash-plugins/logstash-input-google_pubsub/blob/master/lib/logstash/inputs/google_pubsub.rb#L224-L227.

You might want to consider creating a logstash-filter plugin rather than modifying this one, that gives a few benefits:

• You can use whatever input plugin you want, decrypting events stored in GCS, BigQuery, Kafka and Pub/Sub all become the same.
• You can use separate credentials for the decryption and Pub/Sub portions without needing to trust the same ones to both.
• Logstash now supports "pure" Java plugins which can be easier to write.

Let me know if that helps!

[ShahNewazKhan]

Hi @josephlewis42,

Thanks for the input! I will try and resolve this as an extension for this plugin with the java jar for a quick poc win. If not I will pursue the the logstash-filter plugin method you have suggested.

Regarding the auth method, I tried passing in the fixedCredentialsProvider object used by the Pubsub client and it still hangs at the same point. Unfortunately the audit logs does not log any of the kms client create actions, only the subsequent get, list, decrypt, encrypt actions.

public byte[] decryptWrappedKey(String wrappedKey, String cid, FixedCredentialsProvider creds)
      throws IOException {

    if (this.testMode) { logger.log(Level.INFO, "Decrypting wrapped key");}

    byte[] wrappedKeyBytes = Base64.decodeBase64(wrappedKey);

    // Create the KeyManagementServiceClient using try-with-resources to manage client cleanup.
    if (this.testMode) { logger.log(Level.INFO, "Creating KMS Client");}
    try {
      if (this.testMode) { logger.log(Level.INFO, "Creating KMS settings");}
      KeyManagementServiceSettings keyManagementServiceSettings =
      KeyManagementServiceSettings.newBuilder()
          .setCredentialsProvider(creds)
          .build();

      if (this.testMode) { logger.log(Level.INFO, "Creating KMS settings");}
      KeyManagementServiceClient client =
          KeyManagementServiceClient.create(keyManagementServiceSettings);

      logger.log(Level.INFO, "ENTERED KEY CREATION");
      String keyResourceName = CryptoKeyName.format(
        this.projectId,
        this.locationId,
        this.keyRingId,
        cid);

      // Decrypt the ciphertext with Cloud KMS.
      if (this.testMode) { logger.log(Level.INFO, "Decrypting KMS response"); }
      DecryptResponse response = client.decrypt(keyResourceName, ByteString.copyFrom(wrappedKeyBytes));
      if (this.testMode) { logger.log(Level.INFO, "Returning decrypted wraped key"); }
      client.close();
      logger.log(Level.INFO, "**************CLOSING KMS CLIENT ***********");
      return response.getPlaintext().toByteArray();
    }catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

I will try to take the decrypt logic out of the .java file and re-write it as a jruby function and see if I get any more verbose log messages.

[ShahNewazKhan]

@josephlewis42 an update:

EDIT

I was able to find the relevant by instantiating the KeyManagementServiceClient as an instance variable. It seems the 'com.google.cloud', 'google-cloud-kms', '0.81.0-beta' jar I included using jar-dependencies is missing the com/google/cloud/kms/v1/ListKeyRingsRequest class.

java.lang.NoClassDefFoundError: com/google/cloud/kms/v1/ListKeyRingsRequest
        at com.google.cloud.kms.v1.stub.GrpcKeyManagementServiceStub.<clinit>(com/google/cloud/kms/v1/stub/GrpcKeyManagementServiceStub.java:88)
        at com.google.cloud.kms.v1.stub.KeyManagementServiceStubSettings.createStub(com/google/cloud/kms/v1/stub/KeyManagementServiceStubSetting
s.java:292)
        at com.google.cloud.kms.v1.KeyManagementServiceClient.<init>(com/google/cloud/kms/v1/KeyManagementServiceClient.java:154)
        at com.google.cloud.kms.v1.KeyManagementServiceClient.create(com/google/cloud/kms/v1/KeyManagementServiceClient.java:135)
        at com.google.cloud.kms.v1.KeyManagementServiceClient.create(com/google/cloud/kms/v1/KeyManagementServiceClient.java:126)

I have checked $CLASSPATH and found com/google/cloud/kms jar is listed, however that jar does not contain a com/google/cloud/kms/v1/ListKeyRingsRequest class file.