Encryptor mostly fails using AWS KMS provider.

[mcanaves]

Describe the bug Kamus mostly fails to encrypt values when using AWS KMS provider. Looking at encryptor logs I see that fails because key doesn't exist but at AWS the key exist.

Versions used Kamus (API images): 0.6.2.0 Kamus CLI: 0.3.0 Chart version: 0.4.6 KMS provider: AwsKms Kubernetes flavour and version: 1.14.9-aws.8

To Reproduce Steps to reproduce the behavior:

1. Encrypt value using the CLI with AwsKms as provider

Expected behavior Always encrypt the value.

[omerlh]

Can you please share the logs? Also, maybe it's a permissions issue? Can you share the role Kamus has?

[mcanaves]

{
  "Timestamp": "2020-04-23T14:26:13.9530270+00:00",
  "Level": "Error",
  "MessageTemplate": "Unhandled exception while processing request",
  "Exception": "Amazon.KeyManagementService.Model.NotFoundException: Key 'arn:aws:kms:***:********:key/8d6c80c9-e77f-407c-bd0d-da3dcf82a799' does not exist\n ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.\n at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)\n at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)\n --- End of inner exception stack trace ---\n at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)\n at Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext executionContext, Exception exception)\n at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)\n at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Kamus.KeyManagement.AwsKeyManagement.GenerateMasterKey(String keyAlias) in /app/key-managment/AwsKeyManagement.cs:line 94\n at Kamus.KeyManagement.AwsKeyManagement.GenerateEncryptionKey(String keyAlias) in /app/key-managment/AwsKeyManagement.cs:line 68\n at Kamus.KeyManagement.AwsKeyManagement.Encrypt(String data, String serviceAccountId, Boolean createKeyIfMissing) in /app/key-managment/AwsKeyManagement.cs:line 33\n at Kamus.Controllers.EncryptController.Encrypt(EncryptRequest body) in /app/encrypt-api/Controllers/EncryptController.cs:line 45\n at lambda_method(Closure , Object )\n at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n at App.Metrics.AspNetCore.Tracking.Middleware.ErrorRequestMeterMiddleware.Invoke(HttpContext context)\n at Kamus.ErrorHandlingMiddleware.Invoke(HttpContext httpContext) in /app/encrypt-api/ErrorHandlingMiddleware.cs:line 31",
  "Properties": {
    "SourceContext": "Kamus.ErrorHandlingMiddleware"
  }
}

[omerlh]

If the key exists, this is probably a permissions issue...

[mcanaves]

Also, maybe it's a permissions issue?

I don't think so because some times works well and I'm able to encrypt the values.

Can you share the role Kamus has?

Sure (I'm preparing a PR to add this info to the docs). Right now we are using the node wide roles to define the policy. The policy is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:EnableKeyRotation",
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:GenerateDataKey",
                "kms:CreateAlias"
            ],
            "Resource": [
                "arn:aws:kms:***:********:key/*",
                "arn:aws:kms:***:********:alias/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "kms:CreateKey",
            "Resource": "*"
        }
    ]
}

[omerlh]

I think need also list/get, but I'm not sure... Looking at the code, it fails on creating the alias for the key... Meaning, the key was created successfully. Can you please give it also permissions to get the key?

[shaikatz]

Hello @mcanaves It feels like some race condition with the key creation. If you try to run the same encrypt command, does it work on the 2nd try?

[mcanaves]

It feels like some race condition with the key creation.

Yep, I figure that's what happens.

If you try to run the same encrypt command, does it work on the 2nd try?

Not always, some times I can try 10 times that always fails and then maybe 3 successive attempts works. This is why I think on a race-condition problem. If it walks like a duck and it quacks like a duck... :D

[shaikatz]

It's super weird, let me try to reproduce it...

[mcanaves]

Can you please give it also permissions to get the key?

@omerlh I tried with all read permissions and same problem.

[mcanaves]

Sorry to not be more helpful but I have no idea of C#. Looking at the code where it fails I see that between creating the key and creating the alias the rotation of the key is activated if this option is enabled and I have this option enabled.

I tried with key rotation disabled to see what happens and now works every time that I tested.

Looks that with rotation enabled the key creation is taking more time and when label creation happens the key didn't not exist yet.

[shaikatz]

Thanks for the additional info @mcanaves it helps. I was able to reproduce it locally, I'll try to understand why it happens and hopefully submit a fix.

[shaikatz]

Hi @mcanaves the fix was released in Kamus version 0.6.6.0 (chart version 0.4.7) Please give it a go and let me know if it solved your issue.

[mcanaves]

Thanks @shaikatz, it seems to work every time now.

[t0st]

Just for others which stumble upon this issue: List Access IS required to successfully encrypt a secret.

My policy (taken from above and modified) looks like this:

`

{"Version": "2012-10-17",

"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "kms:EnableKeyRotation",
            "kms:Decrypt",
            "kms:Encrypt",
            "kms:GenerateDataKey",
            "kms:CreateAlias"
        ],
        "Resource": [
            "arn:aws:kms:***:********:key/*",
            "arn:aws:kms:***:********:alias/*"
        ]
    },
    {
        "Effect": "Allow",
        "Action": [
            "kms:ListKeys",
            "kms:ListAliases",
            "kms:CreateKey"
        ],
        "Resource": "*"
    }
]

}

`

Without ListKeys and ListAliases I got this error:

"Exception": "Amazon.KeyManagementService.Model.NotFoundException: Alias arn:aws:kms: :alias//***** is not found. .\n ---> Amazon.Runtime.Internal.HttpErrorResponseException