Soluto / kamus

An open source, git-ops, zero-trust secret encryption and decryption solution for Kubernetes applications
https://kamus.soluto.io
Apache License 2.0
929 stars 68 forks source link

Encryptor mostly fails using AWS KMS provider. #508

Closed mcanaves closed 4 years ago

mcanaves commented 4 years ago

Describe the bug Kamus mostly fails to encrypt values when using AWS KMS provider. Looking at encryptor logs I see that fails because key doesn't exist but at AWS the key exist.

Versions used Kamus (API images): 0.6.2.0 Kamus CLI: 0.3.0 Chart version: 0.4.6 KMS provider: AwsKms Kubernetes flavour and version: 1.14.9-aws.8

To Reproduce Steps to reproduce the behavior:

  1. Encrypt value using the CLI with AwsKms as provider

Expected behavior Always encrypt the value.

omerlh commented 4 years ago

Can you please share the logs? Also, maybe it's a permissions issue? Can you share the role Kamus has?

mcanaves commented 4 years ago
{
  "Timestamp": "2020-04-23T14:26:13.9530270+00:00",
  "Level": "Error",
  "MessageTemplate": "Unhandled exception while processing request",
  "Exception": "Amazon.KeyManagementService.Model.NotFoundException: Key 'arn:aws:kms:***:********:key/8d6c80c9-e77f-407c-bd0d-da3dcf82a799' does not exist\n ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.\n at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)\n at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)\n --- End of inner exception stack trace ---\n at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)\n at Amazon.Runtime.Internal.ExceptionHandler`1.Handle(IExecutionContext executionContext, Exception exception)\n at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)\n at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)\n at Kamus.KeyManagement.AwsKeyManagement.GenerateMasterKey(String keyAlias) in /app/key-managment/AwsKeyManagement.cs:line 94\n at Kamus.KeyManagement.AwsKeyManagement.GenerateEncryptionKey(String keyAlias) in /app/key-managment/AwsKeyManagement.cs:line 68\n at Kamus.KeyManagement.AwsKeyManagement.Encrypt(String data, String serviceAccountId, Boolean createKeyIfMissing) in /app/key-managment/AwsKeyManagement.cs:line 33\n at Kamus.Controllers.EncryptController.Encrypt(EncryptRequest body) in /app/encrypt-api/Controllers/EncryptController.cs:line 45\n at lambda_method(Closure , Object )\n at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()\n at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)\n at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n at App.Metrics.AspNetCore.Tracking.Middleware.ErrorRequestMeterMiddleware.Invoke(HttpContext context)\n at Kamus.ErrorHandlingMiddleware.Invoke(HttpContext httpContext) in /app/encrypt-api/ErrorHandlingMiddleware.cs:line 31",
  "Properties": {
    "SourceContext": "Kamus.ErrorHandlingMiddleware"
  }
}
omerlh commented 4 years ago

If the key exists, this is probably a permissions issue...

mcanaves commented 4 years ago

Also, maybe it's a permissions issue?

I don't think so because some times works well and I'm able to encrypt the values.

Can you share the role Kamus has?

Sure (I'm preparing a PR to add this info to the docs). Right now we are using the node wide roles to define the policy. The policy is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:EnableKeyRotation",
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:GenerateDataKey",
                "kms:CreateAlias"
            ],
            "Resource": [
                "arn:aws:kms:***:********:key/*",
                "arn:aws:kms:***:********:alias/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "kms:CreateKey",
            "Resource": "*"
        }
    ]
}
omerlh commented 4 years ago

I think need also list/get, but I'm not sure... Looking at the code, it fails on creating the alias for the key... Meaning, the key was created successfully. Can you please give it also permissions to get the key?

shaikatz commented 4 years ago

Hello @mcanaves It feels like some race condition with the key creation. If you try to run the same encrypt command, does it work on the 2nd try?

mcanaves commented 4 years ago

It feels like some race condition with the key creation.

Yep, I figure that's what happens.

If you try to run the same encrypt command, does it work on the 2nd try?

Not always, some times I can try 10 times that always fails and then maybe 3 successive attempts works. This is why I think on a race-condition problem. If it walks like a duck and it quacks like a duck... :D

shaikatz commented 4 years ago

It's super weird, let me try to reproduce it...

mcanaves commented 4 years ago

Can you please give it also permissions to get the key?

@omerlh I tried with all read permissions and same problem.

mcanaves commented 4 years ago

Sorry to not be more helpful but I have no idea of C#. Looking at the code where it fails I see that between creating the key and creating the alias the rotation of the key is activated if this option is enabled and I have this option enabled.

I tried with key rotation disabled to see what happens and now works every time that I tested.

Looks that with rotation enabled the key creation is taking more time and when label creation happens the key didn't not exist yet.

shaikatz commented 4 years ago

Thanks for the additional info @mcanaves it helps. I was able to reproduce it locally, I'll try to understand why it happens and hopefully submit a fix.

shaikatz commented 4 years ago

Hi @mcanaves the fix was released in Kamus version 0.6.6.0 (chart version 0.4.7) Please give it a go and let me know if it solved your issue.

mcanaves commented 4 years ago

Thanks @shaikatz, it seems to work every time now.

t0st commented 4 years ago

Just for others which stumble upon this issue: List Access IS required to successfully encrypt a secret.

My policy (taken from above and modified) looks like this:

`

{"Version": "2012-10-17",

"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "kms:EnableKeyRotation",
            "kms:Decrypt",
            "kms:Encrypt",
            "kms:GenerateDataKey",
            "kms:CreateAlias"
        ],
        "Resource": [
            "arn:aws:kms:***:********:key/*",
            "arn:aws:kms:***:********:alias/*"
        ]
    },
    {
        "Effect": "Allow",
        "Action": [
            "kms:ListKeys",
            "kms:ListAliases",
            "kms:CreateKey"
        ],
        "Resource": "*"
    }
]

}

`

Without ListKeys and ListAliases I got this error:

"Exception": "Amazon.KeyManagementService.Model.NotFoundException: Alias arn:aws:kms: :alias//***** is not found. .\n ---> Amazon.Runtime.Internal.HttpErrorResponseException