nsmithuk / local-kms

A mock version of AWS' Key Management Service, for local development and testing.
MIT License
198 stars 34 forks source link

can't decrytp the data #11

Closed ozbillwang closed 4 years ago

ozbillwang commented 4 years ago

I start up local-kms on port 8081 (because localstack is started at port 8080)

https://github.com/localstack/localstack/issues/206#issuecomment-542960650

Here is my test result, that I can encrypt data, but can't decrypt it.

# Create key is fine.
$ curl -X "POST" "http://localhost:8081/"      -H 'X-Amz-Target: TrentService.CreateKey'      -H 'Content-Type: application/json; charset=utf-8'      -d $'{}'
{"KeyMetadata":{"AWSAccountId":"111122223333","Arn":"arn:aws:kms:eu-west-2:111122223333:key/67c4b713-0ab3-4be2-aac0-6cfd2be15125","CreationDate":1571278255,"Description":"","Enabled":true,"KeyId":"67c4b713-0ab3-4be2-aac0-6cfd2be15125","KeyManager":"CUSTOMER","KeyState":"Enabled","KeyUsage":"ENCRYPT_DECRYPT","Origin":"AWS_KMS"}}

# encrypt data is fine.
$ curl -X "POST" "http://localhost:8081/"      -H 'X-Amz-Target: TrentService.Encrypt'      -H 'Content-Type: application/json; charset=utf-8'      -d $'{
  "KeyId": "bc436485-5092-42b8-92a3-0aa8b93536dc",
  "Plaintext": "SGVsbG8="
}'
{"KeyId":"arn:aws:kms:eu-west-2:111122223333:key/bc436485-5092-42b8-92a3-0aa8b93536dc","CiphertextBlob":"S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5L2JjNDM2NDg1LTUwOTItNDJiOC05MmEzLTBhYThiOTM1MzZkYwAAAACH7BBmLEn+3YuYzVEnu1ABJSPrqjz8epykbOnSHwS7gA4="}

# But can't decrypt the data
$ curl -X "POST" "http://localhost:8081/"      -H 'X-Amz-Target: TrentService.Decrypt'      -H 'Content-Type: application/json; charset=utf-8'      -d $'{
  "KeyId": "bc436485-5092-42b8-92a3-0aa8b93536dc",
  "CiphertextBlob": "S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5L2JjNDM2NDg1LTUwOTItNDJiOC05MmEzLTBhYThiOTM1MzZkYwAAAAD39qJkWhnxpBI+ZDosHf3vMcphFfUHYGQ9P9JMzGdLLsYHEWRaw80hxArEdRwt3eI1W6sJcSOjOXLyrvw="
}'

{"__type":"InvalidCiphertextException"}
nsmithuk commented 4 years ago

Hi @ozbillwang

In your example test above, the CiphertextBlob that you pass to the decrypt method needs to exactly match the CiphertextBlob returned from the encrypt operation.

So in your example, you'd need to pass decrypt "CiphertextBlob":"S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5L2JjNDM2NDg1LTUwOTItNDJiOC05MmEzLTBhYThiOTM1MzZkYwAAAACH7BBmLEn+3YuYzVEnu1ABJSPrqjz8epykbOnSHwS7gA4=".

Here's an example of the full chain that I've just run, done with httpie to get pretty outputs:

http --json POST http://localhost:8081/ X-Amz-Target:TrentService.CreateKey
HTTP/1.1 200 OK
Content-Length: 329
Content-Type: application/x-amz-json-1.1
Date: Thu, 17 Oct 2019 09:24:50 GMT

{
    "KeyMetadata": {
        "AWSAccountId": "111122223333",
        "Arn": "arn:aws:kms:eu-west-2:111122223333:key/46d26eff-112b-4c64-9723-a8252d4de7b9",
        "CreationDate": 1571304290,
        "Description": "",
        "Enabled": true,
        "KeyId": "46d26eff-112b-4c64-9723-a8252d4de7b9",
        "KeyManager": "CUSTOMER",
        "KeyState": "Enabled",
        "KeyUsage": "ENCRYPT_DECRYPT",
        "Origin": "AWS_KMS"
    }
}

We need to take KeyId from above and pass it as an input to encrypt.

http --json POST http://localhost:8081/ X-Amz-Target:TrentService.Encrypt KeyId=46d26eff-112b-4c64-9723-a8252d4de7b9 Plaintext='SGVsbG8='
HTTP/1.1 200 OK
Content-Length: 259
Content-Type: application/x-amz-json-1.1
Date: Thu, 17 Oct 2019 09:25:36 GMT

{
    "CiphertextBlob": "S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5LzQ2ZDI2ZWZmLTExMmItNGM2NC05NzIzLWE4MjUyZDRkZTdiOQAAAAAFZhJPGVEwyrKX4XLO9vjlQsv9z74g8qz0DZIaWAI4NCo=",
    "KeyId": "arn:aws:kms:eu-west-2:111122223333:key/46d26eff-112b-4c64-9723-a8252d4de7b9"
}

We need to take CiphertextBlob from above and pass it as an input to decrypt.

http --json POST http://localhost:8081/ X-Amz-Target:TrentService.Decrypt CiphertextBlob='S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5LzQ2ZDI2ZWZmLTExMmItNGM2NC05NzIzLWE4MjUyZDRkZTdiOQAAAAAFZhJPGVEwyrKX4XLO9vjlQsv9z74g8qz0DZIaWAI4NCo='
HTTP/1.1 200 OK
Content-Length: 110
Content-Type: application/x-amz-json-1.1
Date: Thu, 17 Oct 2019 09:26:15 GMT

{
    "KeyId": "arn:aws:kms:eu-west-2:111122223333:key/46d26eff-112b-4c64-9723-a8252d4de7b9",
    "Plaintext": "SGVsbG8="
}
nsmithuk commented 4 years ago

Also noticed that the README showed KeyId superfluously being passed into decrypt; but that wouldn't haven any bearing on the issue you saw.

ozbillwang commented 4 years ago

Thanks, it works now.

ozbillwang commented 4 years ago

I am surprise, I am successful to run with awslocal, that's great. But the default port has to be 4599 .

NOTES: not sure why and how to customize the kms localhost port. I searched in both github localstack and awscli-local, no explanation about this port.

So the docker-compose.yml can be updated to

version: '3'

services:
  localstack:
    image: localstack/localstack
    ports:
      - "4567-4597:4567-4597"
      - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"
    environment:
      - SERVICES=${SERVICES- }
      - DEBUG=${DEBUG- }
      - DATA_DIR=${DATA_DIR- }
      - PORT_WEB_UI=${PORT_WEB_UI- }
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
      - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

  kms:
    image: nsmithuk/local-kms
    volumes:
      - ./init:/init
      - ./data:/data
    ports:
      - 4599:8080

Now we are free to run the test with awslocal

$ awslocal kms create-key
{
    "KeyMetadata": {
        "AWSAccountId": "111122223333",
        "KeyId": "950b50d4-7b20-4081-8f73-007ce4462049",
        "Arn": "arn:aws:kms:eu-west-2:111122223333:key/950b50d4-7b20-4081-8f73-007ce4462049",
        "CreationDate": 1571610710,
        "Enabled": true,
        "Description": "",
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyState": "Enabled",
        "Origin": "AWS_KMS",
        "KeyManager": "CUSTOMER"
    }
}

$ awslocal kms list-keys
{
    "Keys": [
        ...,
        {
            "KeyId": "950b50d4-7b20-4081-8f73-007ce4462049",
            "KeyArn": "arn:aws:kms:eu-west-2:111122223333:key/950b50d4-7b20-4081-8f73-007ce4462049"
        },
        ....
    ]
}

It will be more convenient to run the test more than curl.

Let me think if I can raise the PR to update the README.

@nsmithuk

Are you fine to add a new session in README on how to run test with awslocal? Can you also confirm if this way works with your design?

ozbillwang commented 4 years ago

Seems I can only encrypt the plaintext but can't decrypt with awslocal,

$ awslocal kms encrypt --key-id "950b50d4-7b20-4081-8f73-007ce4462049" --plaintext "SGVsbG8="
{
    "CiphertextBlob": "S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5Lzk1MGI1MGQ0LTdiMjAtNDA4MS04ZjczLTAwN2NlNDQ2MjA0OQAAAAB4i9BkwYaqblQtEkP5IAiT3SqnQDaOIfnCUZl69FFIrab2rvw=",
    "KeyId": "arn:aws:kms:eu-west-2:111122223333:key/950b50d4-7b20-4081-8f73-007ce4462049"
}

$ awslocal kms decrypt --ciphertext-blob "S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5Lzk1MGI1MGQ0LTdiMjAtNDA4MS04ZjczLTAwN2NlNDQ2MjA0OQAAAAB4i9BkwYaqblQtEkP5IAiT3SqnQDaOIfnCUZl69FFIrab2rvw="

An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

Share the debug logs for your reference:

$ awslocal kms decrypt --ciphertext-blob "S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5L2JjNDM2NDg1LTUwOTItNDJiOC05MmEzLTBhYThiOTM1MzZkYwAAAABR1nwc3Dzv18+kwzWNk13jZ2a8ypLCwjyjgE9d1U6NGG+u/FA=" --debug
2019-10-21 10:33:53,098 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.16.138 Python/3.6.8 Linux/5.0.0-27-generic botocore/1.12.128
2019-10-21 10:33:53,099 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--endpoint-url=http://localhost:4599', 'kms', 'decrypt', '--ciphertext-blob', 'S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5L2JjNDM2NDg1LTUwOTItNDJiOC05MmEzLTBhYThiOTM1MzZkYwAAAABR1nwc3Dzv18+kwzWNk13jZ2a8ypLCwjyjgE9d1U6NGG+u/FA=', '--debug']
2019-10-21 10:33:53,099 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7faf38518bf8>
2019-10-21 10:33:53,099 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7faf39086b70>
2019-10-21 10:33:53,099 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7faf3903d598>
2019-10-21 10:33:53,106 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7faf38661d90>
2019-10-21 10:33:53,107 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/kms/2014-11-01/service-2.json
2019-10-21 10:33:53,110 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.kms: calling handler <function register_retries_for_service at 0x7faf39a330d0>
2019-10-21 10:33:53,110 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: kms
2019-10-21 10:33:53,112 - MainThread - botocore.hooks - DEBUG - Event building-command-table.kms: calling handler <function add_waiters at 0x7faf385281e0>
2019-10-21 10:33:53,122 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('ciphertext-blob', <awscli.arguments.CLIArgument object at 0x7faf383a82b0>), ('encryption-context', <awscli.arguments.CLIArgument object at 0x7faf383a8390>), ('grant-tokens', <awscli.arguments.ListArgument object at 0x7faf383a83c8>)])
2019-10-21 10:33:53,123 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kms.decrypt: calling handler <function add_streaming_output_arg at 0x7faf385201e0>
2019-10-21 10:33:53,123 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kms.decrypt: calling handler <function add_cli_input_json at 0x7faf3903dbf8>
2019-10-21 10:33:53,124 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kms.decrypt: calling handler <function unify_paging_params at 0x7faf38595e18>
2019-10-21 10:33:53,133 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/kms/2014-11-01/paginators-1.json
2019-10-21 10:33:53,134 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.kms.decrypt: calling handler <function add_generate_skeleton at 0x7faf3857ad08>
2019-10-21 10:33:53,135 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.kms.decrypt: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7faf383a8470>>
2019-10-21 10:33:53,135 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.kms.decrypt: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7faf383a8358>>
2019-10-21 10:33:53,138 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kms.decrypt.ciphertext-blob: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faf3a053780>
2019-10-21 10:33:53,139 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.kms.decrypt: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7faf39082198>
2019-10-21 10:33:53,139 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5L2JjNDM2NDg1LTUwOTItNDJiOC05MmEzLTBhYThiOTM1MzZkYwAAAABR1nwc3Dzv18+kwzWNk13jZ2a8ypLCwjyjgE9d1U6NGG+u/FA=' for parameter "ciphertext_blob": 'S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5L2JjNDM2NDg1LTUwOTItNDJiOC05MmEzLTBhYThiOTM1MzZkYwAAAABR1nwc3Dzv18+kwzWNk13jZ2a8ypLCwjyjgE9d1U6NGG+u/FA='
2019-10-21 10:33:53,140 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kms.decrypt.encryption-context: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faf3a053780>
2019-10-21 10:33:53,140 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kms.decrypt.grant-tokens: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faf3a053780>
2019-10-21 10:33:53,140 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kms.decrypt.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faf3a053780>
2019-10-21 10:33:53,141 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.kms.decrypt.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7faf3a053780>
2019-10-21 10:33:53,141 - MainThread - botocore.hooks - DEBUG - Event calling-command.kms.decrypt: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7faf383a8470>>
2019-10-21 10:33:53,141 - MainThread - botocore.hooks - DEBUG - Event calling-command.kms.decrypt: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7faf383a8358>>
2019-10-21 10:33:53,141 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2019-10-21 10:33:53,142 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2019-10-21 10:33:53,143 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/endpoints.json
2019-10-21 10:33:53,179 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7faf39a131e0>
2019-10-21 10:33:53,182 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.kms: calling handler <function add_generate_presigned_url at 0x7faf39a4d730>
2019-10-21 10:33:53,182 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2019-10-21 10:33:53,189 - MainThread - botocore.endpoint - DEBUG - Setting kms timeout as (60, 60)
2019-10-21 10:33:53,190 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: kms
2019-10-21 10:33:53,191 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.kms.Decrypt: calling handler <function generate_idempotent_uuid at 0x7faf39a309d8>
2019-10-21 10:33:53,192 - MainThread - botocore.hooks - DEBUG - Event before-call.kms.Decrypt: calling handler <function inject_api_version_header_if_needed at 0x7faf39a34510>
2019-10-21 10:33:53,192 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=Decrypt) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'TrentService.Decrypt', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'aws-cli/1.16.138 Python/3.6.8 Linux/5.0.0-27-generic botocore/1.12.128'}, 'body': b'{"CiphertextBlob": "UzJGeWJqcGhkM002YTIxek9tVjFMWGRsYzNRdE1qb3hNVEV4TWpJeU1qTXpNek02YTJWNUwySmpORE0yTkRnMUxUVXdPVEl0TkRKaU9DMDVNbUV6TFRCaFlUaGlPVE0xTXpaa1l3QUFBQUJSMW53YzNEenYxOCtrd3pXTmsxM2paMmE4eXBMQ3dqeWpnRTlkMVU2TkdHK3UvRkE9"}', 'url': 'http://localhost:4599/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7faf3823c0f0>, 'has_streaming_input': False, 'auth_type': None}}
2019-10-21 10:33:53,192 - MainThread - botocore.hooks - DEBUG - Event request-created.kms.Decrypt: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7faf3823c080>>
2019-10-21 10:33:53,192 - MainThread - botocore.hooks - DEBUG - Event choose-signer.kms.Decrypt: calling handler <function set_operation_specific_signer at 0x7faf39a308c8>
2019-10-21 10:33:53,193 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2019-10-21 10:33:53,193 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:localhost:4599
x-amz-date:20191020T233353Z
x-amz-target:TrentService.Decrypt

content-type;host;x-amz-date;x-amz-target
d5cd0005c7969d309641019287970ca2764240e93900d4bd7b47c1283e762ac9
2019-10-21 10:33:53,194 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20191020T233353Z
20191020/us-east-1/kms/aws4_request
2a6c11ed1f907ebff2b6067a1bfe55eaea7c93de318024caeb7aa74a8ceacb62
2019-10-21 10:33:53,194 - MainThread - botocore.auth - DEBUG - Signature:
aa3be5eff24b6adc08e04d077bf915b32a20b6982d370bf5f96d4847ed026a7a
2019-10-21 10:33:53,194 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=http://localhost:4599/, headers={'X-Amz-Target': b'TrentService.Decrypt', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'aws-cli/1.16.138 Python/3.6.8 Linux/5.0.0-27-generic botocore/1.12.128', 'X-Amz-Date': b'20191020T233353Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=_not_needed_locally_/20191020/us-east-1/kms/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=aa3be5eff24b6adc08e04d077bf915b32a20b6982d370bf5f96d4847ed026a7a', 'Content-Length': '230'}>
2019-10-21 10:33:53,195 - MainThread - urllib3.util.retry - DEBUG - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2019-10-21 10:33:53,196 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): localhost
2019-10-21 10:33:53,220 - MainThread - urllib3.connectionpool - DEBUG - http://localhost:4599 "POST / HTTP/1.1" 400 179
2019-10-21 10:33:53,220 - MainThread - botocore.parsers - DEBUG - Response headers: {'Content-Type': 'application/x-amz-json-1.1', 'Date': 'Sun, 20 Oct 2019 23:33:53 GMT', 'Content-Length': '179'}
2019-10-21 10:33:53,220 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"Message":"The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.","__type":"AccessDeniedException"}'
2019-10-21 10:33:53,221 - MainThread - botocore.hooks - DEBUG - Event needs-retry.kms.Decrypt: calling handler <botocore.retryhandler.RetryHandler object at 0x7faf38376cf8>
2019-10-21 10:33:53,221 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2019-10-21 10:33:53,223 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/awscli/clidriver.py", line 207, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "/usr/local/lib/python3.6/dist-packages/awscli/clidriver.py", line 348, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "/usr/local/lib/python3.6/dist-packages/awscli/clidriver.py", line 520, in __call__
    call_parameters, parsed_globals)
  File "/usr/local/lib/python3.6/dist-packages/awscli/clidriver.py", line 640, in invoke
    client, operation_name, parameters, parsed_globals)
  File "/usr/local/lib/python3.6/dist-packages/awscli/clidriver.py", line 652, in _make_client_call
    **parameters)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
2019-10-21 10:33:53,225 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

An error occurred (AccessDeniedException) when calling the Decrypt operation: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
ozbillwang commented 4 years ago

If I used the CiphertextBlob which was generated by awslocal kms encrypt, and use curl/httpie to decrypt it, I can get result, but the result is wrong.

$ http --json POST http://localhost:4599/ X-Amz-Target:TrentService.Decrypt CiphertextBlob="S2Fybjphd3M6a21zOmV1LXdlc3QtMjoxMTExMjIyMjMzMzM6a2V5Lzk1MGI1MGQ0LTdiMjAtNDA4MS04ZjczLTAwN2NlNDQ2MjA0OQAAAAB4i9BkwYaqblQtEkP5IAiT3SqnQDaOIfnCUZl69FFIrab2rvw="
HTTP/1.1 200 OK
Content-Length: 114
Content-Type: application/x-amz-json-1.1
Date: Mon, 21 Oct 2019 00:21:57 GMT

{
    "KeyId": "arn:aws:kms:eu-west-2:111122223333:key/950b50d4-7b20-4081-8f73-007ce4462049",
    "Plaintext": "U0dWc2JHOD0="      # < == Wrong plain text
}  

But if I decode the plaintext, it is the original plaintext:

$ echo  "U0dWc2JHOD0="  |base64 -d
SGVsbG8=

Seems some steps are not matched between http/curl and awslocal

@nsmithuk I gave up. Let me know if you can make it work with awslocal, I will be appreciated.

nsmithuk commented 4 years ago

Hi @ozbillwang

As awslocal is just a wrapper for the aws command, with some defaults set, you can just use the main aws-cli documentation.

The command is confusing for KMS as it expects non-base64 encoded inputs, but always outputs data base64 encoded.

Thus the steps you'd need for localkms are: Create a key

awslocal kms create-key
{
    "KeyMetadata": {
        "AWSAccountId": "111122223333",
        "KeyId": "0579fe9c-129b-490a-adb0-42589ac4a017",
        "Arn": "arn:aws:kms:eu-west-2:111122223333:key/0579fe9c-129b-490a-adb0-42589ac4a017",
        "CreationDate": 1571652484,
        "Enabled": true,
        "Description": "",
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyState": "Enabled",
        "Origin": "AWS_KMS",
        "KeyManager": "CUSTOMER"
    }
}

Encrypt some plaintext using the key. aws[local] kms encrypt does the base64 encoding of the plaintext for us, thus we don't pre-encode it. The decrypt commend expects its input to be binary, thus we base64 decode CiphertextBlob here, and put the resulting binary in a temporary file.

awslocal kms encrypt --key-id "0579fe9c-129b-490a-adb0-42589ac4a017" \
--plaintext "My Test String" \
--output text --query CiphertextBlob | base64 --decode > ExampleEncryptedFile

Decrypt the data, passing the ciphertext from the file above. KMS responds with the string encoded, so we need to base64 decode it.

awslocal kms decrypt --ciphertext-blob fileb://ExampleEncryptedFile \
--output text \
--query Plaintext | base64 --decode
My Test String

I've mostly just lifted the last 2 commands above from the Example sections of the following AWS docs, tweaked for localaws:

https://docs.aws.amazon.com/cli/latest/reference/kms/encrypt.html https://docs.aws.amazon.com/cli/latest/reference/kms/decrypt.html

nsmithuk commented 4 years ago

I think it's a good idea to have some examples about how to use local-kms from the cli; and to be honest for aws kms in general - there isn't too much written about it.

I might do a bit of a "How to use KMS directly" writeup, that'll apply to both local and AWS KMS.

ozbillwang commented 4 years ago

Thanks for the details.

I hate to run aws cli with --endpoint-url=http://localhost:4599 , awslocal is easier.

nsmithuk commented 4 years ago

Hi @ozbillwang

I've updated the README to reflect the examples in this thread.

I've also written up more in-depth guide to using KMS (local or real) from the CLI, here: https://nsmith.net/aws-kms-cli