The `aws_kinesis_firehose` source `store_access_key` option doesn't appear to actually store the access key

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

https://github.com/vectordotdev/vector/commit/2b446f7a434a74bd7d65b930797fc2f518540320 added an option to store the access key used for requests to the AWS Kinesis Firehose source, but this doesn't appear to be working correctly.

With the below config and data file, and running:

curl -i -XPOST -H X-Amz-Firehose-Access-Key: access1 -HX-Amz-Firehose-Request-Id: 123 -H X-Amz-Firehose-Source-Arn: hmm -H X-Amz-Firehose-Protocol-Version: 1.0 --data-binary @/tmp/data.json localhost:8080

The access key is reported as null. This appears to be due to the fact that the key isn't actually stored from the header. Instead it seems to be expecting to find it in the request body.

Configuration

sources:
    firehose:
      type: aws_kinesis_firehose
      address: 0.0.0.0:80
      access_keys: ["access1", "access2"]
      store_access_key: true
  transforms:
    parse_firehose:
      type: remap
      drop_on_abort: true
      drop_on_error: true
      inputs: ["firehose"]
      source: |
        .access_key = get_secret("aws_kinesis_firehose_access_key")
  sinks:
    console:
      type: console
      encoding:
        codec: native_json
      inputs:
        - parse_firehose
      target: stdout

Version

vector 0.31.0

Debug Output

{"log":{"access_key":null,"message":"{ \"requestId\": \"ed4acda5-034f-9f42-bba1-f29aea6d7d8f\", \"timestamp\": 1578090901599, \"records\": [ { \"data\": { \"messageType\": \"DATA_MESSAGE\", \"owner\": \"123456789012\", \"logGroup\": \"log_group_name\", \"logStream\": \"log_stream_name\", \"subscriptionFilters\": [ \"subscription_filter_name\" ], \"logEvents\": [ { \"id\": \"0123456789012345678901234567890123456789012345\", \"timestamp\": 1510109208016, \"message\": \"log message 1\" }, { \"id\": \"0123456789012345678901234567890123456789012345\", \"timestamp\": 1510109208017, \"message\": \"log message 2\" } ] } } ] }\n","request_id":"123","source_arn":"hmm","source_type":"aws_kinesis_firehose","timestamp":"2020-01-03T22:35:01.599Z"}}

Example Data

data.json:

{
  "requestId": "ed4acda5-034f-9f42-bba1-f29aea6d7d8f",
  "timestamp": 1578090901599,
  "records": [
    { "data": "H4sICHogxGQAA3JlY29yZC5qc29uALWQS0vEMBSF9/6KkHULSaavuCtYBxeuOjsZym2TlML0YZIqMvjfTdpaFBFXEhLO5eOec3OvCGv5PEtjHwS+RViKCBoBcUgOkQq5ilhY10BDxThISEQqMoUDhG3Xux7oJ9dE4zQj3B0acx54w2bUwjjyhK4IC7DgtFOuxUArT2+T9Fl3+SmvHouyzI+F9xxfB6k9oOwQxUmaOUfmwWVsj3qcfZbXVeuLaoBebrS0WkL/ic1S7dzMtWl0N9luHO67i5V6He0bqNRC1iZ0Xm2LFznY/R/dsiDydbg/1c9VUUIJZyQjNAn2jWyTo61EFKP34D8y098zmctEZ3eX9+YDF8L9xhkCAAA=" }
  ]
}

Additional Context

No response

References

https://github.com/vectordotdev/vector/commit/2b446f7a434a74bd7d65b930797fc2f518540320

vectordotdev / vector

The `aws_kinesis_firehose` source `store_access_key` option doesn't appear to actually store the access key #18108