Boto3 inside KCL: EC2 instance profile credentials do not work for boto3

[dbmikus]

I have an application that runs the KCL multilang daemon to run a pool of Python processes that consume from Kinesis. Inside the Python processes, I need to put some records out to Kinesis based on the records I consume. I want both the KCL and boto3 to authenticate with EC2 instance profile credentials. This does not appear to be working. If I use environment variables or set values in ~/.aws/credentials, it works, but when I use the DefaultAWSCredentialsProviderChain, the internal boto3 connection fails.

Here are some file snippets to show what I have set up.

The relevant parts from the KCL properties file:

executableName = /usr/lib/python2.7/site-packages/student_ability_service/kinesis_consumer/kinesis_consumer.py
streamName = sas-pre-prod-stream
applicationName = pre-prod-kcl-sas
AWSCredentialsProvider = DefaultAWSCredentialsProviderChain
regionName = us-east-1

The truncated Python file referenced by executableName property:

#!/usr/bin/env python

from __future__ import print_function

import sys
import time
import traceback

from amazon_kclpy import kcl
from amazon_kclpy.v2 import processor

import boto3
import logging
logger = logging.getLogger(__name__ + 'myapp')
hdlr = logging.FileHandler('/tmp/myapp.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr) 
logger.setLevel(logging.DEBUG)

import kcl_config

class RecordProcessor(processor.RecordProcessorBase):
    """
    A RecordProcessor processes data from a shard in a stream. Its methods will be called with this pattern:

    * initialize will be called once
    * process_records will be called zero or more times
    * shutdown will be called if this MultiLangDaemon instance loses the lease to this shard, or the shard ends due
        a scaling change.
    """
    def __init__(self):
        self._SLEEP_SECONDS = kcl_config.SLEEP_SECONDS
        self._CHECKPOINT_RETRIES = kcl_config.CHECKPOINT_RETRIES
        self._CHECKPOINT_FREQ_SECONDS = kcl_config.CHECKPOINT_FREQ_SECONDS
        self._largest_seq = (None, None)
        self._largest_sub_seq = None
        self._last_checkpoint_time = None
        try:
            boto3.client('kinesis', region_name='us-east-1').describe_stream(StreamName='sas-pre-prod-stream')
        except:
            logger.error('DBM first kincon attempt error')
            logger.error(traceback.format_exc())

...
...

Note how, during initialization, I attempt to establish a boto3 client and describe a Kinesis stream. This logs out an error:

2018-02-13 21:08:42,186 ERROR Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/student_ability_service/kinesis_consumer/kinesis_consumer.py", line 53, in __init__
    boto3.client('kinesis', region_name='us-east-1').describe_stream(StreamName='sas-pre-prod-stream')
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 599, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (UnrecognizedClientException) when calling the DescribeStream operation: The security token included in the request is invalid.

If I just go into python REPL:

$ python
Python 2.7.5 (default, Aug  4 2017, 00:39:18)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.client('kinesis', region_name='us-east-1').describe_stream(StreamName='sas-pre-prod-stream')
{'ResponseMetadata': ... ... u'StreamStatus': u'ACTIVE'}}

I truncated the response for brevity and privacy, but it's a valid response.

This leads me to believe that there is an issue establishing a boto3 connection with EC2 instance profile credentials inside a process created by a KCL app. I believe this is a bug. Please let me know if I am wrong or if there is a way around this while still using the EC2 instance profile credentials.

[jamesls]

Can you share a (sanitized) portion of your debug logs? Boto3 will log where it's pulling credentials from, so the first thing I'd want to double check is that boto3 is in fact pulling from the instance metadata service. It's possible that credentials are being set somewhere that are higher priority in the credential chain, especially because instance metadata is the last thing we check (http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials).

[dbmikus]

Hi @jamesls. I can post that in the next couple days. We're reprovisioning our dev environment tomorrow and it's down now. Will have logs before end of Friday.

[dbmikus]

Sorry for the delay @jamesls. Got the boto3 and botocore logs set up, and discovered that, due to an edge case in our environment configuration code, we were setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to the empty string instead of leaving them out of the environment variables completely.

So, it was a bug on my end. I'm closing this out. Should have simplified my testing scenario more before opening the issue. Sorry about that, and thanks for the help anyways.