Unable to load AWS credentials from any provider in the chain

[aaronlangford31]

So I am working on setting up TitanDB in AWS. I am following the set up instructions as closely as I can (this tutorial and accompanying set up resources need improvement). I deployed my cloud resources using the CloudFormation stack creation template provided in this repository.

When I connect to the gremlin server via my local gremlin console, it appears that Java can't find the AWS credentials. The way I dug this up was to run:

:> g = TitanFactory.open("/usr/local/packages/dynamodb-titan100-storage-backend-1.0.0-hadoop1/conf/gremlin-server/dynamodb.properties")

The exception comes back as: org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException: Unable to load AWS credentials from any provider in the chain

If I am not mistaken, this can only happen if it is not found in Environment variables, Java system properties, The default credential profiles file, Amazon ECS container credentials, and Instance profile credentials fail to yield information on the credentials. (http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html)

This seems like unexpected behavior, where the CloudFormation template should have made sure that credentials would be available to Gremlin when started.

What is the expected behavior here? Do I need to include credentials in the dynamodb.properties file? Did I do something wrong to screw up the instance profile? How would I verify this kind of mistake?

[aaronlangford31]

The issue that I had was that the EC2 instance did not have permission to assume the IAM role that I specified in the CloudFormation template.

Others who have this issue can verify that this is the case by running the following commands on their EC2 instance:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<NAME OF IAM ROLE>

Where is the name of the role you created for the CloudFormation template. The response should look something like this if there is an issue:

{
  "Code" : "AssumeRoleUnauthorizedAccess",
  "Message" : "EC2 cannot assume the role Titan-Cloudformation-Builder.  Please see documentation at http://docs.amazonwebservices.com/IAM/latest/UserGuide/RolesTroubleshooting.html.",
  "LastUpdated" : "2017-05-27T00:54:26Z"
}

The way that I resolved this was to go to the IAM Mgmt Console, find the role, and adjust the Trust Policy such that the Statement array now included the following:

{
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
}

Had to wait 5 minutes before the effect took place. Verified that the change was successful by running the following once again:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<NAME OF IAM ROLE>

Which gave me some json that indicated that the role had now been assumed.

I then had to restart my gremlin server:

sudo service gremlin-server stop && sudo service gremlin-server start

Titan then began doing it's thing, I saw the tables show up in my DynamoDB console, and on to bigger and better things.

I'm not sure if this was the "AWS way", so I would love some feedback if any care to give it.

Credit to the following links for leading me to the solution:

• http://s3.thinkaurelius.com/docs/titan/1.0.0/configuration.html#_using_configuration
• http://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html
• https://github.com/awslabs/dynamodb-titan-storage-backend/issues/55

[amcp]

The documentation needs to improve on how the IAM role the instance profile assumes needs to be created. Regarding your specific issue, the reason the lack of STS AssumeRole permission is not caught is twofold:

1. AWS CloudFormation parameters do not support IAM roles as an AWS specific parameter type, as is the EC2 key pair name.
2. The user-data script does not check for the presence of an assumable role. Perhaps the user-data script should fail if STS cannot assume the specified role. I will investigate this check in a soon to come PR.

[amcp]

It appears that CloudFormation does not check that STS trust provider is ec2.amazonaws.com when creating the AWS::IAM::InstanceProfile resource. AFAIK InstanceProfiles are only useful in an EC2 context, so perhaps this check is useful. In my work so far, I found that creating a stack with a role with anything but a ec2.amazonaws.com identity provider causes the EC2 instance to be unreachable after startup. I think the best thing to be done in this case is to improve the documentation about the IAM roles you need to create. @aaronlangford31 I suggest you take the JanusGraph compatibility available on the master branch out for a spin. CloudFormation process is smoother, and incorporates support for recent versions of TinkerPop.