Instance Profile Credential authentication fails during upload to S3

[christophergrant]

On AWS, instance profiles are a supported authentication mechanism that are supposed to be part the default credential provider chain used here.

With the command

java  -jar /path/to/neptune-export.jar nesvc \
  --root-path /path/to/data \
  --json '{
            "command": "export-pg",
            "outputS3Path" : "s3://bucket/neptune-export",
            "params": {
              "endpoint" : "some-database.neptune.amazonaws.com"
            }
          }'

(numbered just for reference)

1. When using instance profile authentication as configured on the EC2 instance, we get a series of 403 Access Denied errors on each run.

2. This just works when providing AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN.

3. We pulled out the relevant AWS SDK authentication code uploadDirectory in pursuit of a root cause but successfully authenticated with instance profile credentials - the same credentials as (1), so are thinking that context is somehow being lost.

Seems similar to https://github.com/aws/neptune-export/issues/16 but the error is slightly different, 403 instead of Unable to load AWS credentials from any provider in the chain

[Cole-Greer]

Hi @christophergrant, I'm looking into this issue and agree that you should be able to authenticate via instance profiles.

Could you share a bit more detail about where the 403 error is coming from? Do you happen to have a stack trace? In what phase of the export job are you getting this error? Is it right at the start when it attempts to connect to Neptune or is it near the end when the task is trying to connect to S3?

One thing to note here is that EC2 instance profiles is the lowest priority credentials source in the default credential provider chain, so it is important that no other providers on that chain are producing the wrong credentials.

[christophergrant]

Thanks for the super prompt reply! Here's a stacktrace, I can send a non-redacted version if it helps.

redacted 19:25:57 INFO ExportToS3NeptuneExportEventHandler: Uploading export files to s3://bucket/prefix

redacted 19:25:57 ERROR ExportToS3NeptuneExportEventHandler: Upload to S3 failed: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: redacted; S3 Extended Request ID: redacted; Proxy: redacted)

We see intermediate outputs from Neptune on the local filesystem, so it seems like the errors are happening when trying to connect to S3.

[Cole-Greer]

Thanks @christophergrant, could you confirm that the role attached to your EC2 instance has the correct permissions to write to S3?

[christophergrant]

@Cole-Greer That instance profile has been used to successfully authn + authz to that same bucket with other systems e.g. Databricks %fs commands.

We also pulled out java SDK code and ran it separate from neptune-export and it worked (see point 3 in the original issue).

[Cole-Greer]

Thanks @christophergrant, so far I have been unable to reproduce this issue in my own account using your exact command with v1.0.6. Could you confirm which version of neptune-export you are using?

[christophergrant]

@Cole-Greer 1.0.6. Are there any separate configurations that we need to make?

[Cole-Greer]

@christophergrant No additional configuration should be needed here. Perhaps it's noteworthy that my tests here are against an S3 bucket with ACLs disabled and with S3 managed keys. Neptune Export support is not limited to that exact setup but I'm hoping to understand what is different about my setup which is preventing me from reproducing your issue.

[christophergrant]

Do you have any ideas on how we could debug this, maybe there is a way to get diagnostic information? I'm happy to modify some code, build the jar, and launch to test.

Maybe adding logging to com.amazonaws.auth?

This is probably an aside but may be helpful: when we took the profile off of the instance, the connection to neptune fails due to lack of credentials. When we put it back on, the neptune part works, but the s3 part fails. We have no IAM DB authentication on the target neptune instance, either.

[Cole-Greer]

There definitely seems to be some strange interaction going on here. Currently Neptune Export doesn't modify the credential source for the S3Client at all, it simply uses the DefaultCredentialsProviderChain. There is debug logging com.amazonaws.auth.AWSCredentialsProviderChain which may shed some light on which provider in the chain is being utilized here.

Additionally it may be worthwhile to bypass the provider chain and force the S3 client to use the instance credentials by modifying com.amazonaws.services.neptune.util.TransferManagerWrapper as such:

    public TransferManagerWrapper(String s3Region) {

        AmazonS3ClientBuilder amazonS3ClientBuilder = AmazonS3ClientBuilder.standard()
                .withCredentials(InstanceProfileCredentialsProvider.getInstance());

        if (StringUtils.isNotEmpty(s3Region)) {
            amazonS3ClientBuilder = amazonS3ClientBuilder.withRegion(s3Region);
        }

        transferManager = TransferManagerBuilder.standard()
                .withS3Client(amazonS3ClientBuilder.build())
                .build();
    }

[christophergrant]

We were missing s3 permissions for tagging, see prereqs here. Thanks @Cole-Greer for troubleshooting with us.

I will open a docs PR to add a reference to this in the repo readme.

[Cole-Greer]

Glad you got it all working. Thanks for updating the readme.

[christophergrant]

Closed by #55.