Open jlafaye opened 1 year ago
Hi @jlafaye - Thanks for opening the issue and apologies that things aren't working out for you. Just to clarify - you're SSH'ed into an EC2 instance and then also have a devcontainer/Docker environment running on that instance?
How does your devcontainer authenticate to AWS? In other words, if the IAM role is attached to the EC2 instance itself, how does the devcontainer make use of that role?
I haven't tried running this in a remote environment so bear with me. A couple things to try:
EMR: Select AWS Profile
from the command palette, are you provided with a list of profiles? And if so, are they from your local computer or the dev environment?EMR: Select AWS Region
command.If neither of those provide insight, I'll both try to set up a remote environment and add better error logging. At the moment, if you click on the OUTPUT
tab in VS Code, there is an "Amazon EMR" section, but the current logs are just status logs.
Hi @dacort - Thank you for taking the time to read my message.
My devcontainer authenticates to AWS through instance Metadata inherited from the instance the container is running on. I have set AWS_EC2_METADATA_DISABLED to false in devcontainer.json.
Sorry not being able to provide more info.
Just leaving some debugging notes:
aws s3 ls
worksdocker run --rm -it amazonlinux:2
, aws s3 ls
worksdocker run --rm -it --entrypoint /bin/bash 895885662937.dkr.ecr.us-west-2.amazonaws.com/spark/emr-6.10.0
, aws s3 ls
worksNext, need to try setting up ssh + devcontainer.
Create local Spark environment
command and None
for credential option is able to aws s3 ls
aws s3 ls
with "AWS_EC2_METADATA_DISABLED": "true",
aws s3 ls
with "AWS_EC2_METADATA_DISABLED": "false",
AWS_EC2_METADATA_DISABLED
is changed to false.~AWS_EC2_METADATA_DISABLED
is changed to false.For some reason looks like AwsCredentialIdentityProvider
in aws_context.ts
isn't finding the instance credentials.
Think I figured this out! 😮💨
Can you try removing AWS_EC2_METADATA_DISABLED
entirely from the containerEnv
section of your devcontainer.json
file?
When the SDK tries to retrieve credentials from IMDS, it checks for that environment variable, but uses the following code:
if (process.env[ENV_IMDS_DISABLED])
Unfortunately, environment variables come in as strings so even if we set it to false
or 0
, it evaluates to true.
Leaving this issue open as I'd like to add IMDS as an auth option when creating the container.
Hello,
Trying to use the toolkit with the following setup
What works
What does not work
It would be great if you could provide guidance on how to troubleshoot this. I would be happy to provide more details if needed. The toolkit is a great addition to VSCode and I'm sure it can ease the developer's lives.