Closed scottjbaldwin closed 1 year ago
Additional information: This works fine in ap-southeast-4 if using a standard user profile (i.e. long-term credentials, access-key/secret-access-key) for an IAM user, just not when you use the AssumeRole approach.
Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.
Whoops, accidentally closed the issue. Sorry...
Hi @scottjbaldwin,
Good afternoon.
Thanks for reporting the issue. Somehow I'm unable to reproduce the issue. Below are the steps I followed:
testassumerole
in my account with the below trust relationship:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<<account-id>>:root"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}
You may replace arn:aws:iam::<<account-id>>:root
above with the IAM user which you want to use for assume role.
AmazonS3FullAccess
, AWSCloudFormationFullAccess
and below custom policy attached (these permissions are necessary for deploying serverless application using .NET CLI extensions):
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"lambda:CreateFunction",
"iam:GetRole",
"lambda:TagResource",
"lambda:ListFunctions",
"apigateway:PUT",
"lambda:GetFunction",
"iam:CreateRole",
"iam:DeleteRole",
"iam:AttachRolePolicy",
"apigateway:DELETE",
"iam:PassRole",
"lambda:AddPermission",
"iam:DetachRolePolicy",
"apigateway:PATCH",
"lambda:DeleteFunction",
"apigateway:POST",
"apigateway:GET"
],
"Resource": "*"
}
]
}
%USERPROFILE%/.aws/config
, configured the new profile testassumerole
as below:
[profile testassumerole]
role_arn=arn:aws:iam::<<account-id>>:role/testassumerole
source_profile=default
In this case, my default
profile has credentials that gives permissions to assume the role.
ap-southeast-2
region (non-default).region
in aws-lambda-tools-defaults.json
to ap-southeast-2
.dotnet tool update -g Amazon.Lambda.Tools
.dotnet lambda deploy-serverless --profile testassumerole --region ap-southeast-2 --s3-bucket testassumerole-bucketname
. It deployed the application successfully:
Amazon Lambda Tools for .NET Core applications (5.6.3)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
Enter CloudFormation Stack Name: (CloudFormation stack name for an AWS Serverless application)
teststack
Processing CloudFormation resource AspNetCoreFunction
Initiate packaging of . for resource AspNetCoreFunction
Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder 'D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole.\bin\Release\net6.0\publish'
... dotnet publish "D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole." --output "D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole.\bin\Release\net6.0\publish" --configuration "Release" --framework "net6.0" /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 --self-contained False
... publish: MSBuild version 17.5.0-preview-23061-01+040e2a90e for .NET
... publish: Determining projects to restore...
... publish: All projects are up-to-date for restore.
... publish: TestServerlessAssumeRole -> D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\bin\Release\net6.0\linux-x64\TestServerlessAssumeRole.dll
... publish: TestServerlessAssumeRole -> D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\bin\Release\net6.0\publish\
Zipping publish folder D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole.\bin\Release\net6.0\publish to C:\Users\ashdhin\AppData\Local\Temp\AspNetCoreFunction-CodeUri-Or-ImageUri-638143917090263972.zip
... zipping: Amazon.Lambda.APIGatewayEvents.dll
... zipping: Amazon.Lambda.ApplicationLoadBalancerEvents.dll
... zipping: Amazon.Lambda.AspNetCoreServer.dll
... zipping: Amazon.Lambda.Core.dll
... zipping: Amazon.Lambda.Logging.AspNetCore.dll
... zipping: Amazon.Lambda.Serialization.SystemTextJson.dll
... zipping: appsettings.Development.json
... zipping: appsettings.json
... zipping: aws-lambda-tools-defaults.json
... zipping: TestServerlessAssumeRole
... zipping: TestServerlessAssumeRole.deps.json
... zipping: TestServerlessAssumeRole.dll
... zipping: TestServerlessAssumeRole.pdb
... zipping: TestServerlessAssumeRole.runtimeconfig.json
Created publish archive (C:\Users\<
Timestamp Logical Resource Id Status
3/14/2023 11:55 AM teststack CREATE_IN_PROGRESS 3/14/2023 11:55 AM AspNetCoreFunctionRole CREATE_IN_PROGRESS 3/14/2023 11:55 AM AspNetCoreFunctionRole CREATE_IN_PROGRESS 3/14/2023 11:55 AM AspNetCoreFunctionRole CREATE_IN_PROGRESS 3/14/2023 11:56 AM AspNetCoreFunctionRole CREATE_COMPLETE 3/14/2023 11:56 AM AspNetCoreFunction CREATE_IN_PROGRESS 3/14/2023 11:56 AM AspNetCoreFunction CREATE_IN_PROGRESS 3/14/2023 11:56 AM AspNetCoreFunction CREATE_COMPLETE 3/14/2023 11:56 AM ServerlessRestApi CREATE_IN_PROGRESS 3/14/2023 11:56 AM ServerlessRestApi CREATE_IN_PROGRESS 3/14/2023 11:56 AM ServerlessRestApi CREATE_COMPLETE 3/14/2023 11:56 AM AspNetCoreFunctionRootResourcePermissionProd CREATE_IN_PROGRESS 3/14/2023 11:56 AM ServerlessRestApiDeploymentcfb7a37fc3 CREATE_IN_PROGRESS 3/14/2023 11:56 AM AspNetCoreFunctionProxyResourcePermissionProd CREATE_IN_PROGRESS 3/14/2023 11:56 AM AspNetCoreFunctionRootResourcePermissionProd CREATE_IN_PROGRESS 3/14/2023 11:56 AM AspNetCoreFunctionProxyResourcePermissionProd CREATE_IN_PROGRESS 3/14/2023 11:56 AM ServerlessRestApiDeploymentcfb7a37fc3 CREATE_IN_PROGRESS 3/14/2023 11:56 AM ServerlessRestApiDeploymentcfb7a37fc3 CREATE_COMPLETE 3/14/2023 11:56 AM ServerlessRestApiProdStage CREATE_IN_PROGRESS 3/14/2023 11:56 AM ServerlessRestApiProdStage CREATE_IN_PROGRESS 3/14/2023 11:56 AM ServerlessRestApiProdStage CREATE_COMPLETE 3/14/2023 11:56 AM AspNetCoreFunctionRootResourcePermissionProd CREATE_COMPLETE 3/14/2023 11:56 AM AspNetCoreFunctionProxyResourcePermissionProd CREATE_COMPLETE 3/14/2023 11:56 AM teststack CREATE_COMPLETE Stack finished updating with status: CREATE_COMPLETE
Output Name Value
ApiURL https://<
Please review if you are following different steps or if I'm missing anything.
Thanks,
Ashish
@ashishdhingra, ap-southeast-2
is the Sydney region which is a default AWS region. As I said, it works fine in that region, but not in ap-southeast-4
(Melbourne) which is a non-default region.
@ashishdhingra,
ap-southeast-2
is the Sydney region which is a default AWS region. As I said, it works fine in that region, but not inap-southeast-4
(Melbourne) which is a non-default region.
@scottjbaldwin Could you please elaborate what are you referring to by default region? Is it the region in the credentials
/config
file? For me, the default region is us-east-2
.
Note, you will have to enable ap-southeast-4
for your AWS account if you haven't already, but that's how non-default regions work.
@ashishdhingra by non-default region, I mean a region that you don't get access to by default in an AWS account, i.e. an opt-in region as per this table https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions:~:text=for%20a%20resource-,Available%20Regions,-Your%20account%20determines
ing to S3. (Bucket: Key: ) Error uploading to in bucket : The provided token is malformed or otherwise invalid.
@scottjbaldwin Thanks for the inputs, it really helped. It appears to be reproducible for non-default (required to opt-in region). Executing dotnet lambda deploy-serverless --profile testassumerole --region ap-southeast-4 --s3-bucket testassumerole-s3-non-default-region
gives error below:
Amazon Lambda Tools for .NET Core applications (5.6.3)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet
Enter CloudFormation Stack Name: (CloudFormation stack name for an AWS Serverless application)
teststack
Warning: Unable to determine region for bucket testassumerole-s3-non-default-region, assuming bucket is in correct region: The provided token is malformed or otherwise invalid.
Processing CloudFormation resource AspNetCoreFunction
Initiate packaging of . for resource AspNetCoreFunction
Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder 'D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\.\bin\Release\net6.0\publish'
... dotnet publish "D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\." --output "D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\.\bin\Release\net6.0\publish" --configuration "Release" --framework "net6.0" /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 --self-contained False
... publish: MSBuild version 17.5.0-preview-23061-01+040e2a90e for .NET
... publish: Determining projects to restore...
... publish: Restored D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\TestServerlessAssumeRole.csproj (in 389 ms).
... publish: TestServerlessAssumeRole -> D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\bin\Release\net6.0\linux-x64\TestServerlessAssumeRole.dll
... publish: TestServerlessAssumeRole -> D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\bin\Release\net6.0\publish\
Zipping publish folder D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\.\bin\Release\net6.0\publish to C:\Users\<<redacted>>\AppData\Local\Temp\AspNetCoreFunction-CodeUri-Or-ImageUri-638144061995484474.zip
... zipping: Amazon.Lambda.APIGatewayEvents.dll
... zipping: Amazon.Lambda.ApplicationLoadBalancerEvents.dll
... zipping: Amazon.Lambda.AspNetCoreServer.dll
... zipping: Amazon.Lambda.Core.dll
... zipping: Amazon.Lambda.Logging.AspNetCore.dll
... zipping: Amazon.Lambda.Serialization.SystemTextJson.dll
... zipping: appsettings.Development.json
... zipping: appsettings.json
... zipping: aws-lambda-tools-defaults.json
... zipping: TestServerlessAssumeRole
... zipping: TestServerlessAssumeRole.deps.json
... zipping: TestServerlessAssumeRole.dll
... zipping: TestServerlessAssumeRole.pdb
... zipping: TestServerlessAssumeRole.runtimeconfig.json
Created publish archive (C:\Users\<<redacted>>\AppData\Local\Temp\AspNetCoreFunction-CodeUri-Or-ImageUri-638144061995484474.zip).
Lambda project successfully packaged: C:\Users\<<redacted>>\AppData\Local\Temp\AspNetCoreFunction-CodeUri-Or-ImageUri-638144061995484474.zip
Uploading to S3. (Bucket: testassumerole-s3-non-default-region Key: TestServerlessAssumeRole/AspNetCoreFunction-CodeUri-Or-ImageUri-638144061995484474-638144062032939311.zip)
Error uploading to TestServerlessAssumeRole/AspNetCoreFunction-CodeUri-Or-ImageUri-638144061995484474-638144062032939311.zip in bucket testassumerole-s3-non-default-region: The provided token is malformed or otherwise invalid.
Executing AWS CLI command aws s3api get-bucket-location --bucket testassumerole-s3-non-default-region- --profile testassumerole
returns correct region:
{
"LocationConstraint": "ap-southeast-4"
}
Also using the default
profile directly with configured credentials works fine for ap-southeast-4
region.
So the issue appears to be with the underlying .NET SDK that fails to successfully call AmazonS3Client.GetBucketLocationAsync()
for assume role flow and fails with The provided token is malformed or otherwise invalid.
error. This needs investigation and further review.
@scottjbaldwin Upon further investigation, Managing AWS STS in an AWS Region mentions that Session tokens from the global STS endpoint are valid only in AWS Regions that are enabled by default
. While per the mentioned article, you could change the setting in AWS account, the other workaround is to set environment variable AWS_STS_REGIONAL_ENDPOINTS
with value regional
in the current session. I tested this and it worked.
In PowerShell session:
$env:AWS_STS_REGIONAL_ENDPOINTS="regional"
In command line (Windows):
set AWS_STS_REGIONAL_ENDPOINTS=regional
Also refer AWS STS Regionalized endpoints where it states that AWS recommends using Regional AWS STS endpoints instead of the global endpoint. So, you may set it at config
file level instead.
Please let me know if it works for you. After your confirmation, I would consider this issue as resolved (since it is the default behavior as per above documentation) and convert this issue into Q&A
discussion so that it's easily accessible to other users experiencing similar issue.
Thanks, Ashish
Hey @ashishdhingra, thanks so much for diving into this issue and providing the resolution. I really appreciate your diligence on this one.
This does indeed fix the issue, and I am able to deploy my application (although there were still some remaining regional issues, but completely unrelated to this issue). I guess that means I'll need to update my blogpost, as technically it is not so much a bug, as a corner case that people need to be aware of.
Totally support this issue being converted into a Q&A
as I am sure I will not be the last person to burn considerable hours on this issue, and I would love to discuss possible paths forward that are not quite as opaque.
Again, thanks for your help.
Describe the bug
When trying to deploy a serverless application to a non-default region (i.e.
ap-southeast-4
) using a profile that uses a session token fromsts
, the cli is unable to upload the packaged lambda code to the S3 bucket.Expected Behavior
The lambda code in the zip file should upload and the serverless template should be deployed. This works perfectly fine if the region is a default region like
ap-southeast-2
.Current Behavior
The output of
dotnet lambda deploy-serveress
first hints an an issue by sayingand then after compiling and zipping the code, when it attempts to upload the zip file, it errors out with
And exits the deployment.
Reproduction Steps
serverless.AspNetCoreWebAPI --name mel-test --output .\mel-test --region ap-southeast-4
(or equivalent non-default region)Possible Solution
Looking through the source code for the aws extensions for dotnet, I couldn't see anything obvious. I did see that you use the C# sdk, so the issue may be with it's handling of profiles, but because I found the issue in this lbrary, I decided to raise the bug here.
I did check the results of
aws s3api get-bucket-location --bucket <mel-bucket> --profile <assumerole-profile>
and recieved the expected results:Which says to me that the profile and the bucket are set up fine, and given this is the same call that GetBucketRegionAsycn makes, I can only assume that it's some difference between the way the cli handles the profile vs how the C# sdk handles the profile.
Additional Information/Context
I am using MFA for my roles, and I'm using this powershell module AWSCredentialsManager to assist with the AssumeRole and MFA details. There was an issue with this library using non-default regions initially, but this issue was addressed in version 0.1.6 which is the version of the library I am currently using.
Targeted .NET platform
7.0.201
CLI extension version
Environment details (OS name and version, etc.)
Windows 10