Compatibility with localstack

[maranqz]

Is your feature request related to a problem? Please describe. If we use localstack for correct using AWS Explorer should change default endpoint.

Describe the solution you'd like Adding options to set endpoints for localstack aws services.

Example log

``` WARN - tProjectAccountSettingsManager - Failed to switch to profile 'Profile:localstack' software.amazon.awssdk.services.sts.model.StsException: The security token included in the request is invalid. (Service: Sts, Status Code: 403, Request ID: 5cb0bde5-daf8-4dbe-bb77-f887f4d18c25) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:189) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:121) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:147) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:101) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) at software.amazon.awssdk.services.sts.DefaultStsClient.getCallerIdentity(DefaultStsClient.java:905) at software.amazon.awssdk.services.sts.StsClient.getCallerIdentity(StsClient.java:1682) at software.aws.toolkits.jetbrains.services.sts.StsResources$ACCOUNT$1.invoke(StsResources.kt:13) at software.aws.toolkits.jetbrains.services.sts.StsResources$ACCOUNT$1.invoke(StsResources.kt:10) at software.aws.toolkits.jetbrains.core.ClientBackedCachedResource.fetch(AwsResourceCache.kt:182) at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache.fetch(AwsResourceCache.kt:341) at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache.fetchIfNeeded(AwsResourceCache.kt:327) at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache.access$fetchIfNeeded(AwsResourceCache.kt:218) at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache$getCachedResource$1$result$1.apply(AwsResourceCache.kt:256) at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache$getCachedResource$1$result$1.apply(AwsResourceCache.kt:218) at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908) at software.aws.toolkits.jetbrains.core.DefaultAwsResourceCache$getCachedResource$1.run(AwsResourceCache.kt:255) at com.intellij.util.RunnableCallable.call(RunnableCallable.java:20) at com.intellij.util.RunnableCallable.call(RunnableCallable.java:11) at com.intellij.openapi.application.impl.ApplicationImpl$1.call(ApplicationImpl.java:255) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) ```

[kiiadi]

How would you envision this working? Effectively this would be a new "region" to the toolkit; would it make sense to have some sort of capability to add a region? Would need to be able to understand the differences between global (e.g. IAM) and regionalized services etc.

[maranqz]

Did I right understand that you say about this.

I thought it would be a new profile field endpoint_url.

There is plugin for aws-cli which prepares command calling for selected profile. Also there is wrapper which adds endpoint-url in command.

[abrooksv]

The toolkit does not use the AWS CLI to make calls to AWS so the value of endpoint_url has no effect on the toolkit.

Allowing for a user to create a custom region may make sense as a feature though.

We currently will always make a call to sts::getCallerIdentity to validate the credentials are valid. We would need to figure out how to handle that in the world of custom of region.

[delitescere]

If it has to be hardcoded, I'd not suggest a new region, rather a new "partition", called localstack - but it'll still need to be configurable if the user has changed the port it's running on. (see https://github.com/aws/aws-sdk-java/blob/1.11.30/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json)

However, it would be far better if there was a configuration item in the IDE plugin that allowed the endpoint to be set (e.g. to http://localhost:4566).

The plugin would need to call the AWS SDK client setEndpoint, or the newer client builder withEndpointConfiguration.

[andreashe]

any news regarding this feature request?

[abrooksv]

Sadly no, we wrote a proposal to the Java SDK but it didn't align 100% with what they had in mind longer term in respect to region management.

We need to iterate on it and decide if the SDK supporting it is the way to go, or if we should build our own system on top.

[andreashe]

I understand. One of the issues with localstack is; there is no GUI out there. Seems only commandeer (which is expensive). Maybe a wrapper around aws cli is the only chance. Furtunately it returns JSON. I am thinking about a wrapper as well - but not sure yet, if it is worth it - also because localstack has fundamental bugs. And if so, an opensource lib could be a nice thing. I am more a Java guy, but I wonder if Python makes more sense here? Is an Intellj-Plugin forced to be a Java code?

[abrooksv]

Yes, IntelliJ plugins are written in a JVM language (we went with Kotlin).

Our support would not affect the CLI though to be clear, it would only affect the IDEs integrations such as the S3 bucket browser.

[andreashe]

Ok, but does that mean, if someone will provide a Java CLI wrapper, you would consider it integrating/using it (could be a jar then)?

[abrooksv]

A 3rd party middle layer may not integrate well. We leverage the Java SDK v2 for all calls to AWS to be clear, we do not leverage the AWS CLI.

The toolkit already has a region system, it is powered by the endpoints.json file today. We would have to extend that and build a UI on top of it in the Settings screen to create a custom "Partition"

https://github.com/aws/aws-toolkit-jetbrains/blob/master/core/src/software/aws/toolkits/core/region/ToolkitRegionProvider.kt https://github.com/aws/aws-toolkit-jetbrains/blob/master/jetbrains-core/src/software/aws/toolkits/jetbrains/core/region/AwsRegionProvider.kt

What is currently unclear is:

1. Do we allow adding custom endpoint data into the SDK and we pass that custom Region into the SDK and have them handle it? or
2. Do we move all the endpoint.json data processing up into the toolkit (probably need to expose the APIs from the SDK in this case as well) and then resolve the endpoint data and pass it to the SDK client builders.

[delitescere]

We definitely don’t want a different region. It is simply changing the endpoint URL. Some users may want to do that on a per-service basis if they want to use localstack for some services but real AWS for others (not uncommon).

The cli wrapper does this with the —endpoint-url argument.

[erizzo]

Same need for the Eclipse version of AWS Toolkit. I don't see a separate git repo for it to report, though.

[otaviosanches]

Any news ?

[zunsakai]

I come from 2023. Any news?

[vorant94]

it really doesn't sound much complicated... just to add option to change endpoint url... just like localstack themselves are doing with awslocal wrapper... am i missing something?

[cabeaulac]

+1 Can I do a +2023?

[Eptagone]

Custom endpoints are love❤️. Please add this feature.

[otaviosgoncalves]

+1

[oshamoiev]

+1

[bensullivan]

+1

[blueshack112]

A lot of our work is needlessly complicated because this feature is missing, we have to manage our resources in app.localstack.cloud when a simple endpoint URL configuration should fix this issue. AWS Cli officially supports the ability to define a custom endpoint URL for a profile, so does boto3 in python, I'm assuming whatever equivalent AWS API in Kotlin is, it probably supports custom endpoint URLs as well.

Setting the profile as below in ~/.aws/credentials file

[localstack_test]
aws_access_key_id = test
aws_secret_access_key = test
endpoint_url = https://localstack.traefik.url/
ca_bundle = ~\AppData\Local\.certifi\cacert.pem

And making the request aws --profile localstack_test s3api list-buckets works like a charm, it returns the data from localstack. AWS Toolkit already has access to this information, I don't suppose it will be too complicated to code it in?

[obondarenko1]

Last month we commemorated 4 years of this request. Any news for adding one parameter?

[danny-zegel-zocdoc]

Last month we commemorated 4 years of this request. Any news for adding one parameter?

ditto

[srgg]

2024, commercial customer for over a decade. Is there any progress on this? It's been a while. The community eventually pushed Amazon to add endpoint_url, but this issue has not been solved.

Are there any plans for fixing it?

[bertramn]

The Java SDK would simply have to support a standard aws profile configuration from ~/.aws/config like awscli does.

[profile localstack]
region = us-east-1
ignore_configure_endpoint_urls = true
endpoint_url = http://localhost.localstack.cloud:4566
aws_access_key_id = noop
aws_secret_access_key = noop

With OOTB aws-cli

$ aws --version
aws-cli/2.21.2 Python/3.12.7 Darwin/23.6.0 source/arm64

$ aws --profile localstack sts get-caller-identity
000000000000    arn:aws:iam::000000000000:root  AKIAIOSFODNN7EXAMPLE

Note localstack account id is 000000000000 which might fail to validate somewhere.