khorii-cover commented 10 months ago

This discussion has been locked to preserve visibility of the solution, see the final comment accordingly. Original issue description follows...

Describe the bug

I am experiencing a not found, ResolveEndpointV2 error. This error occurs during the process of marshaling a JSON object and passing it to the SQS SendMessage method.

b, err := json.Marshal(*data)
if err != nil {
  return err
}

input := &sqs.SendMessageInput{
    MessageBody: aws.String(string(b)),
    QueueUrl:    aws.String(url),
}

if _, err := n.client.SendMessage(ctx, input); err != nil {
    return err
}

Expected Behavior

The SendMessage method is expected to execute without errors.

Current Behavior

When executing the SendMessage method, the following error occurs:

not found, ResolveEndpointV2

Reproduction Steps

Create a JSON payload similar to the one shown below.
Use the AWS SDK for Go to create an SQS client.
Use the SendMessage method of the SQS client to send the JSON payload to a specified queue.
Observe the ResolveEndpointV2 error during the execution of the SendMessage call.

{
   "target_id": "example-id",
   "target_body": "Example content",
   "target_type": 0,
   "event_type": 0,
   "deep_link_url": "exampleapp:///threads/example-id",
   "image_url": "https://example.com/icon.png",
   "tokens": null
}

Possible Solution

No response

Additional Information/Context

The issue does not occur with previous versions of the SDK, and reverting to an earlier version resolves the issue.

github.com/aws/aws-sdk-go v1.47.11
github.com/aws/aws-sdk-go-v2 v1.22.2
github.com/aws/aws-sdk-go-v2/config v1.25.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.42.1
github.com/awsdocs/aws-doc-sdk-examples/gov2/s3 v0.0.0-20230216203034-1f666e8ece89

AWS Go SDK V2 Module Versions Used

module cpf-api-service

go 1.21

require (
    github.com/aws/aws-sdk-go v1.47.12
    github.com/aws/aws-sdk-go-v2 v1.23.0
    github.com/aws/aws-sdk-go-v2/config v1.25.1
    github.com/aws/aws-sdk-go-v2/service/s3 v1.42.2
    github.com/awsdocs/aws-doc-sdk-examples/gov2/s3 v0.0.0-20230216203034-1f666e8ece89
    github.com/coreos/go-oidc/v3 v3.7.0
    github.com/danielgtaylor/huma v1.14.2
    github.com/gavv/httpexpect/v2 v2.16.0
    github.com/go-redis/redis/v8 v8.11.5
    github.com/go-sql-driver/mysql v1.7.1
    github.com/google/uuid v1.4.0
    github.com/google/wire v0.5.0
    github.com/kelseyhightower/envconfig v1.4.0
    github.com/lestrrat-go/jwx v1.2.26
    github.com/mitchellh/mapstructure v1.5.0
    github.com/pkg/errors v0.9.1
    github.com/uptrace/bun v1.1.16
    github.com/uptrace/bun/dbfixture v1.1.16
    github.com/uptrace/bun/dialect/mysqldialect v1.1.16
    github.com/uptrace/bun/extra/bundebug v1.1.16
    github.com/urfave/cli/v2 v2.25.7
    go.uber.org/zap v1.26.0
    goa.design/goa/v3 v3.14.0
    goa.design/plugins/v3 v3.14.0
    golang.org/x/oauth2 v0.14.0
    google.golang.org/api v0.150.0
    gopkg.in/DataDog/dd-trace-go.v1 v1.57.0
)

require (
    cloud.google.com/go v0.110.8 // indirect
    cloud.google.com/go/compute v1.23.1 // indirect
    cloud.google.com/go/firestore v1.13.0 // indirect
    cloud.google.com/go/iam v1.1.3 // indirect
    cloud.google.com/go/storage v1.30.1 // indirect
    firebase.google.com/go/v4 v4.12.1
    github.com/MicahParks/keyfunc v1.9.0 // indirect
    golang.org/x/text v0.14.0
    gopkg.in/yaml.v2 v2.4.0
)

require (
    github.com/avast/retry-go v3.0.0+incompatible
    github.com/aws/smithy-go v1.17.0
    github.com/google/go-cmp v0.6.0
    github.com/hololive/cpf-push-notification-service/go v0.0.0-20230306103827-2389f7bbb115
    github.com/nicksnyder/go-i18n/v2 v2.2.2
    github.com/rivo/uniseg v0.4.4
    github.com/spf13/cobra v1.8.0
    github.com/stretchr/testify v1.8.4
    golang.org/x/exp v0.0.0-20230321023759-10a507213a29
)

require (
    cloud.google.com/go/compute/metadata v0.2.3 // indirect
    cloud.google.com/go/longrunning v0.5.2 // indirect
    github.com/AnatolyRugalev/goregen v0.1.0 // indirect
    github.com/DataDog/appsec-internal-go v1.0.0 // indirect
    github.com/DataDog/datadog-agent/pkg/obfuscate v0.48.0 // indirect
    github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.48.1 // indirect
    github.com/DataDog/datadog-go/v5 v5.3.0 // indirect
    github.com/DataDog/go-libddwaf v1.5.0 // indirect
    github.com/DataDog/go-tuf v1.0.2-0.5.2 // indirect
    github.com/DataDog/gostackparse v0.7.0 // indirect
    github.com/DataDog/sketches-go v1.4.2 // indirect
    github.com/Microsoft/go-winio v0.6.1 // indirect
    github.com/TylerBrock/colorjson v0.0.0-20200706003622-8a50f05110d2 // indirect
    github.com/ajg/form v1.5.1 // indirect
    github.com/andybalholm/brotli v1.0.6 // indirect
    github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.1 // indirect
    github.com/aws/aws-sdk-go-v2/credentials v1.16.1 // indirect
    github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.4 // indirect
    github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.38 // indirect
    github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.3 // indirect
    github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.3 // indirect
    github.com/aws/aws-sdk-go-v2/internal/ini v1.7.0 // indirect
    github.com/aws/aws-sdk-go-v2/internal/v4a v1.2.3 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.1 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.2.3 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.3 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.16.3 // indirect
    github.com/aws/aws-sdk-go-v2/service/sqs v1.24.4 // indirect
    github.com/aws/aws-sdk-go-v2/service/sso v1.17.2 // indirect
    github.com/aws/aws-sdk-go-v2/service/ssooidc v1.19.2 // indirect
    github.com/aws/aws-sdk-go-v2/service/sts v1.25.2 // indirect
    github.com/cespare/xxhash/v2 v2.2.0 // indirect
    github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
    github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
    github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
    github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
    github.com/dimfeld/httppath v0.0.0-20170720192232-ee938bf73598 // indirect
    github.com/dustin/go-humanize v1.0.1 // indirect
    github.com/ebitengine/purego v0.5.0-alpha.1 // indirect
    github.com/fatih/color v1.15.0 // indirect
    github.com/fatih/structs v1.1.0 // indirect
    github.com/go-chi/chi/v5 v5.0.10 // indirect
    github.com/go-jose/go-jose/v3 v3.0.0 // indirect
    github.com/gobwas/glob v0.2.3 // indirect
    github.com/goccy/go-json v0.10.2 // indirect
    github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
    github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
    github.com/golang/protobuf v1.5.3 // indirect
    github.com/google/go-querystring v1.1.0 // indirect
    github.com/google/pprof v0.0.0-20230817174616-7a8ec2ada47b // indirect
    github.com/google/s2a-go v0.1.7 // indirect
    github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
    github.com/googleapis/gax-go/v2 v2.12.0 // indirect
    github.com/gorilla/websocket v1.5.0 // indirect
    github.com/imkira/go-interpol v1.1.0 // indirect
    github.com/inconshreveable/mousetrap v1.1.0 // indirect
    github.com/jinzhu/inflection v1.0.0 // indirect
    github.com/jmespath/go-jmespath v0.4.0 // indirect
    github.com/klauspost/compress v1.17.1 // indirect
    github.com/lestrrat-go/backoff/v2 v2.0.8 // indirect
    github.com/lestrrat-go/blackmagic v1.0.1 // indirect
    github.com/lestrrat-go/httpcc v1.0.1 // indirect
    github.com/lestrrat-go/iter v1.0.2 // indirect
    github.com/lestrrat-go/option v1.0.1 // indirect
    github.com/manveru/faker v0.0.0-20171103152722-9fbc68a78c4d // indirect
    github.com/mattn/go-colorable v0.1.13 // indirect
    github.com/mattn/go-isatty v0.0.20 // indirect
    github.com/mitchellh/go-wordwrap v1.0.1 // indirect
    github.com/outcaste-io/ristretto v0.2.3 // indirect
    github.com/philhofer/fwd v1.1.2 // indirect
    github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
    github.com/richardartoul/molecule v1.0.1-0.20221107223329-32cfee06a052 // indirect
    github.com/russross/blackfriday/v2 v2.1.0 // indirect
    github.com/sanity-io/litter v1.5.5 // indirect
    github.com/secure-systems-lab/go-securesystemslib v0.7.0 // indirect
    github.com/sergi/go-diff v1.3.1 // indirect
    github.com/spaolacci/murmur3 v1.1.0 // indirect
    github.com/spf13/pflag v1.0.5 // indirect
    github.com/tinylib/msgp v1.1.8 // indirect
    github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc // indirect
    github.com/valyala/bytebufferpool v1.0.0 // indirect
    github.com/valyala/fasthttp v1.50.0 // indirect
    github.com/vmihailenco/msgpack/v5 v5.3.5 // indirect
    github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
    github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
    github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
    github.com/xeipuuv/gojsonschema v1.2.0 // indirect
    github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
    github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0 // indirect
    github.com/yudai/gojsondiff v1.0.0 // indirect
    github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82 // indirect
    go.opencensus.io v0.24.0 // indirect
    go.uber.org/atomic v1.11.0 // indirect
    go.uber.org/multierr v1.10.0 // indirect
    go4.org/intern v0.0.0-20230525184215-6c62f75575cb // indirect
    go4.org/unsafe/assume-no-moving-gc v0.0.0-20230525183740-e7c30c78aeb2 // indirect
    golang.org/x/crypto v0.15.0 // indirect
    golang.org/x/mod v0.13.0 // indirect
    golang.org/x/net v0.18.0 // indirect
    golang.org/x/sync v0.5.0 // indirect
    golang.org/x/sys v0.14.0 // indirect
    golang.org/x/time v0.3.0 // indirect
    golang.org/x/tools v0.14.0 // indirect
    golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
    google.golang.org/appengine v1.6.8 // indirect
    google.golang.org/appengine/v2 v2.0.2 // indirect
    google.golang.org/genproto v0.0.0-20231016165738-49dd2c1f3d0b // indirect
    google.golang.org/genproto/googleapis/api v0.0.0-20231016165738-49dd2c1f3d0b // indirect
    google.golang.org/genproto/googleapis/rpc v0.0.0-20231030173426-d783a09b4405 // indirect
    google.golang.org/grpc v1.59.0 // indirect
    google.golang.org/protobuf v1.31.0 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
    inet.af/netaddr v0.0.0-20230525184311-b8eac61e914a // indirect
    moul.io/http2curl/v2 v2.3.0 // indirect
)

Compiler and Version used

go1.21.4 linux/arm64

Operating System and version

Debian GNU/Linux 12 (bookworm)

KaeDig commented 10 months ago

We've also noticed this bug in many of our modules in our dev environments since the sdk update, thank you for raising this issue.

Edit: And it's not just affecting sqs. Many services seem to be returning this error, in particular on list/get operations which we've encountered in our integration tests. Our modules which worked fine before are failing for example when testing if specific Codebuild projects exist using codebuild.BatchGetProjects, returning the "ResolveEndpointV2" error.

khorii-cover commented 10 months ago

I just read the Release Notes. I think this breaking change is likely the cause.

Feature: BREAKING CHANGE: V2 endpoint resolution middleware has changed steps from Serialize to Finalize. Middleware that indexes off of this field will need to be updated accordingly.

khorii-cover commented 10 months ago

I think this line might possibly be where the error is occurring.

https://github.com/aws/smithy-go/blob/main/middleware/ordered_group.go#L217

khorii-cover commented 10 months ago

It seems like an error is occurring because ResolveEndpointV2 is not in the array, but it has been successfuly added at this point.

https://github.com/aws/aws-sdk-go-v2/blob/main/service/sqs/api_client.go#L139

DeeptimanQlik commented 10 months ago

We are also facing same issue while initialising the aws.NewCredentialsCache(creds) and trying to retrieve the creds from the cache.

role := "arn:aws:iam::1222233333:role/sample-assume-role"
creds := stscreds.NewAssumeRoleProvider(stsSvc, role, func(aro *stscreds.AssumeRoleOptions) {
        aro.RoleSessionName = "RoleSession"
        aro.RoleARN =  role
    })

credCache := aws.NewCredentialsCache(creds)
credentials, err := credCache.Retrieve(ctx)
if err != nil {
   // getting error as "failed to refresh cached credentials, not found, ResolveEndpointV2"

    return aws.Credentials{}, err 
}

khorii-cover commented 10 months ago

I think this line might possibly be where the error is occurring.

https://github.com/aws/smithy-go/blob/main/middleware/ordered_group.go#L217

My earlier statement may have been incorrect. I am looking at this section of the code now.

https://github.com/aws/smithy-go/blob/a2a3b0b7412a5b13cbadeae4f7349e3d1a4c37c2/middleware/ordered_group.go#L191

khorii-cover commented 10 months ago

We are also facing same issue while initialising the aws.NewCredentialsCache(creds) and trying to retrieve the creds from the cache.

role := "arn:aws:iam::1222233333:role/sample-assume-role"
creds := stscreds.NewAssumeRoleProvider(stsSvc, role, func(aro *stscreds.AssumeRoleOptions) {
      aro.RoleSessionName = "RoleSession"
      aro.RoleARN =  role
  })

credCache := aws.NewCredentialsCache(creds)
credentials, err := credCache.Retrieve(ctx)
if err != nil {
   // getting error as "failed to refresh cached credentials, not found, ResolveEndpointV2"

    return aws.Credentials{}, err 
}

I think it's because the same addProtocolFinalizerMiddlewares function is being called internally.

https://github.com/aws/aws-sdk-go-v2/blob/435199fc01ab47020ab36dab07d8115e20687f73/service/sts/api_op_AssumeRole.go#L87

lucix-aws commented 10 months ago

As you've called out from the release notes, we moved endpoint resolution from the Serialize step to Finalize. Per the notes, if you have calling code that was previously indexing off of that middleware (e.g. stack.Serialize.Insert(..., "ResolveEndpointV2", middleware.Before) it will have to be adjusted accordingly.

That said, I can't reproduce this, at least not minimally. This snippet runs without hitting the middleware insert error:

func main() {
    cfg, _ := config.LoadDefaultConfig(context.Background())
    svc := sts.NewFromConfig(cfg, func(o *sts.Options) {
        o.Region = "us-east-1"
    })
    if _, err := svc.AssumeRole(context.Background(), &sts.AssumeRoleInput{
        RoleArn:         aws.String("arn:aws:iam::1222233333:role/sample-assume-role"),
        RoleSessionName: aws.String("bar"),
    }); err != nil {
        panic(err)
    }
}

What I'm not seeing in either of your examples is how your client is constructed. Are you doing anything to modify APIOptions on the client? (or perhaps using some 1st or 3rd-party API that does so) Would you be able to provide the snippet for how your service clients are instantiated?

That aside I'm currently checking to see if we've missed anything that needed updated in our own APIs. The default path appears to be sound (our integration tests are all passing, this wouldn't have even been able to release if they hadn't), my current guess is we have some helper API somewhere that's still expecting ResolveEndpointV2 to be in Serialize.

DeeptimanQlik commented 10 months ago

As you've called out from the release notes, we moved endpoint resolution from the Serialize step to Finalize. Per the notes, if you have calling code that was previously indexing off of that middleware (e.g. stack.Serialize.Insert(..., "ResolveEndpointV2", middleware.Before) it will have to be adjusted accordingly.

That said, I can't reproduce this, at least not minimally. This snippet runs without hitting the middleware insert error:
func main() {
    cfg, _ := config.LoadDefaultConfig(context.Background())
    svc := sts.NewFromConfig(cfg, func(o *sts.Options) {
        o.Region = "us-east-1"
    })
    if _, err := svc.AssumeRole(context.Background(), &sts.AssumeRoleInput{
        RoleArn:         aws.String("arn:aws:iam::1222233333:role/sample-assume-role"),
        RoleSessionName: aws.String("bar"),
    }); err != nil {
        panic(err)
    }
}
What I'm not seeing in either of your examples is how your client is constructed. Are you doing anything to modify APIOptions on the client? (or perhaps using some 1st or 3rd-party API that does so) Would you be able to provide the snippet for how your service clients are instantiated?

That aside I'm currently checking to see if we've missed anything that needed updated in our own APIs. The default path appears to be sound (our integration tests are all passing, this wouldn't have even been able to release if they hadn't), my current guess is we have some helper API somewhere that's still expecting ResolveEndpointV2 to be in Serialize.

@lucix-aws The AWS client config is initialised with region & request retryer.

please check here full code-snippet

package main

import (
    "context"
    "fmt"
    "time"
    "io"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/aws/retry"
    awsCfg "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/kms"
    "github.com/aws/aws-sdk-go-v2/credentials/stscreds"
    "github.com/aws/aws-sdk-go-v2/service/kms/types"
    "github.com/aws/aws-sdk-go-v2/service/sts"
)

func main() {

    ctx := context.Background()
    region := "us-east-1"

    kmsCfg, err := awsCfg.LoadDefaultConfig(ctx,
        awsCfg.WithRetryer(func() aws.Retryer {
            return retry.AddWithMaxAttempts(retry.NewStandard(), 10)
        }),

        awsCfg.WithRetryer(func() aws.Retryer {
            return retry.AddWithMaxBackoffDelay(retry.NewStandard(), time.Duration(10))
        }),
        awsCfg.WithRegion(region),
    )
    if err != nil {
        panic(err)
    }

    stsSvc := sts.NewFromConfig(kmsCfg)
    role := "arn:aws:iam::1222233333:role/sample-assume-role"
    creds := stscreds.NewAssumeRoleProvider(stsSvc, role, func(aro *stscreds.AssumeRoleOptions) {
        aro.RoleSessionName = "RoleSession"
        aro.RoleARN = role
    })
    credCache := aws.NewCredentialsCache(creds)
    kmsCfg.Credentials = credCache

    credentials, err := credCache.Retrieve(ctx)
    if err != nil {
        panic(err)
    }

    fmt.Println(credentials.AccessKeyID)

    client := kms.NewFromConfig(kmsCfg)
    res, err := client.GenerateDataKey(ctx, &kms.GenerateDataKeyInput{
        KeyId:             aws.String("arn:aws:kms:us-east-1:1122222333:key/21231232132132bf3232443242ae"),
        KeySpec:           types.DataKeySpecAes256,
        EncryptionContext: map[string]string{"fake-user": "fake-value"},
    }, func(o *kms.Options) {
        o.RetryMaxAttempts = 5
        o.EndpointOptions.Logger = o.Logger
    })
    if err != nil && err != io.EOF {
        panic(err)
    }
    fmt.Println(&res.KeyId)
}

ucirello commented 10 months ago

@lucix-aws

func defaultConfig(_ *testing.T) aws.Config {
    return aws.Config{
        Region: "us-west-2",
        EndpointResolverWithOptions: aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
            return aws.Endpoint{URL: "http://localhost:8000/"}, nil
        }),
        Credentials: credentials.StaticCredentialsProvider{
            Value: aws.Credentials{
                AccessKeyID:     "fakeMyKeyId",
                SecretAccessKey: "fakeSecretAccessKey",
            },
        },
    }
}
...
svc := dynamodb.NewFromConfig(defaultConfig(t))
...
_, err := c.dynamoDB.PutItem(ctx, putItemRequest)
    if err != nil {
        return nil, parseDynamoDBError(err, "cannot store lock item: lock already acquired by other client")
    }

fails for me. You can refer to source and tests failing at: https://github.com/cirello-io/dynamolock/pull/303

lucix-aws commented 10 months ago

To @ucirello , @DeeptimanQlik , @khorii-cover , @KaeDig and anyone else facing this issue--

Please upgrade ALL of your dependencies under the github.com/aws/aws-sdk-go-v2 namespace to the latest tags from yesterday's release, that should resolve the issue. Please chime in with the results if you can.

It appears that due to the breaking changes around middleware reordering, service client modules tagged on or after yesterday are now fundamentally incompatible with runtime modules tagged before yesterday (and vice versa).

For example, in the case of @ucirello's package:

This combination of versions is broken:

    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday, expects ResolveEndpointV2 to be in Finalize
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.1 // released before yesterday, ResolveEndpointV2 is in serialize
)

This combination is valid (I have verified that your tests under v2/ pass on my system after upgrading):

    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.2 // released yesterday, ResolveEndpointV2 is now in Finalize
)

More detailed explanation to follow.

phuongdnguyen commented 10 months ago

To @ucirello , @DeeptimanQlik , @khorii-cover , @KaeDig and anyone else facing this issue--

Please upgrade ALL of your dependencies under the github.com/aws/aws-sdk-go-v2 namespace to the latest tags from yesterday's release, that should resolve the issue. Please chime in with the results if you can.

It appears that due to the breaking changes around middleware reordering, service client modules tagged on or after yesterday are now fundamentally incompatible with runtime modules tagged before yesterday (and vice versa).

For example, in the case of @ucirello's package:

This combination of versions is broken:
    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday, expects ResolveEndpointV2 to be in Finalize
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.1 // released before yesterday, ResolveEndpointV2 is in serialize
)
This combination is valid (I have verified that your tests under v2/ pass on my system after upgrading):
    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.2 // released yesterday, ResolveEndpointV2 is now in Finalize
)
More detailed explanation to follow.

Thanks, it works for me with aws.NewCredentialsCache

koreyGambill commented 10 months ago

@lucix-aws this combination of dependencies is broken for me when using SSO

    github.com/aws/aws-sdk-go-v2 v1.23.0
    github.com/aws/aws-sdk-go-v2/config v1.25.1
    github.com/aws/aws-sdk-go-v2/service/ec2 v1.134.0

this code

    cfg, err := config.LoadDefaultConfig(
        context.TODO(),
        config.WithSharedConfigProfile(o.Profile),
    )
    if err != nil {
        return fmt.Errorf("error loading config: %w", err)
    }
    client := ec2.NewFromConfig(cfg)
    _, err = client.DescribeInstances(context.TODO(), &ec2.DescribeInstancesInput{})
    if err != nil {
        return fmt.Errorf("error describing instances: %w", err)
    }

gives this error

error describing instances: operation error EC2: DescribeInstances, get identity: get credentials: failed to refresh cached credentials, the SSO session has expired or is invalid: failed to read cached SSO token file, open ..<redacted>.. no such file or directory

lucix-aws commented 10 months ago

@koreyGambill does downgrading config to 1.23.0 resolve that issue?

More specifically, I see you're using an SSO session config, if you're using an sso_start_url in the config file with a fragment in the URL that's currently broken due to #2369 (a fix for that is merged, to be released today)

jkakar commented 10 months ago

I hit this issue and got past it by upgrading the github.com/aws/aws-sdk-go-v2 packages to the latest version.

DeeptimanQlik commented 10 months ago

@lucix-aws Upgrading all the pending chore updates into one branch solves the issues for me.

go get github.com/aws/aws-sdk-go-v2@v1.23.0
go get github.com/aws/aws-sdk-go-v2/service/kms@v1.26.2
go get github.com/aws/aws-sdk-go-v2/credentials@v1.16.1
go get github.com/aws/aws-sdk-go-v2/config@1.25.1

koreyGambill commented 10 months ago

@koreyGambill does downgrading config to 1.23.0 resolve that issue?

More specifically, I see you're using an SSO session config, if you're using an sso_start_url in the config file with a fragment in the URL that's currently broken due to #2369 (a fix for that is merged, to be released today)

I've tried 1.23.0 and 1.21.0 and neither worked while the other deps are upgraded.

lucix-aws commented 10 months ago

@koreyGambill That fix was released earlier today - please roll forward all of your dependencies and verify whether that issue is still present. If it is (as in, you're no longer seeing not found: ResolveEndpointV2 but are still seeing the SSO error) please open a separate issue with further details.

koreyGambill commented 10 months ago

@lucix-aws the new version of config seems to have fixed the error. Thanks!

go: upgraded github.com/aws/aws-sdk-go-v2/config v1.25.1 => v1.25.2

khorii-cover commented 10 months ago

@lucix-aws Sorry for the late reply. It seems that updating the SQS and config has resolved the issue. In our project, we are auto-updating several modules, but one of them was not well-managed, and the issue was caused by one of the modules being outdated.

ucirello commented 10 months ago

@lucix-aws -- after an all package upgrade, seems to be OK now.

lucix-aws commented 10 months ago

Glad to hear people are getting unblocked, we greatly appreciate everybody chiming into the thread. I'm going to leave this open at the top of the list for a while for visibility.

As for a post-mortem, this is clearly an unfortunate side-effect of the breaking change that went in on 11/15. In reality I suspect we have a host of undocumented compatibility breaks across our release history. Something as simple as an API added in one module to be consumed by another technically constitutes a break in this way, #2372 is an example of this. Even though our release tooling handles updating our own module dependencies, people are free to pin to their own versions and run into breaks like this.

The magnitude of this break is obviously much larger, but I can assert with confidence that changes on this level will be few and far between. The underlying change here was to reorder several key steps of the SDK operation lifecycle in order to align with the Smithy client reference architecture and enable advanced authentication behavior (as you may have guessed from the ubiquitous error message, endpoint resolution was one of those steps). With the way that the operation middleware API works (inserting steps into the operation can be keyed off of an existing slotted phase) it rendered anything pre-11/15 fundamentally broken against anything afterwards.

A true semantic versioning system would help mitigate this if not prevent it outright, but that's not something we have in place at this time. Moreover that generally doesn't align with our versioning strategy in the SDKs org where we tend to bake the "major version" into the repository name itself e.g. aws-sdk-go-v2, aws-sdk-java-v2. It's something we're considering moving forward, though, and this is obviously a strong supporting data point for that argument.

bitomaxsp commented 10 months ago

I have that failing with the same error when working with SFN:

        github.com/aws/aws-sdk-go-v2/service/sfn v1.21.0
    github.com/aws/aws-sdk-go-v2/config v1.25.4
    github.com/aws/aws-sdk-go-v2/service/s3 v1.42.0
    github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.23.3

github-actions[bot] commented 10 months ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

idanovo commented 9 months ago

I'm still getting this when using this these versions:

github.com/aws/aws-sdk-go-v2 v1.23.1
github.com/aws/aws-sdk-go-v2/config v1.25.5
github.com/aws/aws-sdk-go-v2/credentials v1.16.4
github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.12.3
github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression v1.6.3
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.14.3
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.3
github.com/aws/aws-sdk-go-v2/service/glue v1.63.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.45.0
github.com/aws/aws-sdk-go-v2/service/sts v1.25.4
github.com/aws/smithy-go v1.17.0

chrisclapham commented 9 months ago

I was encountering the not found, ResolveEndpointV2 error while trying to upload a file to S3. Updating the S3 dependency with go get github.com/aws/aws-sdk-go-v2/service/s3@latest resolved the issue.

kumackey commented 9 months ago

~I have used the following version and still get the same error.~

    github.com/aws/aws-sdk-go-v2 v1.23.5
    github.com/aws/aws-sdk-go-v2/config v1.25.11
    github.com/aws/aws-sdk-go-v2/service/kinesis v1.24.2

I checked carefully and found the following other error, so I was mistaken.

could not select an auth scheme

droberts-sea commented 9 months ago

Command to update all AWS SDK dependencies in one go:

$ go get -u "github.com/aws/aws-sdk-go-v2/..."

ArunNKutty commented 8 months ago

Was getting the same error for secrets. Did this and it worked. Basically, any AWS package which is breaking with the same error needs to update v2 and the respective service that is breaking

go get github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 go get github.com/aws/aws-sdk-go-v2/service/secretsmanager

OR run

go get -u "github.com/aws/aws-sdk-go-v2/..."

joonash commented 8 months ago

We're still getting the error with the most current versions of the SDK dependencies. At least the ECS DescribeServices call fails with "not found, ResolveEndpointV2".

versions: github.com/aws/aws-sdk-go-v2 v1.24.1 github.com/aws/aws-sdk-go-v2/config v1.26.3 github.com/aws/aws-sdk-go-v2/credentials v1.16.14 github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.30.2 github.com/aws/aws-sdk-go-v2/service/ecs v1.36.0 github.com/aws/aws-sdk-go-v2/service/kms v1.27.8 github.com/aws/aws-sdk-go-v2/service/s3 v1.47.8 github.com/aws/aws-sdk-go-v2/service/sqs v1.29.7 github.com/aws/aws-sdk-go-v2/service/sts v1.26.7 github.com/aws/smithy-go v1.19.0

renanqts commented 7 months ago

I also faced this issue in 1.24.1. I had to rollback to 1.23.0 to make it work

HemantDubey-ACS commented 7 months ago

I'm still facing this issue while reading the secret manager

    cfg, _ := config.LoadDefaultConfig(context.TODO(), config.WithRegion("ap-south-1"))
    svc := secretsmanager.NewFromConfig(cfg)
    secrets, err := svc.GetSecretValue(context.TODO(), &secretsmanager.GetSecretValueInput{
        SecretId:     &secretId,
        VersionStage: aws.String("AWSCURRENT"),
    })
    if err != nil {
        fmt.Println("SECRET_STRING ", err.Error())  
    }

Expected Behavior Retrieve the secret manager without errors as same is working on github.com/aws/aws-sdk-go-v2 v1.21.2

Current Behavior When executing the code, the following error occurs:

not found, ResolveEndpointV2

Compiler and Version used go1.21.1 darwin/arm64

Operating System and version ProductName: macOS ProductVersion: 14.2.1 BuildVersion: 23C71

AWS Go SDK V2 Module Versions Used

module github.com/AsthaBhopal/authentication

go 1.21

require (
    github.com/AsthaBhopal/pkgGoAsthaKycSchema v0.1.73
    github.com/AsthaBhopal/pkgGoAsthaLogs v0.9.0
    github.com/AsthaBhopal/pkgGoClevertapEvents v0.1.5
    github.com/AsthaBhopal/pkgGoFlowModels v1.0.63
    github.com/AsthaBhopal/pkgGoRedisAuthenticator v1.0.9
    github.com/aws/aws-sdk-go-v2 v1.25.0
    github.com/aws/aws-sdk-go-v2/config v1.18.22
    github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.10.39
    github.com/aws/aws-sdk-go-v2/service/apigatewaymanagementapi v1.12.5
    github.com/aws/aws-sdk-go-v2/service/cognitoidentityprovider v1.22.6
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.21.5
    github.com/aws/aws-sdk-go-v2/service/lambda v1.39.5
    github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.21.3
    github.com/aws/aws-sdk-go-v2/service/ses v1.16.10
    github.com/dgrijalva/jwt-go v3.2.0+incompatible
    github.com/gin-contrib/cors v1.4.0
    github.com/gin-contrib/pprof v1.4.0
    github.com/gin-gonic/gin v1.9.1
    github.com/gofrs/uuid v4.4.0+incompatible
    github.com/golang-jwt/jwt v3.2.2+incompatible
    github.com/joho/godotenv v1.5.1
    github.com/lib/pq v1.10.9
    github.com/nats-io/nats.go v1.32.0
    github.com/pquerna/otp v1.4.0
    github.com/yeqown/go-qrcode v1.5.10
    go.uber.org/zap v1.25.0
    golang.org/x/crypto v0.18.0
    gorm.io/driver/mysql v1.5.1
    gorm.io/driver/sqlserver v1.5.1
    gorm.io/gorm v1.25.4
    gorm.io/plugin/dbresolver v1.4.7
)

require (
    github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.13 // indirect
    github.com/aws/aws-sdk-go-v2/credentials v1.13.36 // indirect
    github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.11 // indirect
    github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.0 // indirect
    github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.0 // indirect
    github.com/aws/aws-sdk-go-v2/internal/ini v1.3.42 // indirect
    github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.25 // indirect
    github.com/aws/aws-sdk-go-v2/service/apigateway v1.18.0 // indirect
    github.com/aws/aws-sdk-go-v2/service/dynamodbstreams v1.15.5 // indirect
    github.com/aws/aws-sdk-go-v2/service/eventbridge v1.17.0 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.14 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.7.35 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.35 // indirect
    github.com/aws/aws-sdk-go-v2/service/sso v1.13.6 // indirect
    github.com/aws/aws-sdk-go-v2/service/ssooidc v1.15.5 // indirect
    github.com/aws/aws-sdk-go-v2/service/sts v1.21.5 // indirect
    github.com/aws/smithy-go v1.20.0 // indirect
    github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc // indirect
    github.com/bytedance/sonic v1.10.2 // indirect
    github.com/cespare/xxhash/v2 v2.1.2 // indirect
    github.com/chenzhuoyu/base64x v0.0.0-20230717121745-296ad89f973d // indirect
    github.com/chenzhuoyu/iasm v0.9.1 // indirect
    github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
    github.com/fogleman/gg v1.3.0 // indirect
    github.com/gabriel-vasile/mimetype v1.4.3 // indirect
    github.com/gin-contrib/sse v0.1.0 // indirect
    github.com/go-errors/errors v1.4.2 // indirect
    github.com/go-playground/locales v0.14.1 // indirect
    github.com/go-playground/universal-translator v0.18.1 // indirect
    github.com/go-playground/validator/v10 v10.16.0 // indirect
    github.com/go-redis/redis/v8 v8.11.5 // indirect
    github.com/go-sql-driver/mysql v1.7.0 // indirect
    github.com/goccy/go-json v0.10.2 // indirect
    github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
    github.com/golang-sql/sqlexp v0.1.0 // indirect
    github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
    github.com/google/uuid v1.3.0 // indirect
    github.com/jackc/pgpassfile v1.0.0 // indirect
    github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
    github.com/jackc/pgx/v5 v5.3.1 // indirect
    github.com/jinzhu/inflection v1.0.0 // indirect
    github.com/jinzhu/now v1.1.5 // indirect
    github.com/jmespath/go-jmespath v0.4.0 // indirect
    github.com/json-iterator/go v1.1.12 // indirect
    github.com/klauspost/compress v1.17.2 // indirect
    github.com/klauspost/cpuid/v2 v2.2.6 // indirect
    github.com/leodido/go-urn v1.2.4 // indirect
    github.com/mattn/go-isatty v0.0.20 // indirect
    github.com/microsoft/go-mssqldb v1.1.0 // indirect
    github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
    github.com/modern-go/reflect2 v1.0.2 // indirect
    github.com/nats-io/nkeys v0.4.7 // indirect
    github.com/nats-io/nuid v1.0.1 // indirect
    github.com/pelletier/go-toml/v2 v2.1.1 // indirect
    github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
    github.com/ugorji/go/codec v1.2.12 // indirect
    github.com/yeqown/reedsolomon v1.0.0 // indirect
    go.uber.org/multierr v1.10.0 // indirect
    golang.org/x/arch v0.6.0 // indirect
    golang.org/x/image v0.0.0-20200927104501-e162460cd6b5 // indirect
    golang.org/x/net v0.19.0 // indirect
    golang.org/x/sys v0.16.0 // indirect
    golang.org/x/text v0.14.0 // indirect
    google.golang.org/protobuf v1.32.0 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
    gorm.io/driver/postgres v1.5.2 // indirect
)

lucix-aws commented 7 months ago

All--

Since we're still getting comments on this issue, I think the diagnosis/solution has been somewhat buried.

To reiterate: this error occurs when using a version of the core aws-sdk-go-v2 module at or above v1.23.0 (essentially, released on or after 11/15/23) in combination with a service module released before that date. The most recent comment from @HemantDubey-ACS is an example of this - the service/secretsmanager v1.21.3 is from before that breaking point.

Upgrading all modules under the aws-sdk-go-v2 namespace will resolve this issue. You can do so using the following command (TYVM @droberts-sea):

go get -u github.com/aws/aws-sdk-go-v2/...

This was originally diagnosed here, and you can read a post-mortem on this issue in this comment.

To be absolutely clear, the bug users experienced here was NOT intentional, and CAN be avoided in the future independent of any middleware reordering we do. This incident has greatly informed how we should be writing/managing the code responsible for building the operation middleware stack. I've spawned #2507, the addressing of which should eliminate the possibility for breaks like this to occur in the future.

To guarantee the visibility of this message, I'm going to lock this thread. If you are still seeing this issue after updating all of your SDK modules, please open a new issue.

aws / aws-sdk-go-v2

[SOLUTION IN THREAD] "not found, ResolveEndpointV2" service modules before 11/15/23 are incompatible against newer root modules #2370

This discussion has been locked to preserve visibility of the solution, see the final comment accordingly. Original issue description follows...

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

AWS Go SDK V2 Module Versions Used

Compiler and Version used

Operating System and version

⚠️COMMENT VISIBILITY WARNING⚠️