SqsListener for SQS in different regions not working in 3.0.1

[jlustin]

Type: Bug

Component: "SQS"

Describe the bug I am currently in the process of migrating from 2.x to 3.x.

When using 2.x version, it was possible to override the SQS async client region in the @SqsListener annotation by specifying the full queue URL. Thus, it was possible for me to listen to multiple queues from different regions using the same SqsListener annotated method, and the regions from the URL would override the region specified when creating the SQS async client. Ref: https://stackoverflow.com/questions/56396694/multiple-sqs-listeners-from-different-aws-regions/58631642#58631642

However, after upgrading to version 3.x, it seems that the region in the client does not get overwritten anymore by the URL specified in the SqsListener. When the region configured in the SQS client is not matching, I am getting software.amazon.awssdk.services.sqs.model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version., which gets fixed only when the client's region matches the queue's. In cases where I am listening from one queue, there is no problem as the queue region can simply be specified in the client. Nonetheless, this error is still bound to happen in my use case where the SqsListener is listening from multiple queues from different AWS regions, and needing to send back acknowledgements manually.

It seems like there is no way for me to specify multiple regions for a SQS async client, that will be used by the same SqsListener that listens on multiple queues from different regions. As mentioned earlier, in 2.x, it seems like that it was possible to override the region in the queue URL in the SqsListener, but that no longer seems possible in 3.x.

Is there a bug or was this feature taken away? Are there any workarounds that you can propose?

Samples

// application.sqs.endpoint would be something like "https://sqs.us-east-1.amazonaws.com/000/queue-name,https://sqs.us-west-2.amazonaws.com/000/queue-name"
  @SqsListener("#{'${application.sqs.endpoint}'.split(',')}") 
  public void listen(final Message message, final Acknowledgement ack)
  {
    ...
    ack.acklowledge();
  }

As suggested in https://github.com/awspring/spring-cloud-aws/blob/v3.0.1/docs/src/main/asciidoc/sqs.adoc#sqs-integration, I am using SqsMessageListenerContainerFactory and SqsAsyncClient as Configuration. Any help would be appreciated.

Thanks!

[vmsds]

I am not sure if it solves your problem or if it is the best approach, but I think you can overwrite the factory in the @SqsListener(factory = "customFactory")

What I suggest you do is create two factory beans and override the queue region client.

Once done, you should create two SQS listeners and point each to a different factory.

[tomazfernandes]

Hey @jlustin, let me see if I understand correctly.

If you specify a queue URL with the region directly in the @SqsListener does it not work?

From the looks of it, this might be a change of behavior from AWS SDK v1 to v2, or an error in resolving the SpEL.

If you can please try to set the queue names directly to the annotation and see if that works that would help us finding that out.

If you can also add a breakpoint in QueueAttributesResolver#resolveQueueAttributes that you should be able to see the actual URLs it's trying to resolve.

You can also try @dottnul's suggestion.

Please let us know what you think. Thanks.

[jclavoie22]

Hey @jlustin, let me see if I understand correctly.

If you specify a queue URL with the region directly in the @SqsListener does it not work?

From the looks of it, this might be a change of behavior from AWS SDK v1 to v2, or an error in resolving the SpEL.

If you can please try to set the queue names directly to the annotation and see if that works that would help us finding that out.

If you can also add a breakpoint in QueueAttributesResolver#resolveQueueAttributes that you should be able to see the actual URLs it's trying to resolve.

You can also try @dottnul's suggestion.

Please let us know what you think. Thanks.

Hi @tomazfernandes . I am working with @jlustin. This is a changed of behavior from SDKv1 to v2. On V1, the request is made directly to the queue url rather than to the configured endpoint. See QueueUrlHandler#beforeRequest While on v2, the queue url is ignored & passed as queue name when the request is built.

[tomazfernandes]

Hi @jclavoie-jive, thanks, so it seems it's a restriction on the SDK side. I'm not sure I can be of much help then.

If you need to configure the region at SqsAsyncClient level, @dottnul's approach seems valid - just have a separate factory for each client and specify in the @SqsListener which factory to use.

Do you think that's worth giving a shot?

[jclavoie22]

Hi @jclavoie-jive, thanks, so it seems it's a restriction on the SDK side. I'm not sure I can be of much help then.

If you need to configure the region at SqsAsyncClient level, @dottnul's approach seems valid - just have a separate factory for each client and specify in the @SqsListener which factory to use.

Do you think that's worth giving a shot?

Yeah that's pretty much the only way unless we have a way to manually register consumers rather than using the @SqsListener annotation.

[tomazfernandes]

Well, you sure can manually create containers, if that's what you mean: https://docs.awspring.io/spring-cloud-aws/docs/3.0.1/reference/html/index.html#sqsmessagelistenercontainer

[tomazfernandes]

Hey @jclavoie-jive, it seems you already found a solution, so I'm closing this issue.

If there's anything else you'd like to discuss about this we can reopen it.

Thanks!