Create SQS listeners dynamically

[anne-k]

The only way to set up a message listener as far as I can tell is through the \@SqsListener or \@MessageMapping annotations. But these annotations require the queue name to be a constant.

I am writing a little library to handle SQS queue logic. The library does not know any queue names and does not even know how many listeners there are going to be - that will be determined by the applications that are going to use the library. I can effectively not use the Spring AWS messaging library to set up a skeleton listener class in my library project because of this.

Would you consider adding a way to setup an SQS listener dynamically, through code, so I can add and remove queues and listeners at runtime? I found at least one other person who once had a need for this (1.5 years ago though).

[raoofm]

+1 for this, would be a great feature

[mmaryo]

+1

[marcoblos]

If you want to deal with dynamic queue's name you can use SPEL like this:

 @SqsListener(value = "my-queue-base-name-${my.param.name}")
public void queueListenerWithDynamicName(MyObject myObject) {
    System.out.println(myObject.getField());
}

And in your properties config you need to set a value to my.param.name, like this: my.param.name=dev or my.param.name=staging or what you want.

I hope it helps.

Marco

[anne-k]

I ended up doing something like that, yes, but that's not truly dynamic. That just extracts the name of the queue from your code into a property file. It doesn't change the fact that you need to know in advance how many queues there are going to be (you need one listener object per queue name in an application).

What I'm talking about is the equivalent of creating queues at runtime and then dynamically creating listeners for them. You can't do that with annotations.

[marcoblos]

Hi @anne-k I got your point now and I don't know how to help you now. If I find a way, I will comment here.

[kmualem]

Hi guys,

Do you have an update about the creation of the SQS listener dynamically? Is there any way to do it programmatically using the spring cloud library?

Some other use case that I can think of is that the AWS credentials are not static and need additional work in runtime in order to read them... then, there should be an option to initiate the listener on the fly.

Any help is appreciated!

Thanks, Kobi.

[cesargomezvela]

Hi there,

I faced the same problem. Do you have an update about this topic?

Meanwhile, here you are a workaround.

@Configuration
public class SpringCloudAwsConfig {
    @Autowired
    private AmazonSQSAsync amazonSqsAsync;

    public static final String QUEUE_NAME = UUID.randomUUID().toString();

    @PostConstruct
    public void configure(  ) {
        CreateQueueResult createResult = amazonSqsAsync.createQueue(QUEUE_NAME);
        System.setProperty("my.param.name", QUEUE_NAME);
    }
}

@RestController
public class SQSController {
    @SqsListener("${my.param.name}")
    private void receiveMessage(String message) {
        System.out.println("***receive message: "+message);
    }
}

[trouptelia]

The fact org.springframework.cloud.aws.messaging.listener.SimpleMessageListenerContainer does not implement an interface and instead this concrete class is created and autowired we certainly have a few challenges to achieving this with the code as it stands.

I have not tried this myself but one complete hack that could be attempted would be to autowire in the created instance of SimpleMessageListenerContainer and use reflection to directly access the queueAttributes private method to get the queue attributes then directly add the queue and queue attributes directly to the registeredQueues HashMap private field. You can then "start" polling of the queue by calling the public start method on the SimpleMessageListenerContainer instance passing in your queue name. You then need to worry about adding a method into the list of message handler so it gets invoked when a message is pulled from that queue. At that point I gave up.

This solution clearly violates the intentions of the class designer and is therefore in no way recommended. There could also be unintended consequences that I have not noticed. But if you find yourself completely trapped with no alternative it could be a get out of jail (partially) free card.

[joegoogle123]

I read a blog post taking about a SQS event listening library : https://reflectoring.io/spring-robust-sqs-client/

The mini-framework this author wrote is exactly the sort of thing the original poster is looking for. You can create a registry bean where message handlers can be added alongside arbitrary queue names.

The author even have a spring auto configuration that could easily be repackaged as a spring-starter similar to aws-messaging. If someone were to reach out to the author maybe this could be integrated into the spring aws cloud codebase.

I actually am using SQS and spring for a project and have come across this exact pain point. I would be open to integrating this into the spring aws messaging library of course with the authors blessing/permission.

[guilhermeandraschko]

@anne-k I've presented one answer to that stackoverflow post Programatically consume from multiple queues sqs

[omerhakanbilici]

I read a blog post taking about a SQS event listening library : https://reflectoring.io/spring-robust-sqs-client/

The mini-framework this author wrote is exactly the sort of thing the original poster is looking for. You can create a registry bean where message handlers can be added alongside arbitrary queue names.

The author even have a spring auto configuration that could easily be repackaged as a spring-starter similar to aws-messaging. If someone were to reach out to the author maybe this could be integrated into the spring aws cloud codebase.

I actually am using SQS and spring for a project and have come across this exact pain point. I would be open to integrating this into the spring aws messaging library of course with the authors blessing/permission.

@joegoogle123 your reflectoring io link goes somewhere else. can you edit please?