Can't subscribe to SQS queues in Load Balanced Web Service

[wub]

Hi, I have two services:

1. Service A (Load Balanced Web Service)
2. Service B (Worker Service)

Service A has the following config in its manifest:

publish:
  topics:
    - name: foo_requests

# You can register to topics from other services.
# The events can be received from an SQS queue via the env var $COPILOT_QUEUE_URI.
subscribe:
  topics: 
    - name: foo_responses
      service: service_b

Service B has the following config in its manifest:

publish:
  topics:
    - name: foo_responses

subscribe:
  topics: 
    - name: foo_requests
      service: service_a

In Service B, the Worker, I have all the required SNS/SQS running, and the environment variables have been set:

COPILOT_QUEUE_URI=https://sqs.ap-x.amazonaws.com/xxx/x-EventsQueue-xxx
COPILOT_SNS_TOPIC_ARNS={"foo_responses":"arn:aws:sns:xxx:xxxxx:xxxx-foo_responses"}

In Service A, the Load Balanced Web Service, I don't get COPILOT_QUEUE_URI (it doesn't seem to ever be created). However, SNS is connected correctly:

COPILOT_SNS_TOPIC_ARNS={"foo_requests":"arn:aws:sns:xxx:xxxxx:xxxx-foo_requests"}

Is it intentional that Load Balanced Web Services can't subscribe to SQS queues? Or am I doing something wrong? Thank you.

[wub]

I've just found a quote from @efekarakus on Gitter:

The publish field is available for all service manifest types, however subscribe is strictly unique to type: Worker Service.

It would be great to know what the recommended architecture would be, or the rationale for this, as I assume that polling queues from a web service is a valid use case, but I could be wrong!

We could always just POST to an API endpoint, but then we lose out on all the value of queues in the first place.

[KollaAdithya]

Hello @wub ! This is Copilot pub/sub architecture.

The reason why the subscribe field is not available in load balanced services in AWS Copilot CLI is because load balanced services are typically designed to handle incoming HTTP requests from users or other services.

If you need to use message queues for your application, you could consider using a separate worker service that subscribes to the message queue and processes the messages. You could then have your load balanced service send messages to the queue as needed.

I am not currently aware of the use cases for having a load balanced service subscribe to an SQS queue. It would be interesting 🤔 to know from you what some use cases are where a load balanced service needs to interact with a message queue.

[wub]

Hi @KollaAdithya, thanks for the info! Hopefully this provides more context:

In short: we have a monolith.

Service A is our C# SaaS monolith - this is a web server, and is attached to our main Aurora database. Service B is one of our Python machine learning services - this is a worker service.

1. Service A determines the entities that we need to run ML models on, and inserts the entities into a queue
2. Service B polls the queue, executes the models, and stores the results into another queue
3. Service A polls that queue, fetches the results, validates them based on centralised domain logic, and stores them in its own database, so we can show them to users.

We're cautious of exposing the database directly to Service B, because our monolith has all of our domain logic, database logic, etc., and this opens us up to scenarios where that logic may be skipped, or will need to be duplicated. (And we're trying to stick to "Database-per-service" as recommended by AWS).

At this stage, it would be unneeded complexity for us if we were to try and split out our domain logic and central database into yet another service, with another layer in between.

I'm not sure of the main business goals for Copilot, but I can totally see the value in Copilot for people (like us) who have monoliths with several auxiliary services. Using queues will be great for us, as it allows each component of our stack to be more resilient (and all the other queue benefits).

It could also be the case that we're going down the wrong road entirely, and there's a better solution that we haven't thought of!

I hope this helps! Thank you.

[wub]

Hey @KollaAdithya, did you have any feedback on the above? Thank you.

In the meantime, we've had to manually set up our own SQS/SNS outside of Copilot - it would be nice to have it all in one place.

[paragbhingre]

Hi @wub, We definitely understand your use case and why you would want to have SQS on a Load Balanced Web Service. As you already know that Copilot doesn't provide a support for using SQS with LBWS, but you can definitely try using addons for creating SQS on LBWS and try subscribing to a topic foo_responses. We are also releasing Extending Copilot with CDK soon that could be another easy way to achieve this.

[wub]

Thanks @paragbhingre!

[SoundsSerious]

@paragbhingre @build-with-aws-copilot @KollaAdithya

I really dont understand why this feature isn't supported, as setting up SQS / SNS permissions is really quite simple as you recommend with addons. This is ultimately a permission/config issue and the whole point of copilot as I understand it is to reduce workload in dealing with verbose AWS concepts.

Just because you can't imagine a scenario where you'd want your front end to respond to events doesn't mean it doesnt exist. Lets your users decide how to use the copilot tools!

To add to @wub 's use case scenario, I have a a front end system that serves a computationally intensive package that is continually cached by backend services and takes 80min/cpu to prepare all packages for serving. Having a front end system subscribe to events to load these packages would be the ideal scenario vs polling on the minute.

I think subscribe should also be a feature for backend service, ie something that needs to connect to the web and also subscribe to events. In fact what is the purpose of worker service at all?? backend service can be isolated all together. It feels like your boxing us into cookie-cutter solutions when copilot should be capable of much more.

Please make this happen!