search

bbc / sqs-consumer

Build Amazon Simple Queue Service (SQS) based applications without the boilerplate
https://bbc.github.io/sqs-consumer/
Other
1.71k stars 330 forks source link

chore: upgrade @aws-sdk/client-sqs to version which supports sqs json protocol #440

Closed petrisorciprian-vitals closed 7 months ago

petrisorciprian-vitals commented 7 months ago

Description: Updates the aws sdk to a version which supports the AWS SQS JSON protocol.

Type of change:

Why is this change required?: It enables better performance due to usage of the AWS SQS JSON communication protocol instead of the AWS SQS Query protocol between client and server.

Code changes:

Checklist:

petrisorciprian-vitals commented 7 months ago

The localstack upgrade is required since the previous version did not support the SQS JSON protocol as it was old; only the 'latest' tagged image seems to work properly (https://github.com/localstack/localstack/pull/9607).

The healthcheck path and init script path also changed with in more recent versions of localstack and it is also unnecessary to mount a tmp dir.

More recent versions of docker also have a 'compose' subcommand instead, so I updated the initIntTest.sh script for that; let me know if this is somehow breaking in CI or something.

This change should be fully compatible with AWS SQS APIs, it's only in localstack that issues appear.

The benefit of this update is that "based on AWS performance tests for a 5KB message payload, JSON protocol for Amazon Simple Queue Service reduces end-to-end message processing latency by up to 23% and reduces application client side CPU and memory usage."

nicholasgriffintn commented 7 months ago

Our CLA check is broken for some reason so I'll just have to do that manually...

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment reply with the following:""

"I have read the CLA Document and I hereby sign the CLA"

dciprian-petrisor commented 7 months ago

I have read the CLA Document and I hereby sign the CLA

codeclimate[bot] commented 7 months ago

Code Climate has analyzed commit 9d8cc912 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (80% is the threshold).

This pull request will bring the total coverage in the repository to 98.2% (0.0% change).

View more on Code Climate.

nicholasgriffintn commented 7 months ago

To note: This would require a major version of the package as the new version of aws-sdk is breaking against anything that's expects the XML protocol.

petrisorciprian-vitals commented 7 months ago

@nicholasgriffintn I've pushed a new commit which removes some obsolete envs from the localstack service in docker-compose.yml See https://docs.localstack.cloud/references/configuration/#legacy for details.

nicholasgriffintn commented 7 months ago

Hey thanks for the updates and sorry, I've been super busy this week, I'm gonna slot some time to look into this on Friday hopefully, if not before.

github-actions[bot] commented 7 months ago

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

I have read the CLA Document and I hereby sign the CLA

1 out of 2 committers have signed the CLA.
:white_check_mark: (nicholasgriffintn)[https://github.com/nicholasgriffintn]
:x: @petrisorciprian-vitals
You can retrigger this bot by commenting recheck in this Pull Request

nicholasgriffintn commented 7 months ago

CLA Assistant Lite bot: Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

I have read the CLA Document and I hereby sign the CLA

1 out of 2 committers have signed the CLA.✅ (nicholasgriffintn)[https://github.com/nicholasgriffintn]❌ @petrisorciprian-vitalsYou can retrigger this bot by commenting recheck in this Pull Request

This was signed here: https://github.com/bbc/sqs-consumer/pull/440#issuecomment-1814101497