SQS Persistence Broken

[nitesh237]

Hello,

Firstly, I want to express my gratitude for your efforts in maintaining the persistence of LocalStack.

After trying out the latest version of the LocalStack container, it seems that SQS persistence is currently broken. I encountered that SQS states aren't persisting when attempting to restart or stop the container.

Here are the details of my environment:

LocalStack Version: 3.0.2
Podman Version:  4.6.2
OS: darwin/arm64

Steps to Reproduce:

• Run container and mount the path /persisted-data as recommended in the README. Here is the compose file that I have been using.

  localstack:
  image: gresau/localstack-persist:3.0.2
  privileged: true
  environment:
    SERVICES: s3, sqs, sns, dynamodb, sts, kms, kinesis
    DEFAULT_REGION: ap-south-1
    PORT_WEB_UI: 8015
    PERSIST_SQS: 1
  ports:
    - "4566:4566"
    - "8015:8015"
  volumes:
    - /var/home/core/localstack:/persisted-data

• Create SQS Queueus

  awslocal sqs create-queue --attributes '{ "VisibilityTimeout": "3" }' --queue-name test-queue
  awslocal sqs create-queue --attributes '{ "VisibilityTimeout": "3" }' --queue-name test-queue-1
  awslocal sqs create-queue --attributes '{ "VisibilityTimeout": "3" }' --queue-name test-queue-2
  awslocal sqs create-queue --attributes '{ "VisibilityTimeout": "3" }' --queue-name test-queue-3

• Create SNS Queues

  awslocal sns create-topic --name test-topic
  awslocal sns create-topic --name test-topic-1
  awslocal sns create-topic --name test-topic-2

• Create S3 Buckets

  awslocal s3 mb s3://test-bucket
  awslocal s3 mb s3://test-bucket-1
  awslocal s3 mb s3://test-bucket-2

• Copy a file to one of the s3 bucket

  awslocal s3 cp setup.py s3://test-bucket

• Restart Or Stop the container

  podman container restart <container-name>

Expected Behavior:

• All the resources creates in SQS, SNS, S3 should be intact.

Actual Behavior:

• All the resources of SNS and S3s are intact. The resources in SQS are lost post the container restart. Below command returns empty

  awslocal sqs list-queues

• Upon inspecting the files in persisted-data, I could only see the states from sns and s3 services in place.

  root@05c5c1ab4a5b:/opt/code/localstack# ls /persisted-data/
  s3  sns

This issue is critical for our integration testing suits. If there's anything else needed from my end, please let me know. I appreciate your time and attention to this matter.

[GREsau]

Thanks for the report - I could reproduce this in gresau/localstack-persist:3.0.2, but not in gresau/localstack-persist:latest. After digging a bit further, I found that this would only occur when running localstack-persist based on LocalStack 3.0.1, and it did not occur when using a base image of LocalStack 3.0.2.

I've now published localstack-persist:3.0.3 which is based on LocalStack 3.0.2 (which to be honest I should have done a while ago), so that should fix your problem. Could you give it a try?

I couldn't find exactly what changed in LocalStack to break/fix this behaviour, but I'm fairly sure it's something to do with the updates in 3.0 to add support for the JSON SQS protocol that's used by newer versions of the AWS SDK. This bug seemed to only occur when using older SDKs, which includes awslocal. This also explains why this bug was unfortunately not caught by localstack-persist's automated tests - as part of the tests, a create-queue request is made using the JSON API, which causes persistence to work.

[nitesh237]

Hey @GREsau, thanks a lot for spending time here. It's working in version 3.0.3.