Localstack SQS Network Failure

[paladan-artium]

Hi there!

I just downloaded Commandeer and love it so far. Managing all of our AWS services with the AWS GUI has been a huge pain - I'm loving the transparency that Commandeer provides around which services we are using and how.

Our project is currently using localstack inside of a docker container to manage our local S3 and SQS dependencies. Commandeer was able to detect and provide visibility into the S3 bucket by default without issue, but for SQS it displays a "Network Failure" error message. The app works fine when looking at our AWS environment SQS queues.

Is there any way to get further insight into what might be causing the Network Failure with localstack? Thanks!

[fsproru]

Hi @paladan-fractal,

Thanks for the feedback. I just stood up a fresh install of LocalStack in a Docker container with localstack start and the SQS shows as connected. Would you mind sharing some of your logs (feel free to obfuscate any sensitive info)?

The logs are located at the following paths:

on Linux: ~/.config/Commandeer/log.log. on macOS: ~/Library/Logs/Commandeer/log.log. on Windows: %USERPROFILE%\AppData\Roaming\Commandeer\log.log.

[paladan-artium]

Thanks for the quick response! Unfortunately the logs don't yield much information:

[2019-09-19 13:33:30.681] [info] App starting...
[2019-09-19 13:33:30.774] [info] Checking for update
[2019-09-19 13:33:31.122] [info] Generated new staging user ID: 02486f19-0ed6-50c8-abfc-fdb8771b6244
[2019-09-19 13:33:31.362] [info] Update for version 0.2.5 is not available (latest version: 0.2.5, downgrade is disallowed).

An additional detail about our setup is that we are using docker-compose in order to spin up the localstack container from Docker Hub, so we don't actually call the localstack start command ourselves. I've included our obfuscated docker-compose.yml below.

version: "3"
services:
  db:
    image: "postgres:11"
    container_name: "postgres"
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=xxx
      - POSTGRES_PASSWORD=xxx
      - POSTGRES_DB=xxx
    volumes:
      - touchpoints_authorizations:/var/lib/postgresql/data
  localstack:
    image: localstack/localstack
    ports:
      - "4576:4576"
      - "4572:4572"
    environment:
      - SERVICES=sqs:4576,s3:4572
      - DEBUG=true
      - DOCKER_HOST=unix:///var/run/docker.sock
      - DEFAULT_REGION=us-west-1
    volumes:
      - "/tmp/localstack:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./scripts/createBucket.sh:/docker-entrypoint-initaws.d/createBucket.sh"
      - "./scripts/createQueues.sh:/docker-entrypoint-initaws.d/createQueues.sh"
volumes:
  touchpoints_authorizations:

[fsproru]

Hey @paladan-fractal, just stood it up with the following docker-compose.yml.

version: "3"
services:
  db:
    image: "postgres:11"
    container_name: "postgres"
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=xxx
      - POSTGRES_PASSWORD=xxx
      - POSTGRES_DB=xxx
    volumes:
      - touchpoints_authorizations:/var/lib/postgresql/data
  localstack:
    image: localstack/localstack
    ports:
      - "4576:4576"
      - "4572:4572"
    environment:
      - SERVICES=sqs:4576,s3:4572
      - DEBUG=true
      - DOCKER_HOST=unix:///var/run/docker.sock
      - DEFAULT_REGION=us-west-1
    volumes:
      - "/tmp/localstack:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
volumes:
  touchpoints_authorizations:

It shows SQS as running in Commandeer when I go to LocalStack and hit the Scan button. The only modification I made to docker-compose.yml is removing the two volumes records at the bottom:

      - "./scripts/createBucket.sh:/docker-entrypoint-initaws.d/createBucket.sh"
      - "./scripts/createQueues.sh:/docker-entrypoint-initaws.d/createQueues.sh"

Some thoughts:

• Wondering if there is something going on in the createBucket.sh and createQueues.sh?
• Are you using the latest version of Commandeer 0.2.5?
• What happens when you hit the Scan button again? Does it refresh the status?

[paladan-artium]

Hey again! Sorry for the delay.

We were also able to successfully avoid the Network Failure error when we removed those scripts from the docker-compose. We think the scripts are working because we are able to stand up our application that connects to our queues.

createBucket.sh - This one appears to work. We see the localstack S3 bucket created in this script appear in Commandeer.

#!/usr/bin/env bash
export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx
export AWS_DEFAULT_REGION=us-west-1
aws s3 mb s3://bucket-name --endpoint-url http://localhost:4572 --region us-west-1

createQueues.sh

#!/usr/bin/env bash
export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx
export AWS_DEFAULT_REGION=us-west-1
aws sqs create-queue --queue-name queue-dev-1.fifo --endpoint-url http://localhost:4576 --region us-west-1 --attributes '{"FifoQueue": "true", "ContentBasedDeduplication":"true"}'
aws sqs create-queue --queue-name queue-dev-2.fifo --endpoint-url http://localhost:4576 --region us-west-1 --attributes '{"FifoQueue": "true", "ContentBasedDeduplication":"true"}'
aws sqs create-queue --queue-name queue-dev-3.fifo --endpoint-url http://localhost:4576 --region us-west-1 --attributes '{"FifoQueue": "true", "ContentBasedDeduplication":"true"}'

It's only after running the createQueues script that we receive the Network Failure in Commandeer, but our app is able to interact with the queues as expected.

Thanks again for helping us with this!

[fsproru]

Hi @paladan-fractal, thanks for your feedback too. We'll keep an eye on this issue. If you see anything else, don't hesitate to reach out.

[jbentus]

hello @fsproru. I am having the same issue as well.

After I create an SQS queue, I start getting a Network Failure error and I can't see my queue in the SQS tab, however the localstack dashboard works fine.

[david-dacostad]

Like @jbentus ... i am also getting this issue... i am creating the localstack assets using terraform... i am able to see the sqs queues via the localstack dashboard

[jbentus]

@fsproru I think this issue should be reopened. The trigger to replicate is to create an sqs queue (e.g. terraform) and that will make Commandeer throwing Network Failure error in the SQS tab

[fsproru]

@david-dacostad @jbentus, will be happy to look into it. Would you mind sharing a terraform template you're running to create the SQS queue causing the Network Failure in Commandeer?

[jbentus]

Hi @fsproru.

It's easier to reproduce this if you use the following AWS CLI command: aws --endpoint-url=http://localhost:4576 sqs create-queue --queue-name test_sqs_queue

I've upgraded to the latest version of Commandeer and tried provision the SQS queue via terraform and aws cli and either approach Commandeer does not detect any SQS queue, even though I can see them on localstack's dashboard.

In the latest Commandeer version, if you retry to connect, the Network Failure will disappear but you will still not detect any sqs queues.

[fsproru]

Hey @jbentus, thanks for providing the details. Just tried creating a queue with the SQS command you provided. Here is the exact command I used:

AWS_ACCESS_KEY_ID=NO_KEYS AWS_SECRET_ACCESS_KEY=NO_KEYS aws --endpoint-url=http://localhost:4576 sqs create-queue --queue-name test_sqs_queue_3 --region us-east-1

The topics show up in Commandeer on the SQS tab with the account switched to local. Here is a screenshot.

The issue may be in your local setup. Could you navigate to the help menu in the top right corner, click on data path and send me your account.uuid.json where it says isLocal: true. It's your local account file. Wondering if the endpoint URL got messed up somehow.

[simenko]

I have got the same bug. I run localstack container with the following docker-compose config:

localstack:
    image: localstack/localstack
    ports:
      - "4567-4599:4567-4599"
    environment:
      - SERVICES=sqs, sns
      - DEBUG=1
      - DATA_DIR=./localstack/data
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "./localstack/tmp:/tmp/localstack"
      - "./localstack/bootstrap/createTopics.sh:/docker-entrypoint-initaws.d/createTopics.sh"
      - "./localstack/bootstrap/createQueues.sh:/docker-entrypoint-initaws.d/createQueues.sh"
    container_name: localstack
    networks:
      - test

I create one test topic and one test queue as follows:

#createTopics.sh
#!/usr/bin/env bash
export AWS_ACCESS_KEY_ID=NO_LOCAL_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=NO_LOCAL_SECRET_ACCESS_KEY
export AWS_DEFAULT_REGION=us-west-1
aws sns create-topic --name test --endpoint-url http://localhost:4575 --region us-west-1 

# createQueues.sh
#!/usr/bin/env bash
export AWS_ACCESS_KEY_ID=NO_LOCAL_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=NO_LOCAL_SECRET_ACCESS_KEY
export AWS_DEFAULT_REGION=us-west-1
aws sqs create-queue --queue-name test --endpoint-url http://localhost:4576  --region us-west-1  --attributes '{"DelaySeconds": "60","MessageRetentionPeriod": "86400","WaitTimeSeconds": "20"}'

Both topic and queue are successfully created:

localstack          | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initaws.d/createQueues.sh
localstack          | {
localstack          |     "QueueUrl": "http://localhost:4576/queue/test"
localstack          | }
localstack          | 
localstack          | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initaws.d/createTopics.sh
localstack          | 13:40:37.462 [elasticmq-akka.actor.default-dispatcher-9] INFO  o.elasticmq.actor.QueueManagerActor - Creating queue QueueData(test,MillisVisibilityTimeout(30000),PT60S,PT0S,2020-03-04T13:40:37.431Z,2020-03-04T13:40:37.431Z,None,false,false,None,None,Map())
localstack          | {
localstack          |     "TopicArn": "arn:aws:sns:us-east-1:000000000000:test"
localstack          | }

I can access SQS via browser at http://localhost:4576/queue/test:

<ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/">
      <Error>
        <Type>Sender</Type>
        <Code>MissingAction</Code>
        <Message>MissingAction; see the SQS docs.</Message>
        <Detail/>
      </Error>
      <RequestId>00000000-0000-0000-0000-000000000000</RequestId>
    </ErrorResponse>

I can see SNS test topic in Commandeer, but there is now SQS queues and scan fails with Network Error for each region: If I don't create a queue, scan is OK.

Here is my account.json:

{
   "isLocal":true,
   "isProduction":false,
   "id":"0dd0925b-3d22-4cec-96f7-6888a61cdad5",
   "name":"local",
   "algolia":{
      "applicationId":"",
      "apiKey":""
   },
   "aws":{
      "region":"us-west-1",
      "accessKeyId":"NO_LOCAL_ACCESS_KEY_ID",
      "secretAccessKey":"NO_LOCAL_SECRET_ACCESS_KEY",
      "apigatewayEndpoint":"http://localhost:4567",
      "cloudwatchEndpoint":"http://localhost:4582",
      "cloudwatchLogsEndpoint":"http://localhost:4586",
      "cloudwatchEventsEndpoint":"http://localhost:4587",
      "dynamoEndpoint":"http://localhost:4569",
      "ec2Endpoint":"http://localhost:4597",
      "iamEndpoint":"http://localhost:4593",
      "lambdaEndpoint":"http://localhost:4574",
      "s3Endpoint":"http://localhost:4572",
      "s3Prefix":"",
      "snsEndpoint":"http://localhost:4575",
      "sqsEndpoint":"http://localhost:4576"
   },
   "database":{
      "host":"",
      "initialDatabase":"",
      "username":"",
      "password":"",
      "type":"POSTGRES"
   },
   "slack":{
      "token":""
   }
}

[jbentus]

@fsproru I didn't find the account.uuid.json file but i've got a similar one called account.local.json which has:

{"id":"local","name":"local","algolia":{"applicationId":"","apiKey":""},"aws":{"region":"us-east-1","accessKeyId":"NO_LOCAL_ACCESS_KEY_ID","secretAccessKey":"NO_LOCAL_SECRET_ACCESS_KEY","apigatewayEndpoint":"http://localhost:4567","cloudwatchEndpoint":"http://localhost:4582","dynamoEndpoint":"http://localhost:4569","ec2Endpoint":"http://localhost:4597","iamEndpoint":"http://localhost:4593","lambdaEndpoint":"http://localhost:4574","s3Endpoint":"http://localhost:4572","s3Prefix":"","snsEndpoint":"http://localhost:4575","sqsEndpoint":"http://localhost:4576"},"firebase":{"databaseUrl":"DEFAULT_LOCAL_URL"},"slack":{"token":""},"isProduction":false,"isLocal":true}

[fsproru]

Hey guys, just ran the bash scripts to create test SNS and SQS and they're displayed in Commandeer after hitting the refresh button. If you can make sure you hit refresh, that would be great.

The only thing I can think of if it's a platform-specific issue. We have some machines we can test on too.

I'm on macOS Catalina. @jbentus, @simenko, which platform and version you're on?

[fsproru]

Also just worked on Ubuntu 18.04

[jbentus]

@fsproru yeah it might be. I am running on a Windows 10 machine, and running localstack:0.10.4 as a docker image

[bwship]

@jbentus - we are using this version - localstack/localstack:0.10.6 in Commandeer, not sure if that is affecting you. I will try this on my windows 10 box too though to see.

[markingram]

I'm hitting this same problem (using the Docker setup for SQS) on MacOS Catalina 10.15.4 and Commandeer v1.0. Any further insights into why there's a network error?

[markingram]

If I go to Infrastructure -> LocalStack -> Dashboard and click the option to "Start All Services" then when all comes up I can successfully run commands like the following and see it reflected in Commandeer as expected.

$ AWS_ACCESS_KEY_ID=NO_LOCAL_ACCESS_KEY_ID1 AWS_SECRET_ACCESS_KEY=NO_LOCAL_SECRET_ACCESS_KEY1 aws --endpoint-url=http://localhost:4576 sqs create-queue --queue-name test_sqs_queue_key --region eu-west-1
{
    "QueueUrl": "http://localhost:4576/queue/test_sqs_queue_key"
}

[fsproru]

Thanks @markingram, yep, starting localstack fully is best. We also do it using Docker so the environment is consistent on all platforms. Let me know if you run into anything else.

On a related note, we pushed the 1.0.0 version today with a lot of new features. You should check it out ;-)

[markingram]

That was the trick: starting localstack fully. For others, here's the Docker Compose file that works fine for me (on MacOS using Commandeer v1.0.0):

  localstack:
    image: localstack/localstack
    ports:
      - "4566-4599:4566-4599"
      - "12345:8080"
    environment:
      - DEFAULT_REGION=eu-west-1
      - DEBUG=true
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "/private${TMPDIR}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./localstack/createQueues.sh:/docker-entrypoint-initaws.d/createQueues.sh"
      - "./localstack/createTables.sh:/docker-entrypoint-initaws.d/createTables.sh"
      - "./localstack/create-...-table.json:/create-...-table.json"