Cannot use local mock SQS.

[parabolic]

Hi, I am having an issue where I am following the wiki for using the mock sqs with shoryuken and I cannot seem to get it working.

I am using a docker compose environment in which I build a moto docker and I try to initiate a connection to the SQS docker with shoryuken but I get the following error.

/bundle/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The security token included in the request is invalid. (Aws::SQS::Errors::InvalidClientTokenId)

This looks like shoryuken is still trying to access aws and I am unable to use the mock SQS.

I am using moto with a simple docker file and the shoryuken docker can access the endpoint. What I noticed so far is that these env variables need to passed even with dummy data or shoryuken fails at an earlier stage.

AWS_REGION=region
AWS_SECRET_ACCESS_KEY=dude
AWS_ACCESS_KEY_ID=dude

Thanks!

[phstc]

Hi @parabolic

Could you paste the error again. It is not showing up in your issue description.

What's the version of Shoryuken are you using?

[parabolic]

Hi @phstc, sorry about that. Copy pasting moves the focus to the comment button so I was just editing my half empty message :))) props for the speedy response! :)

[phstc]

@parabolic can you add more lines of the backtrace, the one you shared is only showing the AWS error, would be great to see it down to the Shoryuken classes.

What's the version of Shoryuken are you using?

Also that ☝️

and how you start Shoryuken? How's your shoryuken.xml?

[parabolic]

Hi @phstc, after reading your questions I realized that I hadn't configured it exactly as described in the wiki. So now that it's configured properly ( my half dead brain thinks that it is ) I get a different error

/bundle/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call': The specified queue does not exist for this wsdl version. (Aws::SQS::Errors::QueueDoesNotExist)

Here's the full error output https://gist.github.com/parabolic/d0a27113d4bcc643f5348f5203d27cd5

Gem versions:

aws-sdk-core (3.6.0)
aws-sdk-sqs (1.3.0)
shoryuken (3.1.12)

We don't have a shoryuken.xml and we start shoryuken as a script.

Shoryuken.configure_client do |config|
  config.sqs_client = Aws::SQS::Client.new(
    endpoint: 'http://moto_sqs:4576',
    verify_checksums: false
  )
end
opts = { queues: [queue_name] }.freeze
Shoryuken::Runner.instance.run(opts)

I've double checked that moto is running properly and is listening on that address.

telnet moto_sqs 4576
Trying 192.168.80.2...
Connected to moto_sqs.
Escape character is '^]'.
^]
telnet> quit

I am starting moto with the following command.

ENTRYPOINT ["moto_server", "-H", "0.0.0.0", "-p", "4576","sqs"]

[phstc]

Aws::SQS::Errors::QueueDoesNotExist

@parabolic did you create the queue?

[parabolic]

Hi @phstc, sorry about the late reply. You are right, the queue needed to be created, sorry about that. So now it works with moto but I've the feeling that the wiki needs an update. To me it wasn't that obvious that the queue needs to be created. I am more used to the AWS behavior where you just create it and it works :) Thanks a lot for the immediate response. This issue can be closed. Cheers.

[chr0n1x]

@parabolic @phstc I'm currently running into the same issue here on a local instance of rails with moto. @parabolic - what did you do to create? I'm creating my queues in an initializer script:

sqs = Aws::SQS::Client.new(
    endpoint: ENV["LOCAL_SQS_ENDPOINT"],
    verify_checksums: false
)

queues.each do |queue_name|
    begin
        sqs.create_queue(queue_name: queue_name)
    end
end

Shoryuken.configure_server { |config| config.sqs_client = sqs }
Shoryuken.configure_client { |config| config.sqs_client = sqs }

But that doesn't seem to be working since I'm getting the same error that you had in that gist 😕

[parabolic]

Hi @chr0n1x, we've moved on to https://github.com/localstack/localstack because I think it's a bit better than moto but the principle is the same. Here's what I have and it works well. Hope this helps.

Shoryuken.configure_client do |config|
    config.sqs_client = Aws::SQS::Client.new(
      endpoint:         ENV["SQS_ENDPOINT"],
      verify_checksums: false
    )
  end
  # Create the sqs with the env var name.
  Shoryuken::Client.sqs.create_queue(queue_name: ENV["SQS_QUEUE_NAME"])

[chr0n1x]

@parabolic ah ok good to know, thanks. I actually got something working using elasticmq. The main issue for me was actually due to migrating from Shoryuken 2.0 to 3.0.

For posterity, the issue that I was facing was that I had to force the Shoryuken Runner to actually account for the Rails configuration while still switching over to an instance of the SQS client that had verify_checksums: false. On older versions of the lib we were just running:

Shoryuken::CLI.instance.run(["-R", "--config", "config/shoryuken.yml"])

to start up our workers. But now we have to:

Shoryuken.sqs_client = Aws::SQS::Client.new(
    verify_checksums: false,
    endpoint: "http://sqs:5000"
)
Shoryuken::Runner.instance.run(config_file: "config/shoryuken.yml", rails: true)

@phstc is this something that is in the docs and I completely missed it?

[phstc]

Hi @chr0n1x, sorry for taking long to answer this.

@phstc is this something that is in the docs and I completely missed it?

I couldn't find any reference in the wiki for Shoryuken::Runner.instance.run. Do you start Shoryuken like that for testing?

[chr0n1x]

@phstc we run an instance of elasticmq locally for development purposes. Our application has multiple server & worker components, the latter being driven by Shoryuken backed by rails so our startup script needed to account for that (i.e.: a custom dev/test shoryuken startup with our rails env injected in @ startup).