Send an SQS notification when a book record changes

[adolski]

When a book record is created or modified as a result of an import or service check, send a messages to an SQS queue. Later, we will modify the Digital Library Search Gateway to consume them.

[gaurijo]

@adolski I'm not familiar with how SQS works but I'll research and look into any docs or videos I can find. In the meantime, do you have any additional resources or context I can take a look at?

[adolski]

SQS is a queue service--the idea is that an application will send a message to a queue that another application can pull out and read. I'm not that familiar with it either, so I'm sure that whatever resources you find will explain it better than I can. To my knowledge, we don't use it a whole lot in our group--we use a RabbitMQ EC2 instead that we manage--but I think SQS should be easier to set up.

[gaurijo]

So it looks like there are two types of queues I can choose to set up - a Standard Queue and FIFO (first in, first out). Here are the main differences I'm seeing:

Standard Queue:

• uses Best Effort Ordering, so the messages might not be sent/processed in order
• possibility of overlap in the messages being delivered, so there's a small chance the same message can be seen more than once
• unlimited throughput (infinite publishing and pulling rate between services)

FIFO:

• first in/first out ordering, so whichever message arrives first is processed first
• Exactly Once processing, so no overlap of messages being seen more than once.
• Has a MAX TPS (transactions per second) of 300 (or 3000 with Batching)
• Allows for multiple channels of messagesso any messages tagged with the same group ID will always be processed in the same order as they were inserted

Based on what we're working with, I personally think using Standard Queue should be fine for our needs. I don't necessarily see a compelling reason to use FIFO, because I don't think the order of messages for when a book record is created/modified matters, as long as the message is sent. Let me know if you disagree or have other thoughts, though.

[adolski]

Agreed. Order doesn't matter and a few duplicated messages is not a problem.

[gaurijo]

@adolski For the production SQS, I'm considering what the best approach is to call on it from the configuration files. Right now we have it set up like so:

sqs:
  queue_name:          book-tracker-demo

And inside send_message() it gets called on like this:

region = Configuration.instance.storage[:books][:region]
queue_name = Configuration.instance.sqs[:queue_name]

Our production SQS is named book-tracker-prod. I was thinking of updating the config files so it separates out the demo queue name from the prod queue name, and then just use some conditional logic inside the send_message()method.

Something like this:

sqs: 
   demo_queue_name:     book-tracker-demo 
   prod_queue_name:       book-tracker-prod 

def send_message(message)
    return 'This feature does not work in development' if Rails.env.development?
    region = Configuration.instance.storage[:books][:region]

   if Rails.env.demo? 
      queue_name = Configuration.instance.sqs[:demo_queue_name]
   else 
      queue_name = Configuration.instance.sqs[:prod_queue_name]
  end 

    ## rest of code 
end 

Would this be an ideal approach, or do you think there might be a better way?

[adolski]

In the demo environment, the demo.yml.enc file is used, and in production, production.yml.enc. All config files should contain the same keys, but the values will differ by environment. There is no need for any conditional logic—just set sqs.queue_name to book-tracker-demo in the demo file and book-tracker-prod in the production file.

[gaurijo]

That makes sense, will do. Theci.yml file also has it set to book-tracker-demo right now. Is it fine to keep that as is?

[adolski]

It's fine, since the test of Book.send_message() is using a mocked Aws::SQS::Client.