search

LD4P / sinopia_server

[Deprecated - switching to MongoDB] Sinopia Back-end CRUD Service. LDP-inspired, HTTP Server taking JSON-LD resources & administrative metadata.
Apache License 2.0
1 stars 1 forks source link

figure out how to turn Trellis notifications into queue entries #47

Closed jmartin-sul closed 5 years ago

jmartin-sul commented 5 years ago

options seem to be:

  1. trellis emits JMS messags that go directly to our SQS queue (see #46), assuming trellis and SQS speak compatible JMS
  2. trellis emits SNS messages, SNS brokers messages to SQS (since SQS is more durable and makes it easier to guarantee receipt, whereas SNS will stop attempting to broadcast if out of retries, even if the message wasn't successfully acknowledged)
  3. something more bespoke using the AWS SDK and some sort of small custom app or lambda or something (hopefully not this option)
mjgiarlo commented 5 years ago

Integrating SQS with Trellis's JMS implementation may not be as straightforward as we thought. Reading through https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-java-message-service-jms-client.html, it seems to me that this is documenting a Java library that Trellis would have to implement in its own source code. That would eliminate option 1 above.

I haven't seen any docs that suggest Trellis can emit SNS messages, and I also am not aware of any SNS/JMS bridge, which would eliminate option 2 above.

If we want to make use of Trellis's built-in JMS publisher, our best hope may be AmazonMQ, which can already handle JMS. Ideally, that would be as easy as pointing the Trellis JMS configuration at our AmazonMQ endpoint. AmazonMQ supports two concepts: topics and queues. If it publishes to an AmazonMQ queue, the message will stick around for a while (akin to SQS). If a topic, and no one is consuming that topic actively, the message will never be seen (akin to SNS). Thus, in the latter scenario, we'd want to make sure we have a persistent AmazonMQ listener that, e.g., publishes those messages somewhere stickier (an AmazonMQ or SQS queue). I'm not sure whether Trellis would publish to a topic or a queue or how to determine that at the moment. This needs more analysis.

This also means we will likely need to write some code that polls the AmazonMQ or SQS queue (see above paragraph) as part of the work on #46.

mjgiarlo commented 5 years ago

@mjgiarlo :speech_balloon:

... AmazonMQ supports two concepts: topics and queues. If it publishes to an AmazonMQ queue, the message will stick around for a while (akin to SQS). If a topic, and no one is consuming that topic actively, the message will never be seen (akin to SNS). Thus, in the latter scenario, we'd want to make sure we have a persistent AmazonMQ listener that, e.g., publishes those messages somewhere stickier (an AmazonMQ or SQS queue). I'm not sure whether Trellis would publish to a topic or a queue or how to determine that at the moment. This needs more analysis.

Looking at the Trellis source code, it looks like it publishes to a queue rather than a topic: https://github.com/trellis-ldp/trellis/blob/master/notifications/jms/src/main/java/org/trellisldp/jms/JmsPublisher.java#L100

See in particular the usage of #createQueue over #createTopic (which are documented in Javadoc: https://docs.oracle.com/javaee/7/api/javax/jms/Session.html#createQueue-java.lang.String-).

jermnelson commented 5 years ago

Hi @mjgiarlo, I just pushed up a branch jms-notification that uses a Docker ActiveMQ image with Trellis (this was just to test how Trellis sends JMS locally) and I believe that you are right, Trellis is using ActiveMQ/AmazonMQ queues. I haven't gone beyond setting up the docker-compose and config.yml changes.

mjgiarlo commented 5 years ago

@jermnelson :speech_balloon:

Hi @mjgiarlo, I just pushed up a branch jms-notification that uses a Docker ActiveMQ image with Trellis (this was just to test how Trellis sends JMS locally) and I believe that you are right, Trellis is using ActiveMQ/AmazonMQ queues. I haven't gone beyond setting up the docker-compose and config.yml changes.

Oh, super! That is enough to get me off the blocks for now. Thank you. :smiley:

mjgiarlo commented 5 years ago

@jermnelson I added the 61613 port to the docker-compose config and now I have things working: https://github.com/LD4P/sinopia_server/issues/46#issuecomment-462521824

Thanks again.