search

LD4P / sinopia

LD4P Sinopia Project repo to hold docs, general issues, schemas, and related spec docs.
https://ld4p.github.io/sinopia/
19 stars 3 forks source link

Fix or Replace Trellis use of ActiveMQ #287

Closed jermnelson closed 4 years ago

jermnelson commented 4 years ago

In the current Sinopia stack, the Trellis LDP publishes messages to an AWS ActiveMQ queue that is then picked up by the Indexing Pipeline that then either adds/updates/deletes the resource document in the Elasticsearch cluster. Occasionally for the past year-and-a-half, Trellis stop connecting to ActiveMQ which halts all Elasticsearch indexing activity. Trellis now supports AMQP or an alternative is to use Camel for direct indexing into Elasticsearch.

mjgiarlo commented 4 years ago

@jermnelson :speech_balloon:

In the current Sinopia stack, the Trellis LDP publishes messages to an AWS ActiveMQ queue that is then picked up by the Indexing Pipeline that then either adds/updates/deletes the resource document in the Elasticsearch cluster. Occasionally for the past year-and-a-half, Trellis stop connecting to ActiveMQ which halts all Elasticsearch indexing activity. Trellis now supports AMQP or an alternative is to use Camel for direct indexing into Elasticsearch.

I have no attachment to ActiveMQ but I am wondering if we have a sense of what causes the disconnects. My worry is that we put effort into swapping out ActiveMQ for an AMQP-based back-end (or Camel) only to have them suffer from the same disconnects.

justinlittman commented 4 years ago

ActiveMQ connections are notoriously flacky because they maintain an active connection. In other words, I think the problem is with the ActiveMQ stack (exasperated by running in AWS, where the network is a black box), not anything Trellis specific.

That being said, nothing to guarantee won't encounter problems with AMQP or Camel. The Camel connector for ElasticSearch might be worth a look, as at least for communicating with ES, is straight HTTP.

jermnelson commented 4 years ago

Closing as Trellis is being depreciated in favor of MongoDB which doesn't use ActiveMQ.