search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.83k stars 24.71k forks source link

JMS River #1109

Closed hvandenb closed 11 years ago

hvandenb commented 13 years ago

ElasticSearch has a river for RabbitMQ. However, it would be nice to have a generic river that leverages JMS. This way JMS providers such as MQ, TIBCO EMS etc can be indexed.

Ideally, it should support the same format as the bulk api. However, ideally an XML representation would be great. Alternatively just being able to index the JMS message directly is another option. Support for TOPICS and QUEUES will be needed.

Be glad to help develop this solution.

kimchy commented 13 years ago

Helping out would be great. I guess the logic should be similar to the rabbitmq one...

hvandenb commented 13 years ago

The main logic should be the same. We would have to deal with some vendor specific libs.

kimchy commented 13 years ago

Yea, that would be the tricky part, vendor specific connection creation and the like...

tblachowicz commented 13 years ago

The implementation could rely on the JMS and JNDI APIs. So it could get the ConnectionFactory remotely from JNDI. Using this approach the challenge is to create initial JNDI context (set the properties, provide the specific libraries) and then it all goes along the standard JMS API.

hvandenb commented 13 years ago

That's the right approach, we'll have to be able to do JNDI & JMS as some JMS systems don't handle JNDI well or in some configurations it's desired to not use JNDI. For example, if an organization does not have JNDI/LDAP for the connection factories. Additionally, we would need to store security info for JNDI & JMS as they might be different.

YannBrrd commented 11 years ago

What is the status of this issue ?

YannBrrd commented 11 years ago

Would this help ? I don't know much about JMS...

http://docs.oracle.com/javaee/1.4/tutorial/doc/JMS5.html

qotho commented 11 years ago

I have put the code for a JMS river plugin in GitHub at https://github.com/iooab23/elasticsearch-river-jms. I am still in the process of unit testing the river, but I have successfully submitted bulk requests using Web Logic. As suggested above, I have used the standard JMS and JNDI APIs to keep the code vendor neutral. I am however having a problem figuring out how to generically handle the unit testing. Any suggestions?

qotho commented 11 years ago

@hvandenb: Regarding your comments about using JMS instead of JNDI. I initially attempted to use session.createQueue and session.createTopic to provide a non-JNDI option, but doing this uses a vendor-specific syntax to identify the queue or topic which I have had a hard time getting to work with WebLogic. For now the plugin is using JNDI.

hvandenb commented 11 years ago

Typically you use a driver url or name for these items so that the driver is loaded dynamically. I can look at the code that you have currently and I'll checkout what I can contribute.

spinscale commented 11 years ago

@qotho thanks a lot for creating a river!

Do you consider it stable enough, so we can add it to the documentation on elasticsearch.org?

spinscale commented 11 years ago

Added your PR from https://github.com/elasticsearch/elasticsearch.github.com/pull/418 - closing this ticket