search

rebus-org / Rebus.Oracle

:bus: Oracle transport for Rebus
https://mookid.dk/category/rebus
Other
5 stars 10 forks source link

Oracle Advanced Queueing Support #9

Open agerchev opened 5 years ago

agerchev commented 5 years ago

Are there any plans to support Oracle AQ as a valid transport ?

mookid8000 commented 5 years ago

No – I didn't know it existed 😁

Is it something you know something about?

agerchev commented 5 years ago

Well, we have used in one project but we used a custom message pump. Now we saw the rebus library, and want to use it, but we have to support the Oracle AQ. I suppose we have to make a new transport.

Do you have any plans to incorporate REBUS-TaskCoordinator made by @BBGONE ?

mookid8000 commented 5 years ago

Wow you just made me realize that I completely forgot about @bbgone 's fork.... I would have to consider it carefully, as it's a pretty central replacement, but it definitely sounds very promising.

I've created this issue, which I might have time to look at either this weekend or at the end of next week.

If you're interested in contributing an Oracle AQ-based Rebus transport, I'll be happy to help you and answer any questions you might have.

agerchev commented 5 years ago

@mookid8000 I have some questions about the implementation.

  1. We have a couple of choices for the persistence of the messages. The one is faster but it has limitations of size of the message: ~2k body, 2k header. I was thinking about configuring the transport for the two cases. One for the faster(limited in size) and the other is the slower. I saw that the tables are created at startup, but the queue configuration is more complex. Do you think we need it this way or just include a DDL in the package ?
agerchev commented 5 years ago
  1. In Oracle AQ rerty logic is handled by the queue itself and moved to the exception queue. Is there any implementation of IRetryStrategy that i can reuse or have to do it myself ?
mookid8000 commented 5 years ago
  1. Is it just a matter of configuration whether to use the "fast & bounded persistence" or the "slow & unrestricted persistence"?

Regarding the necessary schema: "The Rebus Way" 😆is to automatically create all of the necessary entities, like tables & queues, for things to "just work" out of the box.

It should be possible to disable this, though, so the bus can still start e.g. if it's running under an identity that does not have the necessary privileges to manage the database's schema.

  1. Azure Service Bus has its own dead-lettering too, but if you're using Rebus it will still use a separate (ordinary) queue for poison messages. In my experience, this is actually preferable, because then you can handle your dead-lettered messages in another bus instance, and e.g. route them somewhere else, save them in a database, send emails, etc.

Don't know if that is the case with Oracle, though – is it easy in Oracle to return a message from the exception queue back to its source queue?

agerchev commented 5 years ago

  1. The fast one uses buffered messages (not persistent to disk) but cannot use LOB type. it must be RAW and it is restricted in length. I have tested it and is roughly 2 times faster than the normal persisted messages. I was thinking to include some simple script to automatically create the queues, but there can be configured a lot more things and must be used the specific api for that purpose: https://docs.oracle.com/database/121/ARPLS/d_aqadm.htm#ARPLS65307

  2. I understand the "The Rebus way" about message retries, but in our project it is a lot more simpler to use the retry mechanisms for message delay and retry. If i am not mistaken, there is not way in rebus, that i can retry a message exactly the specified number of times, because it will be running in a cluster of nodes (with exact copy the program that is pumping and processing messages). I see that the exceptions are buffered in memory. I may be wrong, that's why i am asking :) That's why we want to use the AQ implementation. We use the retry+delay mechanism, and when the message processing fails, it will automatically be moved to the exception queue. We process the message from the exception queue(just like normal queue) by sending some notification (email .... ) to the admins, then move the message to another queue (DL Queue). When the problem is fixed, we manually move the message from DL Queue to the normal queue for processing.

  3. Is there a way, to configure rebus instance to use more than one input queue or i have to configure more rebus instances in one process? currently, messages are dequeued (including for exceptions queue) with a blocking call to

    dequeue_options.WAIT := 2; -- wait in seconds if there is no message in the queue.

    DBMS_AQ.DEQUEUE( QUEUE_NAME => 'QUEUE_NAME', DEQUEUE_OPTIONS => dequeue_options, MESSAGE_PROPERTIES => message_properties, PAYLOAD => payload, MSGID => message_handle);

We have to configure the backoffStrategy with zero backoff time, so we minimize the latency. The call is blocking(for 2 seconds i saw that you are using that value for the azure/rabbitmq version. ) so there will be no waste of CPU cycles :)

agerchev commented 5 years ago

@mookid8000 I've made a version with the AQ, you can use it if you want.