search

rabbitmq / rabbitmq-server

Open source RabbitMQ: core server and tier 1 (built-in) plugins
https://www.rabbitmq.com/
Other
11.84k stars 3.9k forks source link

Shovel: make it possible to configure `x-stream-offset` (a consumer x-argument) #8508

Open michaelklishin opened 1 year ago

michaelklishin commented 1 year ago

For Shovels to be able to move messages between streams (for now, over AMQP 0-9-1), they must be able to let the user override the x-stream-offset consumer argument.

Without it, currently Shovels will only move the messages published after the Shovel is set up.

ngbrown commented 7 months ago

I have two RabbitMQ clusters, in different locations, and they both have the same data in streams. Because the offset isn't copied between them, I can't switch freely between clusters from the client because the offset's don't match.

I would want something for RabbitMQ streams like Kafka MirrorMaker that can completely replicate a stream, and catch up the mirror up when temporarily disconnected.

michaelklishin commented 7 months ago

Shovel does not replicate, it moves contents.

Commercial RabbitMQ will add DR support for streams reasonably soon. But that’s not related to stream support in Shovel, which is an open source feature currently very vaguely defined and in need of a scope and design.

On Tue, 5 Dec 2023 at 15:55, Nathan Brown @.***> wrote:

I have two RabbitMQ clusters, in different locations, and they both have the same data in streams. Because the offset isn't copied between them, I can't switch freely between clusters from the client because the offset's don't match.

I would want something for RabbitMQ streams like Kafka MirrorMaker https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 that can completely replicate a stream, and catch up the mirror up when temporarily disconnected.

— Reply to this email directly, view it on GitHub https://github.com/rabbitmq/rabbitmq-server/issues/8508#issuecomment-1841593435, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAIQSHQPBC2ESORTGCZ3LYH6CPPAVCNFSM6AAAAAAZARRYI2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBRGU4TGNBTGU . You are receiving this because you were assigned.Message ID: @.***>

-- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.