Question: Does reactor-rabbitmq-streams have a future?

[andricDu]

Hi!

Just curious because I wasn't able to find any public roadmaps, is reactor-rabbitmq-streams still something that pivotal will push forward with, and will it end up being opensource?

[MarcialRosales]

Hi Andric, reactor-rabbitmq-streams is still under internal evaluation process to decide when and how it will be open sourced and supported. Once the decision is made, the roadmap will be public. I can advance that we are working on the next release which addresses :

• resiliency (e.g. performing a rolling upgrade shall not impact live publisher/consumer streams; no message loss or automatic recovery of connections/subscriptions, and automatic redefinition of topology (queues, exchanges, bindings, policies)).
• observability along with grafana dashboards.

May I ask you if you have used it yet ? or whether you intend to use?

[andricDu]

Hi Marcial,

Yes we are currently developing a new solution that is using reactor-rabbitmq-streams.

To give you some background, we are are developing an open source solution that allows the pipelining and execution of computational workflows in the field of cancer genomics. We are building this for a large international project (https://www.icgc-argo.org/).

We would like to stick with reactor-rabbitmq-streams as it perfectly matches our use case for building out a reactive/event driven architecture.

While I understand this is still under internal review for open sourcing, what is the provisional license that this has been released under for use? Are we in some sort of breach of license for using it as a maven dependency?

Thanks!

[MarcialRosales]

Hi Andric, it will be licensed as Apache 2.0. You will not incur in any license breach by using the maven dependency.

Thanks for sharing the background !

[andricDu]

Thanks @MarcialRosales for the continued updates to the project. I'll close this issue as you answered this question.

[MarcialRosales]

Hi @andricDu , I am using this issue you raised a while back to contact you to get your feedback on some changes that are coming on the Reactive RabbitMQ streams. I appreciate if you could share with us your thoughts/opinions.

1. Do you create your messaging topologies declaratively via this library or do you manage them externally?

2. Are you using today quorum queues or planning to use them? I am planning on supporting quorum queues and being able to declaratively create them. Will you be interested?

3. Are you aware of a new queue type called stream which is due to be officially released on 3.9 around mid-June? will you be interested in using them via this library? i.e. as you know what is behind a stream is totally transparent to the producer/consumer. Except for the logic that declares the topology, assuming you use the library to declare topologies.

Btw, you might be interested in taking the latest version (v.0.0.8) as it addresses an issue whereby producers do not recover well after the connection drops and is recovered. However, this release requires you to add an additional maven repository.

I hope everything is going well at your side.

[andricDu]

Hey @MarcialRosales,

1. We create the messaging typologies declaratively using the library. We find it extremely ergonomic.
2. We do not currently use quorum queues but it looks very likely that we will use them in the near future. We would be very interested in an extension to how topologies are already declared to include the quarom queue feature of rabbit.
3. I am not aware of this new queue type though it does sound interesting.

Thanks for the heads up about the new version. I'll investigate.

I will say this library is extremely high value to us. The ergonomics of the topology declaration to working with Fluxes of transactions, everything is top notch. We really appreciate the work.

One thing that I feel I should mention is that we take somewhat manual control over the loading of AVRO schemas into the SchemaManager. Our use case has us loading schemas in from both the classpath and filesystem/registry on app startup. Our needs see that messages on the wire (rabbitmq queues) having their schemas statically declared and enforced (classpath) while dynamic code execution and its output (we run on top of GraalVM) is enforced from schemas found on a registry/filesystem.

[MarcialRosales]

Thank you so much @andricDu for your feedback. We are very pleased to hear that it is so valuable to you.

With regards Quorum Queues, we highly recommend customers switch to them if you are using Mirrored queues. If you are not using mirrored queues then it does not make sense. Quorum Queues are more resilient to failures than mirrored queues. However, although they share most of the common features available in the classic queue, there are some features which are not supported or partially. You need to check your case here https://www.rabbitmq.com/quorum-queues.html#feature-comparison

I am very interested to fully understand your avro schema requirements. If you do not mind, I would like to ask you:

a)

Our use case has us loading schemas in from both the classpath and filesystem/registry on app startup.

do you use a schema registry server or just a local filesystem? When you use the term "registry/filesystem" I am not entirely sure if you mean filesystem-based registry or central registry and filesystem-based registry. I believe you mean the former. If that is the case, I always thought on users like you who did not want to use a central registry but statically load the schemas from filesystem.

b)

Our needs see that messages on the wire (rabbitmq queues) having their schemas statically declared and enforced (classpath)

do you mean that consumer applications (schema readers) only load the schemas from classpath? If that is the case, this means that your build pipeline ensures that consumer apps always have the latest libraries (avro classes) especially for new avro messages or that schema changes are backward compatible.

c)

dynamic code execution and its output (we run on top of GraalVM) is enforced from schemas found on a registry/filesystem.

i believe you are referring to the producer applications here and these ones do have their schemas in their file system because they are the schema's owner.

If my understanding is right, this should be easily provided by the library. The library could load from classpath and also from filesystem, taking precedence filesystem over classpath. In other words, if there is a conflict, we would always take the schema from the filesystem. I don't think it makes sense to make it configurable.

[andricDu]

Hey @MarcialRosales

We have two classes of AVRO schemas in our framework. Those that are provided by the framework at build time, and those that are set at runtime and can be found via registry OR filesystem. We want ALL the schemas available via content type.

What this means is we do some reflection so we can manipulate the schemaManager so we can bring in our AVRO schemas into the content type storage from the classpath.

Example found in a configuration class:

  @SneakyThrows
  private void classPathToContentTypeStorage(String contentType, Schema schema) {
    val schemaManager = this.reactiveRabbit.schemaManager();

    Method registerMethod =
        SchemaManager.class.getDeclaredMethod(
            "importRegisteredSchema", String.class, Schema.class, Integer.class);
    registerMethod.setAccessible(true);

    GraphLogger.info(
        nodeProperties,
        "Loading GraphRun AVRO Schema from classpath into registry with ContentType.");
    registerMethod.invoke(schemaManager, contentType, schema, null);

    val graphRunSchemaObj = schemaManager.fetchReadSchemaByContentType(contentType);
    if (graphRunSchemaObj.isError()) {
      GraphLogger.error(
          nodeProperties,
          "Cannot load %s schema by Content Type, shutting down.",
          schema.getFullName());
      SpringApplication.exit(context, () -> 1);
    } else {
      GraphLogger.info(
          nodeProperties,
          "Successfully loaded schema %s from classpath.",
          graphRunSchemaObj.getFullName());
      GraphLogger.info(
          nodeProperties, "\n\033[32m" + graphRunSchemaObj.toString(true) + "\033[39m");
    }
  }

[smaugfm]

It's been a while since the last comment. @MarcialRosales, what is the current status of the library: is it production-ready, what is the future, etc.? Also, adding a dependency as in here currently does not work. Only adding packagecloud.io repo does, is it intended? Thanks!

[MarcialRosales]

@smaugfm Most likely this year the library will be open sourced and most likely will incorporate Stream queue type. The latter is absolutely essential for some types of event processing use cases. Thanks for your interest.

[smaugfm]

That sounds great!

Currently, I opted to use Reactor Rabbit or the project I work on, which is kind of bare-bones and minimalistic. I also have experience with Spring Cloud Stream. This library is fine, but they made some weird design choices where it may seem that it uses and promotes a reactive approach at the first glance but if you really want to double down on that you are forced to use just the raw Flux with events from your message broker without any of the error handling, retries, etc. Actually, without any of the things people usually choose Spring libraries for.

And then I discovered this lib which is really a perfect blend of the reactive approach and the "Spring way". So, really looking forward to the updates, and thank you for the work!