search

bloomberg / blazingmq

A modern high-performance open source message queuing system
https://bloomberg.github.io/blazingmq/
Apache License 2.0
2.53k stars 131 forks source link

Protocol specification. #28

Open kafkiansky opened 1 year ago

kafkiansky commented 1 year ago

Is there an existing proposal for this?

Is your feature request related to a problem?

I want to implement a client to your broker.

Describe the solution you'd like

Will there be a protocol specification added?

Alternatives you considered

No response

willhoy commented 1 year ago

+1 Until there's something better you'd have to derive it from the bmqp module dox and the control message schema: https://github.com/bloomberg/blazingmq/blob/93e6426d474a4293d38244a9ada6dc8cbafe5711/src/groups/bmq/bmqp/bmqp_ctrlmsg.xsd

https://github.com/bloomberg/blazingmq/blob/93e6426d474a4293d38244a9ada6dc8cbafe5711/src/groups/bmq/bmqp/bmqp_protocol.h

These give you a good sense of the structure of the messages, but less so whats a valid response to a particular communication to the broker.

quarter-note commented 1 year ago

Hi @kafkiansky, thanks for your interest in the product! What @willhoy has pointed out are basically the two files which capture the wire protocol containing low level structs and control messages (encoded requests/responses).

However, these files don't explain the order of exchange of these structs/messages/requests/responses, as well as any contract between client and broker around that exchange.

We have not documented this yet. However, we have a sample producer and consumer implementation which does not use client library. Instead, it simply uses socket APIs to connect/read/write to the broker and sends/receives messages to/from the broker.
For example, one can see that after connecting to the broker, client library sends NegotiationMessage as the first message (see comment Step 3. Send the negotiation message to the Broker in the producer file), etc.

While this is not a formal spec, it should be enough to get you started to have a rudimentary working prototype.

I would also like to add that in order to provide a good experience, the C++ and Java client libraries are quite stateful. Asychronous nature of the APIs also adds complexity in the implementation. Generally speaking, implementing a batteries-included library is a non-trivial effort. However, we are happy to help you out if you decide to implement one.

Out of curiosity, what language do you have in mind for the client library?

kafkiansky commented 1 year ago

Hi, @quarter-note. Thanks for the explanation. I would like to implement the libraries in rust, go and php. I will try to use the information from those files you gave, or will watch tcpdump as a last resort.

quarter-note commented 1 year ago

Ok great, I will try to write a high level overview of the message flow and some other relevant details.

kafkiansky commented 1 year ago

Thank you so much for your help.

quarter-note commented 1 year ago

Hi @kafkiansky Just wanted to mention that we have started working on documenting the protocol. It's WIP but it can be found here. Hoping to wrap it up in the coming weeks.

kafkiansky commented 1 year ago

Hi, @quarter-note. Great, thank you. I've already started reverse-engineering your protocol using a Java library, but it would be much easier with a formal specification.

quarter-note commented 1 year ago

Hi @kafkiansky I have added some more details in the document. It's still WIP, but you may find newly added info useful.

kafkiansky commented 1 year ago

Hi @quarter-note. Yes, I am following the changes and going to continue development this weekend. Thank you for your work!

kafkiansky commented 1 year ago

@quarter-note, I didn't see any information about what order you write the bytes in: little endian or big endian?

willhoy commented 1 year ago

I would be very surprised if the bytes were not written in 'Network byte order' using the equivalent of hton*/ntoh* style functions at the low level.

kafkiansky commented 1 year ago

So it's BigEndian. Thanks for the hint, @willhoy.

quarter-note commented 1 year ago

Yes, it's big endian (or network byte order).

Some additional notes -- in Java, we have two wrapper classes ByteBufferInputStream and ByteBufferOutputStream which under the hood use java.nio.ByteBuffer, which take care of flipping the bytes when reading/writing integers, doubles, etc.

In C++, we use a BigEndian type from one of our helper libraries, which takes care of swapping the bytes seamlessly. As an example, here, we use BigEndianUint32 type to represent an unsigned int32 to capture the fragment and length fields of an EventHeader. Under the hood, various BigEndian classes use hton*/ntoh* routines as @willhoy mentioned.

quarter-note commented 1 year ago

Added some info in the doc.

kafkiansky commented 1 year ago

Hi, @quarter-note. While developing an SDK, I'm trying to run an OpenQueue request, but I'm getting an error {\"rId\":1,\"status\":{\"category\":\"E_UNKNOWN\",\"code\":-1,\"message\":-1,\"message\":\"Domain file '{\/etc\/local\/b/bmq\\\\/domains\\\\/myapp.json' doesn't exist\"}}}. What is that file? I couldn't find anything about such files in the documentation. The queue URI is bmq://myapp/queue.

Perhaps the bmqbrkr logs will be useful:

bmqbrkr_1  | 19AUG2023_18:08:49.300 (139933851313728) INFO mqba_domainresolver.cpp:190 Error reading the domain config file [domain: 'myapp', rc: -1, output:
bmqbrkr_1  | Domain file '/etc/local/bmq/domains/myapp.json' doesn't exist
bmqbrkr_1  | 19AUG2023_18:08:49.300 (139933851313728) WARN mqbblp_queuesessionmanager.cpp:114 #CLIENT_OPENQUEUE_FAILURE local:27885.0@172.19.0.1:55432: Error while qualifying domain: [ category = E_UNKNOWN code = -1 message = "Domain file '/etc/local/bmq/domains/myapp.json' doesn't exist" ], request: [ rId = 1 choice = [ openQueue = [ handleParameters = [ uri = "bmq://myapp/queue" qId = 0 subIdInfo = NULL flags = 12 readCount = 0 writeCount = 1 adminCount = 0 ] ] ] ]
bmqbrkr_1  | 19AUG2023_18:08:49.300 (139933851313728) INFO mqba_clientsession.cpp:349 local:27885.0@172.19.0.1:55432: Sending openQueue failure response: [ rId = 1 choice = [ status = [ category = E_UNKNOWN code = -1 message = "Domain file '/etc/local/bmq/domains/myapp.json' doesn't exist" ] ] ]
bmqbrkr_1  | 19AUG2023_18:08:49.303 (139933074130496) INFO mqbnet_tcpsessionfactory.cpp:734 TCPSessionFactory 'TCPInterface': OnClose channel [session: 'local:27885.0@172.19.0.1:55432', channel: '172.19.0.1:55432#0x7f44c001c768', 0 active channels, status: [ Category = SUCCESS ]]
678098 commented 1 year ago

Hi @kafkiansky

You try to use domain myapp for your queue. We don't have this domain in our sample domain configuration. We store domain configuration for each domain separately in its own json-file in the specified domains folder.

From the path /etc/local/bmq/domains/myapp.json probably you are launching bmqbrkr in a docker setup. In this setup we read domain configurations from this folder (for single node): https://github.com/bloomberg/blazingmq/tree/main/docker/single-node/config/domains

This folder is connected to the container as a volume here: https://github.com/bloomberg/blazingmq/blob/f4a4f55ae74caf24995ca540a0fbf70a33fbd9c3/docker/single-node/docker-compose.yaml#L8

So to fix this problem, you can either:

  1. Use existing domain name, for priority mode for example, bmq.test.mem.priority instead of myapp (check the file bmq.test.mem.priority.json in the folder)
  2. Or make your own domain configuration, by placing myapp.json to the domains folder. To start, you can simply copy and rename any existing domain configuration, this for example: https://github.com/bloomberg/blazingmq/blob/main/docker/single-node/config/domains/bmq.test.mem.priority.json
678098 commented 1 year ago

Also some info about domains: https://bloomberg.github.io/blazingmq/docs/introduction/concepts/#domain

kafkiansky commented 1 year ago

@678098, thank you for the explanation.

quarter-note commented 1 year ago

@kafkiansky I have raised PR https://github.com/bloomberg/blazingmq/pull/95. Feel free to take a look and comment.

kafkiansky commented 9 months ago

Hello, @quarter-note. In the process of developing sdk on go, I am focusing on java sdk. There I noticed different packing of properties depending on the flag https://github.com/bloomberg/blazingmq-sdk-java/blob/main/bmq-sdk/src/main/java/com/bloomberg/bmq/impl/infr/proto/MessagePropertiesImpl.java#L416. What does this mean? Does this need to be supported or can only "newStyleProperties" be used?

kafkiansky commented 9 months ago

Hello, @678098. Maybe you know the answer to the question above?

678098 commented 9 months ago

Hello, @678098. Maybe you know the answer to the question above?

Hi @kafkiansky! Only the new style message properties are needed. They are called "message properties V2" or "extended message properties". Some details about them are in this comment: https://github.com/bloomberg/blazingmq/issues/73#issuecomment-1668697386

kafkiansky commented 9 months ago

Hello, @678098. Maybe you know the answer to the question above?

Hi @kafkiansky! Only the new style message properties are needed. They are called "message properties V2" or "extended message properties". Some details about them are in this comment: #73 (comment)

Thank you, @678098.