petersilva
commented
3 years ago thought experiment picking ugly numbers for illustrative purposes:
If we have a maximum message size, then experience with WMO indicates the average message size will converge to about half of the maximum... (when it was 14K, we observed a message size of 7K on our links... but there was segmentation in the story... who knows how it will change in the new methods.) If we pick 8M as a new max (8MB is the max I am hearing in Aviation circles), then imagine 4M becomes the average. A message without payload is about If we are transferring 100 messages per second. without payload, assume a message is 512 bytes. so at any given message transfer rate, the size difference is 8190:1 ... so if we can support 100 messages/second coming in over the message protocol, then we should get 0.02 messages/second with embedded messages. Or you need the message protocol to run 8000 times faster, or some combination.
It becomes a question of how good are brokers at doing actual data transfer (as opposed to switching.) The traditional answer is: not very good. But I fully get that people intuitively think it should be faster, and in many cases it should be, but in a lot of other common cases, I expect it to turn out to be counter-productive.
golfvert
eliot-christian
amilan17
The original encodings which motivated this work always used links, and never embedded content in the messages, as that has been used for many years in Canada. This was considered an option for future implementation, but had so far been deemed unnecessary. Discussion in the ET-CTS in 2019 led to the "Content" field being added to the message format to include the actual message data in the MQP message literally. This was originally intended to help with long latency, low bandwidth satellite links, a fairly special case where it does make alot of sense.
Some feel that embedding the content within the message is a universal accellerator. Generally speaking, we strive to have transfer protocols that encourage as much parallelization as possible as that maximizes the opportunities for speedups. It needs to be appreciated that a queue is a serialization of a kind, and that it slows down posting of files, because each one needs to be read from disk to be included in the message, and the content will then be in the message flow. With pub/sub, a subscriber performs client-side filtering to exclude messages they are not interested in. The larger messages will make the receipt of each message slower, and increase queueing, when compared to a message stream with no embedding. It is also well understood that:
If we want to avoid segmentation, then we need to establish a maximum message size for data to be embedded. That size has to be supported by protocols efficiently.
So we have one case: If every message in an publication stream is of interest to the subscriber, and the messages are all "small" then a completely embedded stream will perform better than third party downloads, assuming the entire stream bandwidth can be supported. Another option to reduce client-side filtering is to have separate channels for warnings. These are guaranteed not to be busy, and we could simply elect to embed everything sent on such channels.
On the other hand, it is well understood if an advertised stream includes numerical weather prediction outputs, satellite and RADAR imagery, as well as observations, that pushing large data through MQP channels will be awful. Such channels will universally experience queueing, and the warnings may be stuck behind them.
Using standard protocols such as https and sftp allows us to achieve much higher total transfer bandwidth rates, without taxing the MQP brokers doing something they are not intended for (large scale data transfers.)
Another benefit of out of band, non-mqp transfers is that they use protocols which are in very wide use with many opportunities for use of Content distribution networks, web accelleration appliances, etc... for which the analogue would be to have third parties implementing brokers, a far more cumbersome prospect.
summary. Embedding: