Open anoadragon453 opened 1 year ago
Not sure if this is the right discussion venue for this, but I'll put my feedback here anyway ;)
The event_id field indicates the message ID, which is unique across all rooms. This is analogous to the MessageId field in the current draft.
Except that the current draft's messageId consists of two parts: a domain-unique message id and the domain.
The type field states the type of the message. This tells the receiver what kind of content (in content) to expect. This is not quite analogous to the current draft, but is similar to a combination of the "PartSemantics" and "Disposition" mechanisms we currently define.
Except that the part semantics / disposition are per part and thus allow a message to contain arbitrary multiple contents (via singleUnit/processAll semantics). It also allows for much more flexibility as e.g. a file can be marked for inline display (looking at the MSC1767 it seems to be impossible to create a new type that fallbacks to what type=m.image does, because m.image is not a content block, the behavior of m.file does not ask for the file to be displayed inline and there is no disposition field to indicate that)
When a client receives a message, it first looks at the type of the message. If it is a type that the client recognise [...] then they can continue to look for the content blocks that type defines should be rendered. If the client does not recognise the type, then it should proceed to look at the content blocks that the message contains and attempt to associate them with a message type it does know.
So essentially, the type doesn't really matter, because the receiving client will look into the content blocks nonetheless and pick any that it does support (e.g. in your example, a client that understands com.large-vendor.location content block will render that, others will render the text - even if type was any
)
The semantics are very similar to the current content format draft. The main difference is that the receiving implementation is in full control of which content blocks it decides to process and in which order. This was done deliberately to allow for the fallback semantics, while still hinting at what content block should be rendered via the defined message type.
The current draft specifies that the order of entries in the chooseOne semantics indicate the senders preference over the different parts, but that recipient is allowed to pick any. This means the receiving entity remains in full control, but the sender's order gives essentially the feature you provide through the message type.
In summary: the current draft seems to be more powerful than the MSC1767 syntax and be more targeted to the usecase of MIMI.
When asked for the format, I would suggest to go for something like EXI rather than JSON/CBOR/MessagePack/...: It is well extensible, has support for binary data, is generally efficient in encoding, has a reasonable plain text representation (can be converted from/to XML lossless) and also has a logic for string lookup tables, the latter could become relevant, as with the multipart system, we might see the same longer strings (like URLs) appear at multiple parts within the message.
The event_id field indicates the message ID, which is unique across all rooms. This is analogous to the MessageId field in the current draft.
Except that the current draft's messageId consists of two parts: a domain-unique message id and the domain.
Yes, I was confused between the ID of an event at the transport layer (what the "Event ID" in my proposal referred to), and a message ID inside the MLS application message. Depending on how the transport layer develops, these two IDs may end up being the same value, but for now that's up in the air (blocked on the transport side).
In Matrix the Event ID is derived from a hash of some of the fields in the message, which include the domain name. Thus the domain name need not be included directly in the ID.
Regardless, that change should be discussed separately. I'll update the post to mention the difference as pointed out.
The type field states the type of the message. This tells the receiver what kind of content (in content) to expect. This is not quite analogous to the current draft, but is similar to a combination of the "PartSemantics" and "Disposition" mechanisms we currently define.
It also allows for much more flexibility as e.g. a file can be marked for inline display
Just to make sure i understand this correctly, would this translate to e.g. an ODT file being rendered inside the timeline of the room? That might be a case of the receiving client deciding whether to do so for an m.file
based on the mimetype
?
(looking at the MSC1767 it seems to be impossible to create a new type that fallbacks to what type=m.image does, because m.image is not a content block, the behavior of m.file does not ask for the file to be displayed inline and there is no disposition field to indicate that)
If all the necessary content blocks of an m.image
type are present (m.file
, m.text
), then ideally you would be able to fall back. The problem I see is that there is already an m.file
there being used for the video. One solution to solve this is to use an m.image_details
content block, with an m.file
nested inside of it. Then you would not have a conflict between two top-level m.file
content blocks. Still, that conflict feels like a valid limitation. The current draft gets around this by nesting everything.
When a client receives a message, it first looks at the type of the message. If it is a type that the client recognise [...] then they can continue to look for the content blocks that type defines should be rendered. If the client does not recognise the type, then it should proceed to look at the content blocks that the message contains and attempt to associate them with a message type it does know.
So essentially, the type doesn't really matter, because the receiving client will look into the content blocks nonetheless and pick any that it does support (e.g. in your example, a client that understands com.large-vendor.location content block will render that, others will render the text - even if type was
any
)
Types do help in the case where you have two or separate events which share the same set of required content blocks, but that you'd want to render differently. For instance, the types m.video
, m.image
, m.audio
and m.file
all only require the m.file
and m.text
content blocks. The type
informs the receiving client what the sender intended to have rendered. Of course if a client didn't understand m.audio
, it could fall back to m.file
(just displaying a file download) or even just m.text
.
Now m.file
does have a (optional) mimetype
field, and a receiving client could use this instead to determine whether to show a video player/audio player/etc. This is essentially the route that the current draft proposal takes. But it does mean the client needs to use heuristics such as checking against a list of mimetypes in order to know what content to render, rather than an explicit m.video
vs. m.audio
type. We'd need to make similar exceptions (with different fields to check) for other groups of types that require the same content blocks.
It's also important to note that its up to the receiving client to decide the known types they would and would not fall back to. For instance, I suspect clients will not want to fallback to a poll type.
The semantics are very similar to the current content format draft. The main difference is that the receiving implementation is in full control of which content blocks it decides to process and in which order. This was done deliberately to allow for the fallback semantics, while still hinting at what content block should be rendered via the defined message type.
The current draft specifies that the order of entries in the chooseOne semantics indicate the senders preference over the different parts, but that recipient is allowed to pick any. This means the receiving entity remains in full control, but the sender's order gives essentially the feature you provide through the message type.
Good point! Looking at this again, I don't think there's any inherit benefit over the current draft in terms of priority of fallbacks in an event.
When asked for the format, I would suggest to go for something like EXI rather than JSON/CBOR/MessagePack/...: It is well extensible, has support for binary data, is generally efficient in encoding, has a reasonable plain text representation (can be converted from/to XML lossless) and also has a logic for string lookup tables, the latter could become relevant, as with the multipart system, we might see the same longer strings (like URLs) appear at multiple parts within the message.
Interesting, thank you. I've not heard of this format before - I'll need to take a look!
Just to make sure i understand this correctly, would this translate to e.g. an ODT file being rendered inside the timeline of the room? That might be a case of the receiving client deciding whether to do so for an m.file based on the mimetype?
Types do help in the case where you have two or separate events which share the same set of required content blocks, but that you'd want to render differently. For instance, the types m.video, m.image, m.audio and m.file all only require the m.file and m.text content blocks. The type informs the receiving client what the sender intended to have rendered. Of course if a client didn't understand m.audio, it could fall back to m.file (just displaying a file download) or even just m.text.
Now m.file does have a (optional) mimetype field, and a receiving client could use this instead to determine whether to show a video player/audio player/etc. This is essentially the route that the current draft proposal takes. But it does mean the client needs to use heuristics such as checking against a list of mimetypes in order to know what content to render, rather than an explicit m.video vs. m.audio type. We'd need to make similar exceptions (with different fields to check) for other groups of types that require the same content blocks.
I fail to follow your reasoning here:
m.image
and mimetype to image/emf
if the receiving client does not support the Enhanced Metafile format. So the type (just as any of the values for disposition) is no more than a hint to the recipient client how the sender envisioned to transfer the file, but any reasonable client should probably fallback to have support to download the file, so that the user can open it with an external application (except maybe in cases where we explicitly want to make sure that the file is not downloaded/copied, thinking of messages that destroy themselves after a timeout).m.file
(in the example of an ODT), what makes m.image
then different from m.file
? In both cases it is up to the receiving client to decide what disposition to use (although with m.image
there is a strong hint from the sender that they envision the file to be displayed inline).audio
/video
/image
match exactly the toplevel MIME type, so if you're looking for a "heuristic" when to behave as if it was m.audio
/m.video
/m.image
, that would be very trivial.
As a whole, the semantics laid out in the adopted content format draft look sound to me. At this point I'd like to propose a syntax for how messages could concretely look.
"Matrix's Extensible Events" is one possible syntax. It is a content format we've been developing as part of the Matrix Foundation C.I.C., and are actively looking to move the Matrix network over to in the medium term. The format is flexible, with re-usable and nestable pieces and supports clear fallback semantics.
Below is a quick overview of Matrix's Extensible Events and some examples mapping the current draft's content format semantics to it. I will be using JSON to represent these structures, but one could easily use any message format that supports similar semantics to JSON (such as CBOR (RFC 8949), MessagePack, Protobuf, etc.).
Overview of Matrix's Extensible Events
A typical "extensible event" looks like this:
sender
field indicates what user has sent this message. This is analogous to thesenderLeafIndex
derived field in the current draft.event_id
field indicates the message ID, which is unique across all rooms. This is analogous to theMessageId
field in the current draft, except that it uses a hash of the contents to generate a unique ID, rather than "UUID + owning provider domain".room_id
field indicates the room, or MLS Group, that this message was sent in. This is analogous to themlsGroupId
derived field in the current draft.type
field states the type of the message. This tells the receiver what kind of content (incontent
) to expect. This is not quite analogous to the current draft, but is similar to a combination of the "PartSemantics" and "Disposition" mechanisms we currently define.content
map contains zero or more "content blocks" that contain: the data to render, potential relations to other messages, and so forth.Content Blocks
A "content block" is identified by an identifier and some renderable data. For example, this is a message of type
m.message
with am.text
content block:Content blocks are simply any top-level key in the
content
field. They can have any value type (that is also legal in a message generally: string, integer, etc).The
m.text
content block is defined as an ordered array of MIME types, optional language codes and strings that together represent a single marked-up blob of text. Themimetype
is optional and defaults totext/plain;charset=utf-8
. Them.message
type requires am.text
content block, and defines no optional content blocks.A useful property of the
m.message
message type and them.text
content block being defined separately is that we can reuse the latter for other message types. We can also use it when attempting to specify a rendering fallback for message types some clients may not understand.Rendering & Fallback
When a client receives a message, it first looks at the
type
of the message. If it is a type that the client recognises (i.e. it supports a version of MIMI that defines the type, or a custom vendor type) then they can continue to look for the content blocks that type defines should be rendered.If the client does not recognise the type, then it should proceed to look at the content blocks that the message contains and attempt to associate them with a message type it does know.
That culminates our fallback semantics. For example, given the following message with a custom type:
This message has a
com.large-vendor.location
content block in it, which contains a location defined by ageo:
URI (RFC 5870). The name of the content block is namespaced to the vendor, allowing other vendors/users to experiment with similar content block designs without clients misinterpreting one for another.Imagine you have a room with two clients in it; Large Vendor App (LVA) and Other Large Vendor App (OLVA). LVA will recognise a
com.large-vendor.location-pin
message type and render it accordingly. While OLVA may not recognise that type. In the case of OLVA, it should fall back to the other available content blocks and attempt to render a type that it does know. The priority ordering of types to try is up to the implementation.Other Large Vendor App knows that
m.message
requires a singlem.text
content block, so it will render this message as anm.message
with the text "My current location is geo:51.5008,0.1247;u=35". These fallback semantics allow vendors, hobbiests and anyone else to send messages while remaining confident that other clients in the room will still be able to understand their communication.Nesting
Along with other similarities to a
NestablePart
, content blocks can be nested inside each other.The
m.image
message type defines am.file
content block, optionalm.thumbnail
,m.caption
and anm.text
block (for fallback) among other blocks:As
m.text
is quite a versatile content block, we can use it in many areas, even within other content blocks. The same is done form.file
, which is being used bym.thumbnail
to point to a smaller resolution version of the image.These content block should not be used just anywhere however - message types must define required or optional content blocks that clients should expect when rendering them. There is another kind of content block that can be used at the top-level of any message type though, called mixins.
Mixin Content Blocks
A "mixin" is a class of content block that can be added to any message type. An example of a mixin is a
m.automated
content block - for which the presence of such denotes that the message was sent by an automation rather than by a user directly. Slack, for example, uses this information to display anAPP
label next to bot messages.Another type of mixin is
m.thread
- informing the client that a given message took place in a thread. This saves message types from all needing to definem.thread
as an optional content block. Instead, it can be applied to any message type.References
There are several scenarios where a message needs to reference another message. Replies, reactions and deletions are a few examples. A
m.references
content block is an array of zero or more "references"; each containing the type of reference and the ID of another message.For instance, to delete a message a client would send the following:
Receiving clients would interpret this as a past message with ID
message_id@domain
should be removed from the timeline.Using this system, we can translate all of the currently defined functionality (reactions, expiring, VoIP, etc.) to concrete representations.
We've already proposed a number of definitions for various message types (listed in the introduction of MSC1767), which can serve as a starting point.
The semantics are very similar to the current content format draft. The main difference is that the receiving implementation is in full control of which content blocks it decides to process and in which order. This was done deliberately to allow for the fallback semantics, while still hinting at what content block should be rendered via the defined message type.