connect response doesn't indicate support for Enhanced RTMP

[zenomt]

the client can indicate support for Enhanced RTMP by including a fourCcList member of the connect command's argument/command object (though see #10 about the name of that member).

however, clients can't tell if servers support Enhanced RTMP. while an unaware server can simply forward Enhanced RTMP messages as they come in, this won't have the desired effect for clients subscribing to a stream after its publish has started. in particular, servers unaware of Enhanced RTMP won't have special treatment of PacketSequenceStart, PacketTypeMetadata, and PacketTypeMPEG2TSSequenceStart messages, remembering them and sending them to new subscribers before the coded frames.

a publishing client should be able to tell that the server will or won't perform the special sequence/metadata processing for subscribers, and subscribing clients should be able to tell that they may not receive the sequence/metadata messages for enhanced messages. this could be accomplished by echoing the fourCcList (and/or others as appropriate) back in the connect transaction's _result Info Object.

[zenomt]

this should probably be a separate Issue, but on the subject of the fourCcList property: the spec says "supported by the client". given the meaning of the videoCodecs and audioCodecs bitmap properties, one would assume the fourCcList indicates the codecs that the client is capable of receiving, vs ones that it can or might send. if that's the case, that distinction should be made clear.

[veovera]

this should probably be a separate Issue, but on the subject of the fourCcList property: the spec says "supported by the client". given the meaning of the videoCodecs and audioCodecs bitmap properties, one would assume the fourCcList indicates the codecs that the client is capable of receiving, vs ones that it can or might send. if that's the case, that distinction should be made clear.

• Really good feedback, thank you!
• I don't believe indicating which codecs the client can receive vs send is in the original spec?
• fourCcList adopts current behavior while indicating what codecs are supported in the list
• That said that does not mean we should not add more info in the connect command in the next version of the enhancement to indicate clients capabilities to send vs receive any particular fourCc item.
• And yes, agreed this should be a separate issue. Perhaps if you get a chance open an additional issue and provide more color to the behavior?

[veovera]

the client can indicate support for Enhanced RTMP by including a fourCcList member of the connect command's argument/command object (though see #10 about the name of that member).

however, clients can't tell if servers support Enhanced RTMP. while an unaware server can simply forward Enhanced RTMP messages as they come in, this won't have the desired effect for clients subscribing to a stream after its publish has started. in particular, servers unaware of Enhanced RTMP won't have special treatment of PacketSequenceStart, PacketTypeMetadata, and PacketTypeMPEG2TSSequenceStart messages, remembering them and sending them to new subscribers before the coded frames.

a publishing client should be able to tell that the server will or won't perform the special sequence/metadata processing for subscribers, and subscribing clients should be able to tell that they may not receive the sequence/metadata messages for enhanced messages. this could be accomplished by echoing the fourCcList (and/or others as appropriate) back in the connect transaction's _result Info Object.

Nice catch especially since this is so nuanced. To echo the 'fourCcList' is of course straight forward and probably would be a good addition to the spec. BTW: Shouldn't this already potentially be an issue? AVCPacketType has same challenges, although by now most servers support AVC codec id and the AVC sequence header... this is still not guaranteed since today the RTMP spec does not require a server to echo it's set of support codecs?

[zenomt]

in retrospect i don't think echoing the fourCcList is the right answer, since that might imply additional unintended semantics (like which codecs the server is capable of handling). but some indication that the server understands Enhanced RTMP and will do something appropriate especially with the new enhanced start/init/metadata messages i think would be useful for feature & codec negotiation.

[zenomt]

I don't believe indicating which codecs the client can receive vs send is in the original spec?

the original spec is missing a lot of things (i wish i had had the time way back then for a full rewrite instead of just editorial cleanup). :)

the behavior of Adobe Media Server (at least) is to use the videoCodecs and audioCodecs bitmaps as selectors/filters for what can be sent to the client. AMS won't send video or audio messages to a client unless the codec being used has its bit set in the bitmap. AMS doesn't care which codecs a client can send, since AMS can record & forward all (traditional) codecs. also, many clients have asymmetric capabilities (for example, Flash Player can receive & play back AAC, but i don't think it can encode & send AAC) and this asymmetry isn't signaled in the connect/_result handshake.

[zenomt]

And yes, agreed this should be a separate issue. Perhaps if you get a chance open an additional issue and provide more color to the behavior?

i opened #14

[zenomt]

the "Enhancing NetConnection connect Command" section of the latest spec appears to address this Issue, and technically this issue could be closed. however, it's a bit hard to tell with the current wording of the last paragraph of that section:

As you can see, the client declares to the server what enhancements it supports. The server responds with a command, either _result or _error, to indicate whether the response is a result or an error. During the response, the server provides some properties within an Object as one of the parameters. This is where the server needs to state its support for E-RTMP. The server SHOULD state its support via attributes such as videoFourCcInfoMap, capsEx, and similar properties.

i think it'd be valuable for a future revision to reword this section, and perhaps have a separate paragraph for the server's response that explicitly lists that an Enhanced RTMP server should be sending one or more of the enhanced connect properties back to the client in the NetConnection.Connect.Success _result, and how a client should be interpreting such an enhanced response from the server.

also, specifically for capsEx, one could reasonably assume that a missing capsEx means 0, but it'd be good to call that out explicitly.

[zenomt]

meanwhile, i've modified my tcserver to answer thus:

connection onStatus: {
    "audioFourCcInfoMap": {
        "*": 4
    },
    "capsEx": 0,
    "code": "NetConnection.Connect.Success",
    "connectionID": "5c052b8e63ffcf1a12f89b6657ab3bb1b55c08a1321489c139fe4b066394cceb",
    "description": "you connected!",
    "fourCcList": [
        "*"
    ],
    "level": "status",
    "objectEncoding": 0,
    "serverFingerprint": "264886da312d618a82dd3ef57e5d3900126a95eade7c713d551e084e3132d75a",
    "serverInfo": "welcome to REDACTED",
    "videoFourCcInfoMap": {
        "*": 4
    }
}

to say that it can forward any video or audio codecs, but does not support Multitrack, nor will it ever send a NetConnection.Connect.ReconnectRequest to the client.

[veovera]

Thank you for the thoughtful feedback! I'll take a closer look at the wording and make the necessary adjustments. For now, I'll keep this issue open until we've had a chance to polish the wording.