search

dank074 / Discord-video-stream

Experiment for making video streaming work for discord selfbots.
185 stars 35 forks source link

H.265, VP9, AV1 support #70

Open longnguyen2004 opened 8 months ago

longnguyen2004 commented 8 months ago

H.265 also uses Annex B, VP9 and AV1 also uses IVF, so maybe we can reuse code here... PRs made:

longnguyen2004 commented 8 months ago

Notes for people who want to follow along:

  1. H265 is only available on mobile, is it limited based on the user agent in the WebSocket connection?
dank074 commented 8 months ago

Could be sent as an option in the SELECT_PROTOCOL message. Maybe the mobile app sends it but desktop app doesn't

longnguyen2004 commented 8 months ago

Found a way to enable H265 on desktop. You need to have an NVIDIA card with H265 encoding support (not sure if this is a hard requirement or it can gracefully fall back to software encoding). Then enable the following experiment

image

After which, the following is sent by the client

{
    "d": {
        "address": "(redacted)",
        "codecs": [
            {
                "name": "opus",
                "payload_type": 120,
                "priority": 1000,
                "type": "audio"
            },
            {
                "decode": true,
                "encode": false,
                "name": "AV1",
                "payload_type": 101,
                "priority": 1000,
                "rtx_payload_type": 102,
                "type": "video"
            },
            {
                "decode": true,
                "encode": true,
                "name": "H265",
                "payload_type": 103,
                "priority": 2000,
                "rtx_payload_type": 104,
                "type": "video"
            },
            {
                "decode": true,
                "encode": true,
                "name": "H264",
                "payload_type": 105,
                "priority": 3000,
                "rtx_payload_type": 106,
                "type": "video"
            },
            {
                "decode": true,
                "encode": true,
                "name": "VP8",
                "payload_type": 107,
                "priority": 4000,
                "rtx_payload_type": 108,
                "type": "video"
            },
            {
                "decode": true,
                "encode": true,
                "name": "VP9",
                "payload_type": 109,
                "priority": 5000,
                "rtx_payload_type": 110,
                "type": "video"
            }
        ],
        "data": {
            "address": "(redacted)",
            "mode": "aead_aes256_gcm_rtpsize",
            "port": 48542
        },
        "experiments": [
            "fixed_keyframe_interval",
            "bandwidth_estimation/trendline-window-duration-3750,robust-estimator/trendline-window-duration-3750,robust-estimator"
        ],
        "mode": "aead_aes256_gcm_rtpsize",
        "port": 48542,
        "protocol": "udp",
        "rtc_connection_id": "7dc912bc-22fd-4ea2-8265-658ac2b40f6a"
    },
    "op": 1
}
longnguyen2004 commented 8 months ago

@dank074 The comment here said that the packet type is STAP-A, but after reading and re-reading the spec, I determined that the actual packet type is Single NAL Unit Packet, due to the lack of the NALU size in the packet data, which STAP-A requires. Can you confirm it?

https://github.com/dank074/Discord-video-stream/blob/85101666816808eb77936a2687e5030b3393f58e/src/client/packet/VideoPacketizerH264.ts#L83-L100

dank074 commented 8 months ago

@dank074 The comment here said that the packet type is STAP-A, but after reading and re-reading the spec, I determined that the actual packet type is Single NAL Unit Packet, due to the lack of the NALU size in the packet data, which STAP-A requires. Can you confirm it?

https://github.com/dank074/Discord-video-stream/blob/85101666816808eb77936a2687e5030b3393f58e/src/client/packet/VideoPacketizerH264.ts#L83-L100

You're actually right. Not sure why I wrote that 🤔