'text' streams don't get written / cannot be read back - with Gist

[josiahbryan]

TL;DR

I can write custom 'text' packets, but they don't seem to get written to the file and I can't read them back. See the gist for specific working example of the problem:

Gist showing this issue: https://gist.github.com/josiahbryan/70a551d5afa949e57a223f8216dd03e1

Setup

I adapted the mp4 example to add a second stream using the 'text' codec ("Raw Subtitle Codec", codec_id 94210), and added a simple data packet that is written out after every video frame. See gist above, line 37 for stream addition, and lines 83-97 where the data packet is generated:

const dataPacket = beamcoder.packet({
    pts:  lastPts || 0,
    data: Buffer.from("Hello, world!")
});

for (const pkt of [ dataPacket ]) {
    pkt.duration = 1;
    pkt.stream_index = dataStream.index;
    pkt.pts = pkt.pts * 90000/25;
    pkt.dts = pkt.dts * 90000/25;
    await mux.writeFrame(pkt);
}

Then, just to test it out, I read back the file with demuxer = await beamcoder.demuxer(...file...) and then find the stream for the data with let dataStreamInput = demuxer.streams.find(x => x.codecpar.codec_type === 'data').

That works great! I get a dataStreamInput variable, with an index of 1.

However, here is the problem:

Problem

When I try to read in the packets from the video file and find a packet with a stream_index matching the dataStreamInput.index found, there are no packets in the file with that index.

See the gist for specific example - lines 120-128. You'll notice I even tried finding ANY non-video packets - none were found.

Can you offer any insight into what's going wrong here? Why does the stream exist in the output file (even VLC/ffplay show the stream is indexed), but no packets seem to be written anywhere for that stream?

Am I reading it back wrong? Or did I write the packets out wrong?

Thanks so much!

[scriptorian]

It's not obvious but your problem is that you need to set a packet size when you create the dataPacket eg:

const dataBuf = Buffer.from("Hello, world!");
const dataPacket = beamcoder.packet({
    pts:  lastPts || 0,
    data: dataBuf,
        size: dataBuf.byteLength
});

Your pts values for the subtitles look a bit strange but I'll leave that with you!

[josiahbryan]

Ahh I'll give that a try!

Regarding PTS (should I open a different issue)? How DOES one create / calculate PTS when generating frames?? LoL I've tried everything! Googling gives tons of different answers too. For this example problem, I just used the mp4 example and use the PTS it generates, but even that is just a simple "i + 100". Input on how to generate proper PTS is VERY welcome :)

[scriptorian]

I just used i+100 for both video and subtitle. The difficulty with your existing code is that the video encoder doesn't always deliver a packet (they get buffered up) and then will deliver them in optimised decode order, not presentation order.

[josiahbryan]

Just tested (as I'm sure you know) that worked perfectly! Didn't know I had to set size - thanks!!

Ahhh...ummm that makes sense, I should have realised that regarding PTS, packets, and the decoder - you do mention that about packets in the docs, my bad. I didn't think that code through. That should be easy to rectify from a code perspective. Thanks!

For anyone reading this later then, this issue is fixed - just set size and you're golden ✅

I'll open a different issue detailing my more general PTS question with a Gist. Thanks!

[josiahbryan]

I spoke a BIT too soon ... yes, I read back data that I write now. However, when I write 13 bytes, I get back 90 bytes.

TL;DR:

Question: Is the minimum buffer length 90 bytes? If so, then I can adapt my code (just add a 4-byte Uint32LE header to indicate the actual length or something), but if it's a bug or I'm doing something wrong then I'd like to do it the right way :) Thanks!

Details

Write:

const dataBuf = Buffer.from("Hello, world!");
const dataPacket = beamcoder.packet({
    pts:  frame.pts, // frame is the video frame above
    data: dataBuf,
    size: dataBuf.byteLength,
});

console.log(` Writing data pts ${dataPacket.pts * 90000/25} with ${dataBuf.byteLength} bytes`);

Read:

if (packet && packet.stream_index === dataStreamInput.index) {
    console.log(`Data: ${packet.pts} [${packet.data.byteLength} bytes]: ${packet.data.toString()}`)
}

With that code above, I see on my console:

Writing data pts 536400 with 13 bytes

Then, later:

Data: 536400 [90 bytes]: Hello, world!��

I've updated the Gist (https://gist.github.com/josiahbryan/70a551d5afa949e57a223f8216dd03e1) with those changes, so you can run it and see for yourself.

As stated above, Question: Is the minimum buffer length 90 bytes? If so, then I can adapt my code (just add a 4-byte Uint32LE header to indicate the actual length or something), but if it's a bug or I'm doing something wrong then I'd like to do it the right way :) Thanks!

[scriptorian]

This is expected - FFmpeg adds AV_INPUT_BUFFER_PADDING_SIZE bytes to the buffers it creates. However the returned packets have a size parameter which is set to the correct number of bytes, matching the data packets that were written.

[josiahbryan]

Oh lovely! The .size param - didn't think to check it, assumed the .data was "right" lol. My bad. That's great, saves me from having to add my own size value to the start of my buffers. Thanks!

For anyone that cares, here's how I fixed it based on that advice:

const dataBuf = packet.data.subarray(0, packet.size);
console.log(`Data: ${packet.pts} [${dataBuf.byteLength} bytes]: ${dataBuf.toString()}`)

Works 💯 - thanks!!