[QUESTION] how to extract frames given a `start` and `end` time?

[jrafaaael]

I want to extract frames based on a range of seconds: lets say I have a 20-second video and I want to get all the frames in 2 to 7 seconds range Right now, I'm usign the following code (from here) to extract each frame of a given mp4 video and it works fine but I would like to extract only required frames instead of all of them I tried to use the seek method in #onSamples private method of MP4Demuxer class but this doesn't work. My laptop got freezed for some reason. Without the seek LOC all works fine again (I get all frames without any problem)

// mp4-demuxer.js
import MP4Box, { DataStream } from 'mp4box';

// Wraps an MP4Box File as a WritableStream underlying sink.
class MP4FileSink {
    #setStatus = null;
    #file = null;
    #offset = 0;

    constructor(file, setStatus) {
        this.#file = file;
        this.#setStatus = setStatus;
    }

    write(chunk) {
        // MP4Box.js requires buffers to be ArrayBuffers, but we have a Uint8Array.
        const buffer = new ArrayBuffer(chunk.byteLength);
        new Uint8Array(buffer).set(chunk);

        // Inform MP4Box where in the file this chunk is from.
        buffer.fileStart = this.#offset;
        this.#offset += buffer.byteLength;

        // Append chunk.
        this.#setStatus('fetch', (this.#offset / 1024 ** 2).toFixed(1) + ' MiB');
        this.#file.appendBuffer(buffer);
    }

    close() {
        this.#setStatus('fetch', 'Done');
        this.#file.flush();
    }
}

// Demuxes the first video track of an MP4 file using MP4Box, calling
// `onConfig()` and `onChunk()` with appropriate WebCodecs objects.
export class MP4Demuxer {
    #onConfig = null;
    #onChunk = null;
    #setStatus = null;
    #file = null;

    constructor(uri, { onConfig, onChunk, setStatus }) {
        this.#onConfig = onConfig;
        this.#onChunk = onChunk;
        this.#setStatus = setStatus;

        // Configure an MP4Box File for demuxing.
        this.#file = MP4Box.createFile();
        this.#file.onError = (error) => setStatus('demux', error);
        this.#file.onReady = this.#onReady.bind(this);
        this.#file.onSamples = this.#onSamples.bind(this);

        // Fetch the file and pipe the data through.
        const fileSink = new MP4FileSink(this.#file, setStatus);
        fetch(uri).then((response) => {
            // highWaterMark should be large enough for smooth streaming, but lower is
            // better for memory usage.
            response.body.pipeTo(new WritableStream(fileSink, { highWaterMark: 2 }));
        });
    }

    // Get the appropriate `description` for a specific track. Assumes that the
    // track is H.264, H.265, VP8, VP9, or AV1.
    #description(track) {
        const trak = this.#file.getTrackById(track.id);
        for (const entry of trak.mdia.minf.stbl.stsd.entries) {
            const box = entry.avcC || entry.hvcC || entry.vpcC || entry.av1C;
            if (box) {
                const stream = new DataStream(undefined, 0, DataStream.BIG_ENDIAN);
                box.write(stream);
                return new Uint8Array(stream.buffer, 8); // Remove the box header.
            }
        }
        throw new Error('avcC, hvcC, vpcC, or av1C box not found');
    }

    #onReady(info) {
        this.#setStatus('demux', 'Ready');
        const track = info.videoTracks[0];

        // Generate and emit an appropriate VideoDecoderConfig.
        this.#onConfig({
            // Browser doesn't support parsing full vp8 codec (eg: `vp08.00.41.08`),
            // they only support `vp8`.
            codec: track.codec.startsWith('vp08') ? 'vp8' : track.codec,
            codedHeight: track.video.height,
            codedWidth: track.video.width,
            description: this.#description(track)
        });

        // Start demuxing.
        this.#file.setExtractionOptions(track.id, null, { nbSamples: Infinity });
        this.#file.start();
    }

    #onSamples(track_id, ref, samples) {
        // Generate and emit an EncodedVideoChunk for each demuxed sample.
        for (const sample of samples) {
            this.#onChunk(
                new EncodedVideoChunk({
                    type: sample.is_sync ? 'key' : 'delta',
                    timestamp: (1e6 * sample.cts) / sample.timescale,
                    duration: (1e6 * sample.duration) / sample.timescale,
                    data: sample.data
                })
            );
        }
    }
}

[hughfenghen]

You can refer to the link below. Although not entirely equivalent, it closely aligns with your requirements.

DEMO： https://hughfenghen.github.io/WebAV/demo/1_4-mp4-previewer Code： https://github.com/hughfenghen/WebAV/blob/56ab3c240b3347e195184c291676dd9119dea608/packages/av-cliper/src/mp4-utils/mp4-previewer.ts#L85

[Secretmapper]

Hey @hughfenghen, thanks for the link/repo, definitely interesting!

From my understanding you are downloading the entire stream here correct? (Building the videoSamples on init.) Would you happen to know how to only fetch a range of data (say I only wanted to extract from range 5-7s and make sure to only download the minimum necessary range)?

[hughfenghen]

@Secretmapper If you only want to download a portion of the video data, perhaps you should explore some DASH or HLS protocols.