is one chunk equals to one segment ?

[RavikumarTulugu]

I have a question on the streamer interface, does onData callback is given 'complete' clusters, like one full cluster ? is one media cluster encapsulated inside one chunk ? This is needed for live streaming.

[RavikumarTulugu]

@Vanilagy can you pls respond ? Thanks

[RavikumarTulugu]

I spent some time looking at the code and it seems the chunk is not a webm element it is just a block of raw bytes. the onData routine has to still re-assemble chunks into webm elements and send them to the live stream endpoint. we need 2 callbacks like onHeader and onCluster which are invoked with initialization segment and a new cluster. Please correct me if my observations are wrong. Thanks

[Vanilagy]

This is interesting to know! Streaming currently works even without a clustered approach by just adding more and more bytes to the MediaSource, as you can see it done in the demo. My assumption was that the same could then just be done over a network which would result in live streaming. Since I've never actually needed WebM livestreaming anywhere, I'm not too familiar with it. Can you point me to a source which states that streaming data needs to happen at the Cluster boundary?

It shouldn't be too hard to add additional callbacks to the StreamTarget for these purposes once it's clear what needs to be done. However, I'm wondering exactly what your usecase is. It seems to me that taking the exact code I wrote in the streaming demo, but sending the muxed bytes over a network, should still result in valid playback.

[RavikumarTulugu]

The dash webm requires 2 file types,the first file should be the initialization segment with EBML header and the associated track information, the rest of the files need to contain media with in clusters. The data approach would work for mediasource but won't work over network. a callback which can be invoked for every new cluster would simplify lot of work for the library user. This will help the live streaming case. if the onData callback handles cluster boundaries then it is not required.

[Vanilagy]

You're right, I'll get to this as soon as I can. Any ways you suggest I can test/debug this functionality? Is there an easy way for me to set up DASH-based streaming locally?

[RavikumarTulugu]

Unfortunately no !!! if you make the onData to honor the webm media cluster boundaries , that would address the dash requirement. nothing else is needed. one more nice to have callback is initCallback which can be invoked with the parsed initialization segment ( EBML header ).

[Vanilagy]

Yes I need to see how I do it, if I change the behavior of onData (would need a major version bump, which is not a problem btw) or I just add additional callbacks like onHeader or onCluster. I'll get back to you when there is something :) Just need to see how I debug/test it locally to see if it indeed works with DASH.

[RavikumarTulugu]

Hi any update on this ?? Thanks

[Vanilagy]

@RavikumarTulugu Yes, I just released a new version (4.0.0) which changes the signature of StreamTarget and adds the callbacks you asked for. Please, test the hell out of it! Since I personally have no use for this feature, you'll need to tell me if my solution fixes your problem.

I added documentation to the README which includes these new callbacks.

[RavikumarTulugu]

Thank you so much !! will try it out and let you know.

[RavikumarTulugu]

@Vanilagy The webm muxer swallows the input frames but never invokes any of the callbacks. no error messages are seen. Here is my configuration. I am using av1 codec with opus.

The webm-muxer.js file is taken from this commit c8f4a6c as the latest throws a parse error at the start of the file while loading in browser.

Please take a look

let options = { 
    target : new WebMMuxer.StreamTarget (
               ( data, position ) => {}, //onData
               ( data, position ) => {
                 webmClusterCount++;
                 handleWebmInitCluster ( data );  
                 return;
               }, //onHeader.
               ( data, position ) => {
                 webmClusterCount++;
                 handleWebmMediaCluster ( data );  
                 return;
               }, //onCluster.
               { chunked : true, chunkSize : 1024*4 }), 
            video : { 
               codec : 'V_' + videoCodecClass ( vconfig.codec ).toUpperCase(), //av1
               width : vconfig.width, 
               height : vconfig.height, 
               bitrate: vconfig.bitrate 
             },  
            audio : { 
               codec : 'A_' + aconfig.codec.toUpperCase(),  //opus 
               numberOfChannels : aconfig.numberOfChannels, 
               sampleRate : aconfig.sampleRate, 
               bitrate : aconfig.bitrate
             },  
            streaming : true,
            type : 'webm',
            firstTimestampBehavior : 'offset'
      };  
  livecastMuxer = new WebMMuxer.Muxer ( options );

[Vanilagy]

You're not using TypeScript, right? The signature of StreamTarget has changed. It should look like this:

let options = { 
    target: new WebMMuxer.StreamTarget({
        onData: (data, position) => {}, // onData
        onHeader: (data, position) => {
            webmClusterCount++;
            handleWebmInitCluster(data);
        }, // onHeader
        onCluster: (data, position, timestamp) => {
            webmClusterCount++;
            handleWebmMediaCluster(data);
        }, // onCluster
        chunked: true,
        chunkSize: 1024 * 4
    }), 
    video: { 
        codec: 'V_' + videoCodecClass(vconfig.codec).toUpperCase(), // av1
        width: vconfig.width, 
        height: vconfig.height, 
        bitrate: vconfig.bitrate 
    },  
    audio: { 
        codec: 'A_' + aconfig.codec.toUpperCase(),  // opus 
        numberOfChannels: aconfig.numberOfChannels, 
        sampleRate: aconfig.sampleRate, 
        bitrate: aconfig.bitrate
    },  
    streaming: true,
    type: 'webm',
    firstTimestampBehavior: 'offset'
};  
livecastMuxer = new WebMMuxer.Muxer(options);

[RavikumarTulugu]

I adjusted the syntax properly, still no luck !! Can you pls check.

let options = { 
    target : new WebMMuxer.StreamTarget ({
               onData : ( data, position ) => {
                 webmClusterCount++;
                 return;
               },  
               onHeader : ( data, position ) => {
                 webmClusterCount++;
                 handleWebmInitCluster ( data );  
                 return;
               },  
               onCluster : ( data, position ) => {
                 webmClusterCount++;
                 handleWebmMediaCluster ( data );  
                 return;
               },  
               chunked : true, 
               chunkSize : 1024*4
             }), 
            video : { 
               codec : 'V_' + videoCodecClass ( vconfig.codec ).toUpperCase(), 
               width : vconfig.width, 
               height : vconfig.height, 
               bitrate: vconfig.bitrate 
             },  
            audio : { 
               codec : 'A_' + aconfig.codec.toUpperCase(), 
               numberOfChannels : aconfig.numberOfChannels, 
               sampleRate : aconfig.sampleRate, 
               bitrate : aconfig.bitrate
             },  
            streaming : true,
            type : 'webm',
            firstTimestampBehavior : 'offset'
      };  
  livecastMuxer = new WebMMuxer.Muxer ( options );

[RavikumarTulugu]

some error during finalize

webm-muxer.js:566 Uncaught Error: Internal error: Monotonicity violation.
    at ChunkedStreamTargetWriter.flushChunks_fn (webm-muxer.js:566:17)
    at ChunkedStreamTargetWriter.writeDataIntoChunks_fn (webm-muxer.js:516:59)
    at ChunkedStreamTargetWriter.write (webm-muxer.js:483:75)
    at ChunkedStreamTargetWriter.writeEBMLVarInt (webm-muxer.js:211:12)
    at ChunkedStreamTargetWriter.writeEBML (webm-muxer.js:241:18)
    at ChunkedStreamTargetWriter.writeEBML (webm-muxer.js:221:16)
    at ChunkedStreamTargetWriter.writeEBML (webm-muxer.js:236:16)
    at Muxer.createSegment_fn (webm-muxer.js:959:33)
    at Muxer.writeBlock_fn (webm-muxer.js:1112:63)
    at Muxer.finalize (webm-muxer.js:756:59)

[Vanilagy]

Wait, can you tell me what exactly is going wrong? Is it just the error you posted, or something else?

[Vanilagy]

Can you try without chunked: true?

[RavikumarTulugu]

ok let me try with chunked : false, None of the callbacks were getting called when the video frames are fed to the muxer, eventually the app crashes with memory exhaustion. some times i see that error in finalize in the logs , it could be a usage issue with the api.

[RavikumarTulugu]

@Vanilagy a little bit of progress , setting chunked : false, triggers callbacks. onData callback is called once right after the muxer is started and another time during finalize. wonder , whether this is needed at all.
onHeader callback is called when the muxer.finalize is called. onCluster callback is still never called.

[Vanilagy]

Strange, I'll look into this tomorrow. How many frames are you feeding into the muxer, with which timestamps?

[RavikumarTulugu]

I am sorry , I didnt understand the 'timestamps' part, i am feeding like 24 frames per second. though i varies around 19-22.

[Vanilagy]

I see, that's strange. Do you think you can paste some of your code in here so I can try to reproduce it?

[RavikumarTulugu]

we have a pipeline architecture where frames and audio buffers are passed around via message channels. below is my muxer settings and initialization.

let options = { 
    target : new WebMMuxer.StreamTarget ({
               onData : ( data, position ) => {
                 webmClusterCount++;
                 return;
               },  
               onHeader : ( data, position ) => {
                 webmClusterCount++;
                 handleWebmInitCluster ( data );  
                 return;
               },  
               onCluster : ( data, position ) => {
                 webmClusterCount++;
                 handleWebmMediaCluster ( data );  
                 return;
               },  
               chunked : true, 
               chunkSize : 1024*4
             }), 
            video : { 
               codec : 'V_' + videoCodecClass ( vconfig.codec ).toUpperCase(), 
               width : vconfig.width, 
               height : vconfig.height, 
               bitrate: vconfig.bitrate 
             },  
            audio : { 
               codec : 'A_' + aconfig.codec.toUpperCase(), 
               numberOfChannels : aconfig.numberOfChannels, 
               sampleRate : aconfig.sampleRate, 
               bitrate : aconfig.bitrate
             },  
            streaming : true,
            type : 'webm',
            firstTimestampBehavior : 'offset'
      };  
  livecastMuxer = new WebMMuxer.Muxer ( options );

This is how i feed the muxer

(async () => {
    while ( 1 ) {
      while ( casterOn && livecastAudioInputQueue.length ) {
        let chunk = livecastAudioInputQueue.shift ();
        if ( chunk ) {
          livecastMuxer.addAudioChunk ( chunk.data, chunk.meta );
        }
      }

      while ( casterOn && livecastVideoInputQueue.length ) {
        let chunk = livecastVideoInputQueue.shift ();
        if ( chunk ) {
          livecastMuxer.addVideoChunk ( chunk.data, chunk.meta );
        }
      }
      await sleep ( workerSleepTime );
    }
  }) ();

[Vanilagy]

Sorry for not getting back to you back then, I'll close this issue for now since it's stale.