chrisguttandin / extendable-media-recorder-wav-encoder

A Wave file encoder for the extendable-media-recorder package.
MIT License
40 stars 4 forks source link

Pre-compiled JavaScript version #392

Closed guest271314 closed 3 years ago

guest271314 commented 3 years ago

Is there a pre-compiled JavaScript version of extendable-media-recorder-wav-encoder? Or, how to build this as JavaScript for use in the browser?

chrisguttandin commented 3 years ago

Hi @guest271314,

I'm not sure if I understand the question. This package contains a pre-compiled version of the extendable-media-recorder-wav-encoder-worker.

https://github.com/chrisguttandin/extendable-media-recorder-wav-encoder/blob/master/src/worker/worker.ts

Does that solve your question?

guest271314 commented 3 years ago

Hi @guest271314,

I'm not sure if I understand the question.

I am not familiar with TypeScript.

I am able to create WAV files using https://github.com/guest271314/audioInputToWav.

This package contains a pre-compiled version of the extendable-media-recorder-wav-encoder-worker.

https://github.com/chrisguttandin/extendable-media-recorder-wav-encoder/blob/master/src/worker/worker.ts

Does that solve your question?

The un-minified version would be helpful.

https://github.com/kbumsik/opus-media-recorder#limitations at list item 2.

Because audio/wav is not designed for streaming, when mimeType is audio/wav, each dataavailabe events produces a complete and separated .wav file that cannot be concatenated together

It is possible to concatenate WAV files.

I am most interested in how you handle dataavailable, e.g., start(1000) during a recording, for this use case https://github.com/WebAudio/web-audio-api-v2/issues/118.

This is possible using Media Transform API https://plnkr.co/edit/sEupuQ2pYepXQcV6, however that API is only supported at latest Chrome and Chromium.

chrisguttandin commented 3 years ago

The worker.ts file in this repo is a pre-compiled version of the extendable-media-recorder-wav-encoder-worker package.

If start() is called with a timeslice argument the header will be written as if the WAV file would be as large as possible. The code for doing that is here: https://github.com/chrisguttandin/extendable-media-recorder-wav-encoder-worker/blob/master/src/functions/encode-header.ts#L9

It is of course not technically correct since there is no way to know how large the file is going to be when the header needs to be written. It was a pragmatic decision. This was the related issue: https://github.com/chrisguttandin/extendable-media-recorder/issues/263

It would be entirely possible to create an extendable-media-recorder-pcm-encoder which encodes PCM data without any header.

guest271314 commented 3 years ago

I un-minified the worker.ts file. I still do not know how to use the file.

Does the code expect input from a MediaStream, following W3C MediaRecorder?

It would be entirely possible to create an extendable-media-recorder-pcm-encoder which encodes PCM data without any header.

The requirement of the linked Web Audio API feature request is essentially a pass-through audio node, where isince they are on Safari, AudioWorklet and the Media Transform API are not implemented; to read the Float32Array data during playback/recording, for editing purposes.

For this to work as intended (on Safari) we probably need to use input from MediaStream or MediaElementAudioSourceNode.

We do not need WAV headers at all.

guest271314 commented 3 years ago

extendable-media-recorder-wav-encoder.js.zip

chrisguttandin commented 3 years ago

Sorry, I just realized that it is not pointed out anywhere that this packages is meant to be used with the extendable-media-recorder.

extendable-media-recorder was made to allow custom (audio) codecs to be used with the standard MediaRecorder API. The only codec that I implemented so far is this one. The readme contains a little snippet which shows how it can be used.

guest271314 commented 3 years ago

That is why I filed this bug. Effectively requesting a Web demo (GitHub pages; jsfiddle; plnkr; etc.) page of this extension.

chrisguttandin commented 3 years ago

I created a little example on StackBlitz which builds upon the code snippet from the readme.

https://stackblitz.com/edit/js-acfyyc?file=index.js

guest271314 commented 3 years ago

Using start(1) eventually crashes the tab on Chromium Screenshot_2021-04-13_20-41-04

chrisguttandin commented 3 years ago

Yes, I would recommend to not log every single Blob when requesting one every millisecond. I slightly modified the example to only log all the chunks at the end.

https://stackblitz.com/edit/js-68ne9q?file=index.js

guest271314 commented 3 years ago

The use case is "real-time" capture and processing of the WAV, not awaiting completion of all data at the end of a range, for the Web Audio API, in form of raw PCM, something like https://plnkr.co/edit/bK1BfoSgjFUDwkIV?preview where instead of just logging, the underlying float value can be modified then streamed to other inputs and outputs.

I could not find a way to turn off StackBlitz console, which could be an issue. console at plnkr was.

I did download extenadable-media-recorder-bundle.js and extendable-media-recorder-wav-encoder-bundle.js from unpkg. I am not sure how to download @babel/runtime as a bundle - to test outside of StackBlitz.

chrisguttandin commented 3 years ago

I often use jspm to create prototype which JS packages which normally would need to used with a bundler.

This is the StackBlitz example from above as a self contained HTML document which uses jspm.

<!DOCTYPE html>
<html>
    <head>
        <meta charset="UTF-8" />
        <base href="/" />
    </head>
    <body>
        <button id="record" disabled>record</button>
        <button id="stop" disabled>stop</button>
        <script type="module">
            import { MediaRecorder, register } from "https://jspm.dev/extendable-media-recorder";
            import { connect } from "https://jspm.dev/extendable-media-recorder-wav-encoder";

            connect()
                .then(register)
                .then(() => navigator.mediaDevices.getUserMedia({ audio: true }))
                .then(stream => {
                    const recordButton = document.getElementById("record");
                    const stopButton = document.getElementById("stop");

                    recordButton.disabled = false;

                    const mediaRecoder = new MediaRecorder(stream, { mimeType: "audio/wav" });
                    const chunks = [];

                    recordButton.addEventListener("click", () => {
                    recordButton.disabled = true;
                    stopButton.disabled = false;

                    mediaRecoder.ondataavailable = ({ data }) => {
                        chunks.push(data);
                    };
                    mediaRecoder.start(1);
                    });

                    stopButton.addEventListener("click", () => {
                    stopButton.disabled = true;

                    mediaRecoder.ondataavailable = ({ data }) => {
                        chunks.push(data);

                        if (mediaRecoder.state === "inactive") {
                        recordButton.disabled = false;

                        console.log("finished recording", chunks.slice(0));
                        chunks.length = 0;
                        }
                    };
                    mediaRecoder.stop();
                    });
                });
        </script>
    </body>
</html>
guest271314 commented 3 years ago

BaseAudioContext.decodeAudioData() is throwing when passing the encoded WAV. Eventually the tab crashes https://plnkr.co/edit/yc2YsCwVytNq09le?preview. Screenshot_2021-04-18_12-13-44

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8" />
    <base href="/" />
  </head>
  <body>
    <button id="record" disabled>record</button>
    <button id="stop" disabled>stop</button>
    <audio controls autoplay></audio>
    <script type="module">
      import {
        MediaRecorder,
        register,
      } from 'https://jspm.dev/extendable-media-recorder';
      import { connect } from 'https://jspm.dev/extendable-media-recorder-wav-encoder';
      const audio = document.querySelector('audio');
      const generator = new MediaStreamTrackGenerator({ kind:'audio' });
      const {writable} = generator;
      const writer = writable.getWriter();
      audio.srcObject = new MediaStream([generator]);
      let base_time = 0;
      const ac = new AudioContext();
      connect()
        .then(register)
        .then(() => {

          const msd = new MediaStreamAudioDestinationNode(ac);
          const osc = new OscillatorNode(ac);
          osc.connect(msd);
          return msd.stream;
        })
        .then((stream) => {
          const recordButton = document.getElementById('record');
          const stopButton = document.getElementById('stop');

          recordButton.disabled = false;

          const mediaRecoder = new MediaRecorder(stream, {
            mimeType: 'audio/wav',
          });

          recordButton.addEventListener('click', () => {
            recordButton.disabled = true;
            stopButton.disabled = false;

            mediaRecoder.ondataavailable = async ({ data }) => {
              console.log(data);
              const buffer = await ac.decodeAudioData(await data.arrayBuffer());
              const frame = new AudioFrame({
                timestamp: base_time * 1000000,
                buffer: buffer,
              });
              await writer.write(frame);
              base_time += buffer.duration;
            };
            mediaRecoder.start(1);
          });

          stopButton.addEventListener('click', () => {
            stopButton.disabled = true;
            mediaRecoder.stop();
          });
        });
    </script>
  </body>
</html>
chrisguttandin commented 3 years ago

I guess this is expected. At least for all blobs after the first one. Only the first one contains a wav header. Does it work if you copy the header from the first and prepend it to all consecutive blobs?

guest271314 commented 3 years ago

Instead of copying header to subsequent Blobs I will try removing header from first Blob and parsing to PCM.

guest271314 commented 3 years ago

Still freezing the tab. AudioWorklet is the underlying API used, correct?

chrisguttandin commented 3 years ago

It depends on the browser. In Chrome it will use the native MediaRecorder. In Firefox it will indeed use an AudioWorklet and in Safari it will be the good old ScriptProcessorNode.

guest271314 commented 3 years ago

The Web Audio API issue use case was for Safari, where AudioWorklet is not supported, yet and already uses ScriptPreocessorNode. The original feature request was for a pass-through type node.

I am not sure why the tab is freezing at Chromium when running the code.

Is video/x-matroska;codecs=pcm code used at MediaRecorder constructor?

Thanks for your time.

chrisguttandin commented 3 years ago

'audio/webm;codecs=pcm' is the mimeType that extendable-media-recorder uses to get the PCM data in Chrome.

https://github.com/chrisguttandin/extendable-media-recorder/blob/master/src/factories/webm-pcm-media-recorder.ts#L13