Support audio controls in Notebook Renderer

[DonJayamanne]

Request: Add support for Audio Control in Notebook Renderer.

• Originally posted here https://github.com/microsoft/vscode/issues/32540
• Custom implementation for Notebook Renderers https://github.com/microsoft/vscode-notebook-renderers/pull/33

We (Jupyter Extension) can accept the PR, however this would mean the Jupyter extension now has a custom renderer for HTML. And I'm not too keen on owning rendering all HTML content in Notebook renderers (I know VS Code has custom code to handle a lot of the HTML content, links, etc).

Suggestion

• Please can we re-visit this issue
• Is it possible to provide some way for the Notebook to override the preferred renderer based on the content?
  • I.e. I'd prefer to always use the built in renderer for HTML
  • However if VS Code will not support wav natively, then we can override the preferred renderer and use the one we ship.

@rebornix /cc

[mcmar]

This entire issue could be obviated if you just ship a build with ffmpeg. That's probably the preferable solution to all this. See: https://github.com/microsoft/vscode/issues/32540 https://github.com/microsoft/vscode/issues/54097 https://github.com/microsoft/vscode/issues/66050

[robinhad]

I think a good solution will be to build VSCode with some kind of "libre" ffmpeg - with only wav, ogg, vorbis for audio and AV1 for video. It will facilitate the most common use cases for audio/video processing and will not bloat VSCode too much.

[rebornix]

Assigning to @deepak1556 and @mjbvz as this is more about audio control in webview.

[mcmar]

I just want to point out that the whole reason we're here today is because ffmpeg inclusion was previously stonewalled multiple times and we were told to create an HTML workaround. So here we are with a workaround for WAV audio because a codec technically isn't required for WAV audio. Now people are complaining that the workaround is complex. That's fair, but I just don't want to get stuck in-between two solutions where we end up doing nothing. I hope we can all commit to picking between either the HTML-based WAV audio workaround or adding ffmpeg to the build. It's really not fair to users to stonewall this issue forever.

[DonJayamanne]

Now people are complaining that the workaround is complex.

I apologise for making it sound if the solution implemented by @robinhad is complex. That was not my intention at all.

[mcmar]

@DonJayamanne Sorry, I should rephrase that. Just want to make sure that we do something here. Options so far: 1) ffmpeg 2) slim compiled ffmpeg (see here) 3) HTML-based workaround

[rchiodo]

In case anybody is looking for the reason why this is important for jupyter customers, here's our original issue: https://github.com/microsoft/vscode-jupyter/issues/1012

[robinhad]

Hello! Is there any news regarding this issue?

[sporkl]

I haven't tried this, but it looks like it's possible to add the complete ffmpeg libraries oneself as a workaround. https://github.com/VSCodium/vscodium/issues/152#issuecomment-492050247 .

[RandomFractals]

well, some devs using Py and JS notebooks for ML might want to have those audio and video capabilities, including webcam access.

I tested this in JS notebooks and def. would like to play them in vscode some day, including youtube tutorials, and do some extensions with webcam ...

[leoauri]

I would really appreciate this being implemented, as it is blocking me from using the jupyter extension. Cheers

[RandomFractals]

I'd be happy to provide more code samples and test JS notebooks from my end for vscode team to review and come up with a good solution.

I don't think what's been discussed and proposed as a workaround solution in Jupyter repo @rchiodo linked us to is the best long term approach to resolve this, but thank you for linking them.

[RandomFractals]

WebCodecs is now part of Chrome 94 Beta: https://blog.chromium.org/2021/08/chrome-94-beta-webcodecs-webgpu.html

I think that's the spec to watch and webGPU for compute to enable audio/video codecs apis in vscode when Chromium is updated to that version.

[kirianguiller]

I'm a ML engineer working mainly on audio am really looking forward being able to use my notebook + tensorboard audio functionalities :)

[robinhad]

are there any news regarding this?

[detrin]

I am data scientist working with audio records, this issue forces me to use jupyter notebook instead of vscode notebook. I would be pleased to see this to be sorted out in a near future.

[yume-chan]

Sorry for a beginner question: is there any documentation about shipping or stripping ffmpeg from electron build? I mean, there is still a file called "libffmpeg.dylib" (for macOS) in VSCode's package, what's wrong with it?

---

WebCodecs is now part of Chrome 94 Beta: https://blog.chromium.org/2021/08/chrome-94-beta-webcodecs-webgpu.html

WebCodecs API just gives the decoded data back to you instead of play them directly. It doesn't mean Chrome now have its own decoder. If VSCode doesn't know how to decode a format now, it still won't do it in the future.

[dynamicwebpaige]

As .wav and other audio files (.mp3, .mp4, etc.) are first-class for machine learning and data science projects, this is a feature that we'd really love having support for on the Codespaces and github.dev sides. Is there any update on its prioritization in the upcoming iteration cycles?

[carlthome]

Just got caught up on this myself and dropped down to jupyter notebook in the terminal. Speaking as someone working on machine learning for musical audio, I'd love to see audio rendering in VS Code!

[lukepighetti]

I hate to post another +1 style message but, I'm trying to work on audio DS project and I was surprised to see a disabled audio player in VSCode. This is my preferred environment and it's really a shame that I cannot proceed with this project in my preferred environment.

[LucaVignati]

I join the already conspicuous list of audio programmers looking forward to this issue being solved. I hope it will be addressed soon.

[ouoertheo]

I'm a nobody and I have no idea what I'm doing. But I would also like this fixed.

[deepak1556]

Sorry for the delay on this, we are currently discussing with our legal team on adding support for .mp3, .wav and .mp4 codecs to our ffmpeg library. I will update here once we have the necessary green light.

is there any documentation about shipping or stripping ffmpeg from electron build? I mean, there is still a file called "libffmpeg.dylib" (for macOS) in VSCode's package, what's wrong with it?

@yume-chan there is no public documentation for this currently, but today we ship a slim version of ffmpeg with the following codec support

.opus
.webm (encoded with opus codec, the default encoding with vorbis codec is not supported)
.flac

[rgrimwood]

I'm a nobody and I have no idea what I'm doing. But I would also like this fixed.

SNAP!

[p-i-]

There is clearly huge demand for this functionality. At time of writing, 175 upvotes on the OP. It's aggravating to have to move between VSCode's .ipynb editor and Jupyter's browser-based .ipynb editor, just to play audio inline. Nowhere can I see it clearly stated exactly what the block is. @deepak1556 mentions "Legal Team". Is it an ffmpeg-licensing issue? The usual (AFAIU) way around such issues is to provide the user a button to download the component with the offending license, thus making it THEIR responsibility, and not that of VSCode/Microsoft. Another option would be to contact the owner(s) of FFMPEG directly and request they make an exception.

But why not simply create code that plays a raw audio buffer (e.g. mono/sterero np.array)? VSCode is a Desktop application targeting Win/macOS/Linux. How hard can that be? You could look into the JUCE repo to see how they have achieved it. And then leave the conversion of any other format into raw PCM up to the Python community, such that it can be resolved with an import?

I look forward to a situation where VSCode Notebook is able to act as a complete replacement for Jupyter Notebook. You are so close!

[PatrykMiszczak]

But why not simply create code that plays a raw audio buffer (e.g. mono/sterero np.array)

Yes, playing from the buffer will be enough for most cases. I think most people will use this to check the results of their processing

[p-i-]

^ I've created a feature-request for a barebones functionality to play a mono/stereo np.array from a cell output. I think this should be addressed separately from the much larger task of a fully-fledged media player. If the team would care to provide this, it would give us audio engineers enough to work with. Forever grateful for the tremendous work of the VSCode team, and in particular the .ipynb unit. It's a splendid technology.

[lukepighetti]

Ticket https://github.com/microsoft/vscode/issues/151149 has been selected as candidate for backlog. If we upvote it, it will go into the backlog. It requests the ability to play raw waveforms in VSCode.

[Myslicki]

VSCode moved it to their backlog as of July 2nd. Progress! Reference: https://github.com/microsoft/vscode/issues/151149

[isidorn]

@deepak1556 plans to add ffmpeg with the following codecs to VS Code insiders:

• Vorbis
• Flac
• H.264
• VP8
• WAV
• MP3
• Ogg

This should allow the majority of notebook scenarios from this issue. We would really appreciate if you try it out and let us know if there are scenarios which are still not working since they might require codecs which we are still not shipping.

I will comment here once we release VS Code insiders with these changes next week.

Thank you 🙏

[isidorn]

Today's VS Code Insiders has all the codecs I have listed above ☝️ thanks to @deepak1556 and @alexeykuzmin. Please try it out with various Notebook scenarios and do let us know if some flows are still not working.

I will still keep this issue open to collect feedback, and we can close it if things look good.

Thank you 🙏

[bliep]

Fantastic work !

It (also) plays uncompressed signals:

import IPython.display as ipd
import numpy as np
fs = 16000.
ipd.Audio(np.sin(2*np.pi*440*np.arange(5 * fs)/fs), rate=fs)

[isidorn]

Closing this issue as the planned work is done. And in case some scenarios are still not working do let us know or just open a new issue and ping me @isidorn on it. Thank you very much.

[lukepighetti]

Big thank you for the support here

[sugoidesune]

Didn't work, updated vscode and the feature is here and working! Thank you all! ❤🎉🎉