issues/feature request - audio

[ghost]

from my point of sight, it would be interesting for you to:

• including supercollider.js in your framework; (https://github.com/stagas/node-supercollider and https://github.com/crucialfelix/supercolliderjs)
• porting libpd to javascript (perharps using emscripten https://github.com/libpd/libpd)
• porting csound to webvr (perhaprs using emscripten as well https://github.com/csound/csound)
• porting PWGL to WebVR (http://www2.siba.fi/PWGL/)

---

furthermore:

• using an implementation of fft in opencl (i have forked some in case you may want to take a look https://github.com/tiagomoraismorgado88/Max-gpu.fft);
• use that to make an implementation of higher order ambisonics/emulating wave field synthesis in stereo; (possibly based on https://github.com/GameOfLife/WFSCollider)

[jsantell]

Songbird and Omnitone handle this with web audio

[cwilso]

Yep. And doing 3D audio is fairly straightforward in web audio with HRTF-mode PannerNodes.

[ghost]

Two things shall be considered in first place:

• supercollider, puredata, csound and pwgl, are state of the art opensource environments for sound synthesis, algorhtmic composition, etc.
• that means they are not ambisonics implementations

---

by trying to replicate wave field synthesis in stereo and implementing it throughout webcl, for instance:

• you would be able to do things that you are not capable of doing with standard ambisonics, as wfs is an higer order implementation of ambisonics, so more interpolation, etc. and having a framework for wave field synthesis, would allow you to do: vr with real large scale speaker arrays
• i think trying at least to learn about wave field wouldn't be a bad idea: https://en.wikipedia.org/wiki/Wave_field_synthesis
• Its mathematical basis is the Kirchhoff-Helmholtz integral

---

from my standing point, an informed reading of what these environment really are would be pertinent, so, here's some propedeutic web bibliography:

---

Supercollider

SuperCollider is an environment and programming language originally released in 1996 by James McCartney for real-time audio synthesis and algorithmic composition.

Since then it has been evolving into a system used and further developed by both scientists and artists working with sound. It is an efficient and expressive dynamic programming language providing a framework for acoustic research, algorithmic music, interactive programming and live coding.

Released under the terms of the GPLv2 in 2002, SuperCollider is free and open-source software.

• https://en.wikipedia.org/wiki/SuperCollider
• http://sourceforge.net/project/showfiles.php?group_id=54622
• http://www.audiosynth.com/icmc96paper.html
• https://dx.doi.org/10.1162/014892602320991383
• http://supercolliderbook.net/
• http://www.beast.bham.ac.uk/research/mulch.shtml
• http://lac.linuxaudio.org/2010/papers/32.pdf
• http://www.wertlos.org/~rohrhuber/articles/AlgorithmsToday.pdf
• http://www.fredrikolofsson.com/f0blog/?q=node/316
• http://supercollider.sourceforge.net/wiki/index.php/Systems_interfacing_with_SC
• https://github.com/glastonbridge/SuperCollider-Android/wiki
• http://cylob.blogspot.co.uk/2009/11/tiny-music-system.html
• http://supercollider.sourceforge.net/wiki/index.php/SuperCollider_with_emacs:_scel
• https://atom.io/packages/supercollider
• https://github.com/jleben/Scate
• http://akustik.hfbk.net/publications/Uncertainty_and_waiting.pdf
• https://github.com/supercollider-quarks

---

CSound

Csound is a computer programming language for sound, also known as a sound compiler or an audio programming language, or more precisely, an audio DSL. It is called Csound because it is written in C, as opposed to some of its predecessors.

It is free software, available under the LGPL.

Csound was originally written at MIT by Barry Vercoe in 1985, based on his earlier system called Music 11, which in its turn followed the MUSIC-N model initiated by Max Mathews at the Bell Labs. Its development continued throughout the 1990s and 2000s, led by John ffitch at the University of Bath. The first documented version 5 release is version 5.01 on March 18, 2006. Many developers have contributed to it, most notably Istvan Varga, Gabriel Maldonado, Robin Whittle, Richard Karpen, Michael Gogins, Matt Ingalls, Steven Yi, Richard Boulanger, and Victor Lazzarini.

Developed over many years, it currently has nearly 1700 unit generators. One of its greatest strengths is that it is completely modular and extensible by the user. Csound is closely related to the underlying language for the Structured Audio extensions to MPEG-4, SAOL.

• https://en.wikipedia.org/wiki/Csound
• https://csound.github.io/csound30/papersessions.pdf
• https://github.com/csound/csound/releases/tag/6.08.1
• http://sourceforge.net/projects/csound/
• https://github.com/csound/csound
• http://lac.linuxaudio.org/2012/papers/10.pdf
• http://createdigitalmusic.com/2011/10/csound-for-live-the-power-of-csound-in-ableton-with-or-without-any-coding/
• http://www.synthtopia.com/content/2011/10/15/csound-for-live-coming-october-17th/
• http://lac.linuxaudio.org/2012/papers/20.pdf
• http://wiki.laptop.org/go/Csound
• http://www.csounds.com/book/
• http://www.virtual-sound.com/demo/VirtualSound.pdf
• http://en.flossmanuals.net/csound/
• http://csounds.com/manual/html/indexframes.html

---

Pure-Data

Pure Data (Pd) is a visual programming language developed by Miller Puckette in the 1990s for creating interactive computer music and multimedia works. While Puckette is the main author of the program, Pd is an open source project with a large developer base working on new extensions. It is released under a license similar to the BSD license. It runs on GNU/Linux, Mac OS X, iOS, Android and Windows. Ports exist for FreeBSD and IRIX.

Pd is very similar in scope and design to Puckette's original Max program, developed while he was at IRCAM, and is to some degree interoperable with Max/MSP, the commercial successor to the Max language. They may be collectively discussed as members of the Patcher[2] family of languages.

With the addition of the Graphics Environment for Multimedia (GEM) external, and externals designed to work with it (like Pure Data Packet / PiDiP for Linux, Mac OS X), framestein for Windows, GridFlow (as n-dimensional matrix processing, for Linux, Mac OS X, Windows), it is possible to create and manipulate video, OpenGL graphics, images, etc., in realtime with extensive possibilities for interactivity with audio, external sensors, etc.

Pd is natively designed to enable live collaboration across networks or the Internet, allowing musicians connected via LAN or even in disparate parts of the globe to create music together in real time. Pd uses FUDI as a networking protocol.

• https://en.wikipedia.org/wiki/Pure_Data
• https://github.com/pure-data/pure-data/commit/eafd4341fb7554bf352cfd7637fe0c076199c3ec
• http://cycling74.com/support/faq_max4/#1
• https://pedrolopesresearch.wordpress.com/2010/03/02/possibilities2-audio-layer/
• http://modin.yuri.at/publications/reactable_icmc2005.pdf
• http://pc.gamespy.com/pc/spore/853810p1.html
• https://ccrma.stanford.edu/~cburns/NRCI/
• http://www-crca.ucsd.edu/~msp/techniques.htm
• http://pd-tutorial.com/
• http://en.flossmanuals.net/PureData/
• http://pd-graz.mur.at/label/book/bangbook.pdf
• http://mitpress.mit.edu/books/designing-sound

---

PWGL

PWGL is a program that gives the user a graphical interface to doing computer programming to create music. The interface has been designed for musicians, with many objects that allow one to see, hear, and manipulate musical materials. PWGL's interface is similar to other applications, including OpenMusic, Max/MSP, and Pd. It is most similar to OpenMusic, because both share lineage as successors to the 1980s-90s application Patchwork (the PW in PWGL refers to Patchwork.)

For those familiar with Max/MSP or Pd, the biggest difference to know about PWGL is that generally all user patches are organized in the form of a tree, with many computations that happen in the "leaves" and "branches" that feed into one another and end at the bottom of the patch with one object that is the "root." The user activates the patch by evaluating this root object, which then calls all the other objects successively up the tree to the leaves, in a recursive fashion. The outermost leaves then evaluate and feed their results back down. This happens through all levels of the patch back to the root object. When the root object evaluates, it sends the final answer to the user.

Users may evaluate the patch at locations other than the "root" object. The object called for evaluation will call up its own branches and leaves and output its result to the user. Other branches of the patch will not evaluate, nor will levels of the patch below this node. To evaluate an object, select it and hit 'v' (for "eValuate"!). Instructions for how to select objects are below.

---

from my perspective including any of these environments in a webvr framework would be highly benefitial because:

• it would allow for a level of control over sound, specially for things like procedural audio, that you cannot achieve with standar midi and pre-rendered audio stuff

---

bearing this in mind:

• i think you would benefict extensively of such a thing

---

so, my question is

• why not considering wave field syntheis for vr both in stereo emulaton, and as a possibility for doing large scale speakers clusters arrays?

---

Some content on wfs

WFS is based on the Huygens–Fresnel principle, which states that any wave front can be regarded as a superposition of elementary spherical waves. Therefore, any wave front can be synthesized from such elementary waves. In practice, a computer controls a large array of individual loudspeakers and actuates each one at exactly the time when the desired virtual wave front would pass through it.

The basic procedure was developed in 1988 by Professor A.J. Berkhout at the Delft University of Technology.[1] Its mathematical basis is the Kirchhoff-Helmholtz integral. It states that the sound pressure is completely determined within a volume free of sources, if sound pressure and velocity are determined in all points on its surface.

Therefore, any sound field can be reconstructed, if sound pressure and acoustic velocity are restored on all points of the surface of its volume. This approach is the underlying principle of holophony.

For reproduction, the entire surface of the volume would have to be covered with closely spaced monopole and dipole loudspeakers, each individually driven with its own signal. Moreover, the listening area would have to be anechoic, in order to comply with the source-free volume assumption. In practice, this is hardly feasible.

According to Rayleigh II the sound pressure is determined in each point of a half-space, if the sound pressure in each point of its dividing plane is known. Because our acoustic perception is most exact in the horizontal plane, practical approaches generally reduce the problem to a horizontal loudspeaker line, circle or rectangle around the listener.

• The origin of the synthesized wave front can be in any point on the horizontal plane of the loudspeakers. It represents the virtual acoustic source, which hardly differs from a material acoustic source at the same position. Unlike conventional (stereo) reproduction, the virtual sources do not move along if the listener moves in the room. For sources behind the loudspeakers, the array will produce convex wave fronts. Sources in front of the speakers can be rendered by concave wave fronts that focus in the virtual source and diverge again. Hence the reproduction inside the volume is incomplete - it breaks down if the listener sits between speakers and inner source.*

---

well, I have nothing against google, they make really outstanding stuff:

• chrome browser for instance, has a pretty solid architecture, and is one of the state of art browsers available these days.
• i would specially regard this as it is opensource, so you can touch the code and making modifications to it.

---

from my perspective, working with pwgl, csound, libpd and supercollider, within the context of vr may, within thyself, may be benefitial, as:

• these are state of the arts environments for things like sound synthesis and so on so forth.

---

and better:

• most of them are available with ports for openframeworks, which supports out of the box emscripten compilation, so it shouldn't be that difficult difficult to bind those to javascript.

---

in fact: supercollider:

• has clients for both node, and a language side implementation for javascript (i've included links). bearing this in mind,

---

i think it could, within thyself, be a good option to at least consider:

• some form of binding these kinds of things to webvr.
• you guys and everyone in the community could benefict of it into a larger extent, so, why not?

[cwilso]

I'm not sure what you're looking for, but it sounds like you'd be more interested in the Web Audio Working Group (https://www.w3.org/2011/audio/). I know most of us there are familiar with at least CSound and SuperCollider. Baking them into browsers seems very unlikely, but with forthcoming work on AudioWorklets, at least it should be possible to compile them into the web (possibly in WebAssembly, even), and connect them directly to audio outputs.

My understanding of wavefield synthesis is that it wouldn't work in stereo - it needs many transducers to invoke the wave fields, or the spatial aliasing makes it ineffective. If you just want to position synthesized or sampled sound sources in a 3D sound field, the HRTF panner in Web Audio is quite effective, and designed (obviously) for stereo headphones. If you want Ambisonics panning of streamed sources, Hongchan's work on OmniTone has proven that's quite doable on top of Web Audio too.

I would strongly advise against baking in a given sound library (other than Web Audio, which is already built into the browser implementations), and focus on the interactions - like, do we need a sound panner node that is hooked up in audio rate to a camera direction, for better tracking in a headset.

[ghost]

well, from my point of sight:

• it's possible to run supercollider at least in a web browser (as i mentioned you have clients for both javascript and node available)
• it's possible as well to run pure data and csound in a browser, as they are c/c++ based, and you can compile them with emscripten

---

regarding wfs:

• you have several environments for wfs sound spatialization such as wfscollider writen by miguel negrão, which allow you to basically apply wave field synthesis to a massive loudspeaker array, but also to stereo headsets, allowing you to emulate the wfs phenomena in stereo (it's never the same thing, but it could be enhanced)

---

regarding PWGL:

• the framework mr john carmack is writing for vr in oculus for facebook applies some form of lisp programming. so it would be interesting to have a component of audio emulating algorithmic composition throughout something which is already there and PWGL can be a great thing in that regard

[toji]

I certainly don't want to discourage anyone from pursuing novel approaches to various VR-related problems. However, this repo is for development of the WebVR spec itself, while this issue appears to deal primarily with audio libraries? It's not clear to me if changes to the WebVR spec (or any other spec) are being proposed here, and if the desired effects can be achieved without spec changes it's probably best to either create a new repo to implement a library in or move the discussion to the repo of an existing library with similar goals.

Closing this issue to ensure our issues list stays a bit more focused, but if I've missed something and there is indeed a spec change being proposed please let me know!