WebAudio API as DirectShow (Audio) replacement

[smakhtin]

Hello. Currently i'm thinking about implementing ScopeSpread and stuck at Audio ecosystem. I started to write my own FileStream node, but found your own FileStream (Canvas VVVVjs). Do i need to write my own FileStream (DirectShow), that will output Audio as an Object, that contains audio buffer, or i can somehow modify your node? I'm asking this because i found, that you have implemented VideoTexture node, but can't find any DirectShow object.

[zauner]

Hi,

the reason why FileStream (DShow9) has not simply been ported, but somehow been replaced by FileStream (HTML5 VVVVjs), is because it needed some additional output pins (e.g. Network Status). This made it a kind of self-contained thing.

Right now, FileStream (HTML5 VVVVjs) only supports video files (it in fact it creates a hidden <video> element, which is output through the output pin). I think it would be most consistent to make the node create <audio> elements if the input filename is an audio file.

Audio-processing nodes like FFT or ScopeSpread would then use the <audio> element to work on.

Does this somehow comply with how you designed your FileStream node?

[smakhtin]

This is how i think audio part should look https://gist.github.com/3078669 . Looks like Web Audio API don't need audio tag, but maybe i'm wrong - still new in web programming.

[zauner]

From what I read in the last hour, there's no need for the audio element, true. But I think it might be a good idea to use it for various reasons:

• passing the element object from one node to the other might be a suitable counterpart to a DirectShow object, which already comes with all the streaming capabilities. We won't need an own Audio object type.
• At least audio playback will work with browsers which don't support Web Audio API (not sure how relevant that is in context of VVVV.js)
• FileStream (DShow9) doesn't have those Network Status output pins, which of course would be nice for audio as well, so we might want to have an own node anyway.

One could get the source node for audio processing like this, I guess: https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html#AudioElementIntegration

I don't know if there are any disadvantages of this method, however.

[smakhtin]

Sounds right, thx for the link. I check tomorrow how your node operates with Video tag and will try to integrate Audio part, that will depend on file extension.

[zauner]

I see you have already done plenty of work on the audio stuff.

How's it going? I'm very excited about that... anything ready to try out already?

[smakhtin]

Yep, just check audio visualisation example. Placed in examples folder in my fork. If it's working for you, you can merge it with core repo, but i think we need to disqous some stuff about dshow9 category previously. For example, i think we can move AoudioOutput to DShow9 group, this will allow us to remove dummy module. Also, i started reimplementing VVVV's utility methods, need to be sure if i doing all right. Also i checked vvvv.core.js and vvvv.js with JSLint utility and fixed all warnings. Is it ok? Maybe it will be hard to merge, but i hope not.

[zauner]

Ok super, I'll try it now.

Making it JSLint conform is nice. Don't think, those changes will cause trouble when mergin.

I saw you removed the check in Pin.setValue() again, which is good, because I was not sure about that anyway. I might have abused this behaviour to force-trigger pin changes in some nodes.

I'm currently downloading from your repo. It's like 100MB in size. I think we shouldn't keep example media in the repo, but load it from somewhere else.

[zauner]

Oyeah, that's great, hah!

A few things I noticed while checking briefly:

• when determining the file extension, you should consider URLs as well -- there are more than one dot
• video file extensions should include .ogg files
• if I change the filename during runtime, it seems another audio tag will be created, or anything else causing some weirdness. This might have already been an issue before, I don't know :)

Also, it doesn't work with Firefox, of course... maybe we should evaluate if https://github.com/corbanbrook/audionode.js could help here.

Great job! :)

[smakhtin]

Hi. Thx for checking, will try to fix this bugs today. I think audio bug is appearing because i creating multiple audio contexts. We should create only one audio context per appliction, so every node can access it. Where is the place for such global variables? About Firefox, my node is based on w3c audio interfaces specification, so i hope it will work in it in future versions. Btw, i will check the library, that you suggested, maybe i missed something. I will remove media files from my repo and host it in dropbox folder in next commit.

[zauner]

I guess the bug with the filename change is because there's something like this:

if (filenameIn.pinIsChanged()) {
  ...
  createAudio();
  ...
}

which will create a new audio tag, no matter if there's already one, right? The old one will then just remain in the DOM and keep playing.