playing audio files

[metaphorz]

I am having some problems in playing one or more audio files. My goal is to play a mix of two audio files, each .wav being modulated by a separate curve. The primary issue I have now is trying simply to play the .wav. Consider a test.wav file that is 2 minutes in duration. How is this placed into the audio stream?

I looked at Gibberish.Sampler and tried

a = new Gibberish.Sampler({file: 'resources/test.wav'}).connect(); a.note(1);

The first problem is not following the permitted syntax for Gibberish.Sampler. I would have expected a fully-formed URL such as "file:///Users/me/.....wav" or "http://.....wav". What is the grammar for the argument for Gibberish.Sampler ?

The second problem is that regardless of the grammar not being surfaced (I could not find this level of detail in the documentation), the above code doesn't work, leading me to believe that Sampler may be the wrong choice for audio files not meant for being deposited into a sampler (for note playback).

I found several methods in the Gibberish reference but none of them pointed to the ability to play audio (in general).

Oh, also, I tried playing another .WAV file and an error came back that the file was not PCM encoded. What are the acceptable WAV file formats in GIbberish?

Thanks.

[charlieroberts]

The way Gibberish loads audio now was designed to also support the Audio Destination API, which used to be used by Firefox. However, the last couple of versions of Firefox have supported the Web Audio API, so it's probably safe to move to this for loading audio. This should provide much better file compatibility.

I'll try and get this done in the next couple of days.

[metaphorz]

Charlie I don’t really follow what are you saying…probably because I have not used the Web Audio API. Are you proposing that my syntax will work (using the Gibberish Sampler) once you make the change, or are you proposing a different method within Gibberish that I should be using? If you can fill in these details, that would be great. -p

On Feb 16, 2014, at 2:55 PM, charlieroberts notifications@github.com wrote:

The way Gibberish loads audio now was designed to also support the Audio Destination API, which used to be used by Firefox. However, the last couple of versions of Firefox have supported the Web Audio API, so it's probably safe to move to this for loading audio. This should provide much better file compatibility.

I'll try and get this done in the next couple of days.

— Reply to this email directly or view it on GitHub.

Paul Fishwick, PhD Chair, ACM SIGSIM Distinguished Chair of Arts & Technology and Professor of Computer Science Director, Creative Automata Laboratory The University of Texas at Dallas Arts & Technology 800 West Campbell Road, AT10 Richardson, TX 75080-3021 Home: utdallas.edu/atec/fishwick Blog: creative-automata.com

[charlieroberts]

Once I make the changes, the syntax should work as expected and support both remote and local loading (http:// or file://). I'm starting on it now... - Charlie

PS - Aesthetic Computing is on my bookshelf. Thanks for putting together such an interesting collection of writings.

[metaphorz]

Charlie Thanks. I understand - I’ll end up using Gibberish.Sampler then.

Aesthetic Computing was a good one — also see my blog creative-automata.com

On Feb 16, 2014, at 10:23 PM, charlieroberts notifications@github.com wrote:

Once I make the changes, the syntax should work as expected and support both remote and local loading (http:// or file://). I'm starting on it now... - Charlie

PS - Aesthetic Computing is on my bookshelf. Thanks for putting together such an interesting collection of writings.

— Reply to this email directly or view it on GitHub.

Paul Fishwick, PhD Chair, ACM SIGSIM Distinguished Chair of Arts & Technology and Professor of Computer Science Director, Creative Automata Laboratory The University of Texas at Dallas Arts & Technology 800 West Campbell Road, AT10 Richardson, TX 75080-3021 Home: utdallas.edu/atec/fishwick Blog: creative-automata.com

[charlieroberts]

OK, the changes are pushed. I hope they help. Here's a summary of what file types are supported in each browser:

https://developer.mozilla.org/en-US/docs/HTML/Supported_media_formats

Let me know how if there are questions. I'll take a gander at the blog now!

[metaphorz]

Thanks - I am working on something with a Musician. In short the goal is to take the outputs of a Lotka-Volterrar predator Prey model and use the two outputs as modulators of WAV files. So, I have two files:

predator.wav and prey.wav

I am assuming that mixing two streams will happen automagically, if I connect each stream to the audio bus. Here is my approach (given your latest edits). Tell me if it makes sense to you given Gibberish syntax. I have been using Codio for my Javascript IDE. We also employ Codio in my class. We’ve also tried jsfiddle and codepen but their IDEs seem more limiting in terms of file organization.

My Gibberish.Sampler() may have syntax issues. Codio is down today, so I will be unable to try this until tonight.

I am trying to achieve this in terms of a graph, much as one might draw one in MAX/MSP or PD:

Source (prey.wav) ——> modulated by (prey wavetable) —> into MIXER Source (predator.wav) —> modulated by (predator wavetable) —> into MIXER MIXER —> audio output

I confess to being a web graph newbie so not even sure if this will work. If you see anything obvious (especially with he Sampler syntax), let me know.

……..

// Begin Gibberish sound synthesis

    Gibberish.init();
    Gibberish.Time.export();
    Gibberish.Binops.export();  

// Create 2 new samplers based on 2 audio files (predator.wav and prey.wav) // These samplers contain our 2 raw audio sources (2 minutes long of digital audio)

preysampler = new Gibberish.Sampler(
    {https: ‘www.dropbox.com/s/….the path…/Prey.wav’}).connect(preytable);
predatorsampler = new Gibberish.Sampler(        
            {https: ‘www.dropbox.com/s/….the path…/Predator.wav’}).connect(predatortable);

// Create two Wave Tables (one for Predator and one for Prey) // preytable and predatortable will serve as modulators for digital audio // from preysampler and predatorsampler

    var preyarray = [];
    var predatorarray = [];

    preytable = new Gibberish.Table();
    preytable.setTable(preyarray);
predatortable = new Gibberish.Table();
predatortable.setTable(predatorarray);

// connect both tables to the bus which presumably mixes everything. More formally, // I’d like to connect the modulated outputs to a mixer node and then connect the mixer node // to the final output.

preytable.connect();
predatortable.connect();

On Feb 18, 2014, at 1:08 AM, charlieroberts notifications@github.com wrote:

OK, the changes are pushed. I hope they help. Here's a summary of what file types are supported in each browser:

https://developer.mozilla.org/en-US/docs/HTML/Supported_media_formats

Let me know how if this helps or if there are questions. I'll take a gander at the blog now!

— Reply to this email directly or view it on GitHub.

Paul Fishwick, PhD Chair, ACM SIGSIM Distinguished Chair of Arts & Technology and Professor of Computer Science Director, Creative Automata Laboratory The University of Texas at Dallas Arts & Technology 800 West Campbell Road, AT10 Richardson, TX 75080-3021 Home: utdallas.edu/atec/fishwick Blog: creative-automata.com

[charlieroberts]

One immediate problem is that you're loading the files from a different domain than your HTML (at least, I assume this is true based on what you've said). This results in cross-domain loading issues:

https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS

I might be able to mitigate this with CORS, but for now, if you can put the files in the same domain as your HTML is being loaded from that should help download the files correctly.

[metaphorz]

I am having a problem making valid connections in the audio graph (could well be due to the fact that I have not used Web Audio).

From my typical use of signal processing software such as SimuLink, Ptolemy, etc, my thinking of modulation is as follows:

A filter called a Modulator.

Modular takes two inputs and produces one output The two inputs are: raw signal to be modulated and modulation signal The output is the modulated raw signal

That is my mental model of how this ought to work. Gibberish may work like this but I could not tell. So what I have below is what I rigged up (using the ‘file’ option and a Google Chrome option of allowing local audio files to load)—the loading works fine, and no JS errors. I don’t hear any sound though.

Do I need to explicitly measure the length of each prey.wav and predator.wav and put this in a duration for the sampler?

// Begin Gibberish sound synthesis Gibberish.init(); Gibberish.Time.export(); Gibberish.Binops.export();

// Create two Wave Tables (one for Predator and one for Prey) // preytable and predatortable will serve as modulators for digital audio // from preysampler and predatorsampler

var preyarray = [];
var predatorarray = [];

preytable = new Gibberish.Table();
preytable.setTable(preyarray);
predatortable = new Gibberish.Table();
predatortable.setTable(predatorarray);

// Create 2 new samplers based on 2 audio files (predator.wav and prey.wav) // These samplers contain our 2 raw audio sources (2 minutes long of digital audio)

preysampler = new Gibberish.Sampler(
    {file:'resources/prey.wav'}).connect();
predatorsampler = new Gibberish.Sampler(        
    {file:'resources/predator.wav'}).connect();

preysampler.note(1);
predatorsampler.note(1);

On Feb 18, 2014, at 2:18 PM, charlieroberts notifications@github.com wrote:

One immediate problem is that you're loading the files from a different domain than your HTML (at least, I assume this is true based on what you've said). This results in cross-domain loading issues:

https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS

I might be able to mitigate this with CORS, but for now, if you can put the files in the same domain as your HTML is being loaded from that should help download the files correctly.

— Reply to this email directly or view it on GitHub.

Paul Fishwick, PhD Chair, ACM SIGSIM Distinguished Chair of Arts & Technology and Professor of Computer Science Director, Creative Automata Laboratory The University of Texas at Dallas Arts & Technology 800 West Campbell Road, AT10 Richardson, TX 75080-3021 Home: utdallas.edu/atec/fishwick Blog: creative-automata.com

[charlieroberts]

This might not be what your problem is, but JS is asynchronous, which makes certain elements (like loading samples) a little tricky. If the code really is just as you put above, then the note message is received by the sampler object before the sample has downloaded, so you never hear it play.

You can solve this a couple of ways:

1) You can add a the playOnLoad property to the constructor:

a = new Gibberish.Sampler({ file:'resources/snare.wav', playOnLoad:1 }).connect();

2) You can add an onload event handler to the constructor (or to the object immediately after it is created)

a = new Gibberish.Sampler({ file:'resources/snare.wav', }).connect();
a.onload = function() { 
  a.note(1); 
  // also do some other stuff
}

I just noticed that the onload property is incorrectly identified as _onload in the documentation. Don't override _onload! It will mess things up.

[charlieroberts]

Also, in terms of modulation, those are generally expressed via binops in Gibberish. For example:

a = new Gibberish.Sine( 440, .1 )
b = new Gibberish.Sine( 321, .1 )
c = Mul( a, b ).connect()

The above end-user code would be compiled to the following audio callback function by Gibberish:

function (input,sine_5, sine_8, bus2_0){
    var v_14 = sine_5(440, 0.1);
    var v_15 = sine_8(321, 0.1);
    var v_4 = bus2_0(( v_14 * v_15 ), 1, 0);

    return v_4;
}

I'm not sure what type of modulation you're going for... there's lots of interesting possibilities. For example, it might be interesting to modulate the playback rate of the Sampler objects using the wavetable output:

preytable = new Gibberish.Table();
preytable.setTable(preyarray);
predatortable = new Gibberish.Table();
predatortable.setTable(predatorarray);

preysampler = new Gibberish.Sampler({file:'resources/prey.wav'}).connect();
predatorsampler = new Gibberish.Sampler({file:'resources/predator.wav'}).connect();

preysampler.pitch = Abs( preytable )
predatorsampler.pitch = Abs( predatorsampler )

Hope this helps!

[charlieroberts]

I fixed the sampler documentation to reflect everything I said earlier and pushed the changes to the website.

[metaphorz]

It certain helped me get to the next step -- the two wave files load and I used your suggesiton of onload rather than _onload. They are loaded.

I can verify that both of the wavetables have correct signals..buy just connecting them to the bus.

However, the modulation is not working - syntax error in modulating pitch via the table. See below.

It is about 90% done I think!

var preyarray = []; var predatorarray = [];

preytable = new Gibberish.Table();
preytable.setTable(preyarray);
predatortable = new Gibberish.Table();
predatortable.setTable(predatorarray);

/* This works -- correct audio for wavetables
preytable.connect();
predatortable.connect(); */

// Create 2 new samplers based on 2 audio files (predator.wav and prey.wav) // These samplers contain our 2 raw audio sources (2 minutes long of digital audio)

preysampler = new Gibberish.Sampler(
    {file:'resources/prey.wav'});
predatorsampler = new Gibberish.Sampler(        
    {file:'resources/predator.wav'});

preysampler.onload = function() {
     preysampler.note(1);
}

predatorsampler.onload = function() {
   predatorsampler.note(1);
} 

preysampler.pitch = Abs(preytable);
predatorsampler.pitch = Abs(predatortable);

[charlieroberts]

Hmmm... if I run the following code on the Gibberish site (using Chrome's built-in JavaScript console), everything works. The trumpet sample is included in the Gibberish repo.

a = new Gibberish.Table({ amp: 1 });
var t = []
for( var i = 0; i < 1024; i++ ) { t[ i ] = Gibberish.rndf(-1,1) }
a.setTable( t )

b = new Gibberish.Sampler({ 
  file: 'resources/trumpet.wav',
  loops: true,
  pitch: Abs( a )
}).connect()

which yields the following callback function:

function (input,table_2, abs_3, sampler_4, bus2_0){
    var v_7 = table_2(440, 1);
    var v_6 = abs_3(v_7);
    var v_5 = sampler_4(v_6, 1, false, true, 0, 353359, 0, 353359, true, 0);
    var v_1 = bus2_0(v_5, 1, 0);

    return v_1;
}

I might need to see the callback function that you get syntax errors with in order to debug. You can print this at the console by just typing Gibberish.callback.

[metaphorz]

I revised my code to mirror the way you specific modulation (inside of Sampler). I am listing the code followed by the callback information:

var preyarray = [];
var predatorarray = [];

preytable = new Gibberish.Table();
preytable.setTable(preyarray);
predatortable = new Gibberish.Table();
predatortable.setTable(predatorarray);

/* This works -- correct audio for wavetables
preytable.connect();
predatortable.connect(); */

// Create 2 new samplers based on 2 audio files (predator.wav and prey.wav) // These samplers contain our 2 raw audio sources (2 minutes long of digital audio)

preysampler = new Gibberish.Sampler(
    {file:'resources/prey.wav',
    pitch: Abs(preytable)}).connect();
predatorsampler = new Gibberish.Sampler(        
    {file:'resources/predator.wav',
    pitch: Abs(predatortable)}).connect();

preysampler.onload = function() {
     preysampler.note(1);
}

predatorsampler.onload = function() {
   predatorsampler.note(1);
} 

…………………...

LOADED resources/predator.wav 8327087 gibberish_2.0.js:5005 5 Uncaught TypeError: undefined is not a function VM79:7 LOADED resources/prey.wav 13353590 gibberish_2.0.js:5005 1285 Uncaught TypeError: undefined is not a function VM80:6 Gibberish.callback; function (input,abs_4, sampler_5, abs_6, sampler_7, bus2_0){ var v_9 = abs_4(1); var v_8 = sampler_5(v_9, 1, false, true, 0, 13353590, 0, 13353590, 0, 0); var v_12 = abs_6(1); var v_11 = sampler_7(v_12, 1, false, true, 0, 8327087, 0, 8327087, 0, 0); var v_1 = bus2_0(v_8, v_11, 1, 0);

return v_1;

}

On Feb 19, 2014, at 5:46 PM, charlieroberts notifications@github.com wrote:

Hmmm... if I run the following code on the Gibberish site (using Chrome's built-in JavaScript console), everything works. The trumpet sample is included in the Gibberish repo.

a = new Gibberish.Table({ amp: 1 }); var t = [] for( var i = 0; i < 1024; i++ ) { t[ i ] = Gibberish.rndf(-1,1) } a.setTable( t )

b = new Gibberish.Sampler({ file: 'resources/trumpet.wav', loops: true, pitch: Abs( a ) }).connect() which yields the following callback function:

function (input,table_2, abs_3, sampler_4, bus2_0){ var v_7 = table_2(440, 1); var v_6 = abs_3(v_7); var v_5 = sampler_4(v_6, 1, false, true, 0, 353359, 0, 353359, true, 0); var v_1 = bus2_0(v_5, 1, 0);

return v_1;

} I might need to see the callback function that you get syntax errors with in order to debug. You can print this at the console by just typing Gibberish.callback.

— Reply to this email directly or view it on GitHub.

Paul Fishwick, PhD Chair, ACM SIGSIM Distinguished Chair of Arts & Technology and Professor of Computer Science Director, Creative Automata Laboratory The University of Texas at Dallas Arts & Technology 800 West Campbell Road, AT10 Richardson, TX 75080-3021 Home: utdallas.edu/atec/fishwick Blog: creative-automata.com

[metaphorz]

Looks like the issue was with Abs() function which is undefined. What library is this from?

[charlieroberts]

You should get Abs if you do Gibberish.Binops.export()... I should probably rename Binops as Math since there are now a couple of other unit generators in there that only take one argument. But it looks like the abs functions are appearing in the callback... it's the table functions that aren't showing up.

Did the example code I sent work for you? If so, then I'm probably going to have to take a look at the your site to debug the problem... if you post the current version somewhere I'll see if I can figure out the problem.

[metaphorz]

Thanks - I'll send you the code via separate email. The thing is that it is all mixed with jxsgraph code.

Your code snippet worked just fine.

The Gibberish part of interest where the errors arise (with Abs() ) begin with // Begin Gibberish Sound Synthesis and end with the line before //JSXGraph. It isn't much code but the use of Abs(() is what creates the problem.

[metaphorz]

It is all in index.html (but ignore the jxsgraph stuff)—the Gibberish part is at the top where previously noted:

https://github.com/metaphorz/GibberishTesting

On Feb 19, 2014, at 8:55 PM, charlieroberts notifications@github.com wrote:

You should get Abs if you do Gibberish.Binops.export()... I should probably rename Binops as Math since there are now a couple of other unit generators in there that only take one argument. But it looks like the abs functions are appearing in the callback... it's the table functions that aren't showing up.

Did the example code I sent work for you? If so, then I'm probably going to have to take a look at the your site to debug the problem... if you post the current version somewhere I'll see if I can figure out the problem.

— Reply to this email directly or view it on GitHub.

Paul Fishwick, PhD Chair, ACM SIGSIM Distinguished Chair of Arts & Technology and Professor of Computer Science Director, Creative Automata Laboratory The University of Texas at Dallas Arts & Technology 800 West Campbell Road, AT10 Richardson, TX 75080-3021 Home: utdallas.edu/atec/fishwick Blog: creative-automata.com

[charlieroberts]

OK, you were right... there's a bug in Abs. If you only use it once, it works fine, but for some reason the second time you use it the function isn't added to the array that gets passed as arguments to the audio callback function... that's why it comes up as undefined.

This seems like a tricky bug (every bug related to the codegen system is, unfortunately) but I'll try to figure it out soon. In the mean time, you could experiment with other types of modulation on the second sampler... again, it works fine for me with the first one, it's only when I use Abs a second time in the same graph that I get an error.

[charlieroberts]

Fixed and pushed... it wasn't as bad as I thought. Let me know how it goes!

[metaphorz]

The good news is that your fix to Abs() worked. But it does lead to other general questions of load synchronization. For example, I am loading two large .wav audio files, so instead of setting modulation within Sampler(), I thought I would follow one of your previous examples of modulation using Mul(). However, this results in no sound and I am thinking that the problem may be with the fact the browser has not yet fully loaded the .wav files when I try to do the modulation (see preymix and predatormix below). Do you have a particular approach in Gibberish for this type of synchronization (I realize that this could be another thread, although it ultimately relates to long audio file loading).

There are no JS errors with this code - just silence:

(I am playing with both "Add" and "Mul" for modulation...)

preysampler = new Gibberish.Sampler( {file:'resources/prey.wav'})

predatorsampler = new Gibberish.Sampler(        
    {file:'resources/predator.wav'})

preysampler.onload = function() {
     preysampler.note(1);
     preymix = Add(preysampler,preytable).connect();
};
predatorsampler.onload = function() {
     predatorsampler.note(1);
     predatormix = Add(predatorsampler,predatortable).connect(); 
};

[metaphorz]

On the theory that if I cascaded the events, that this would allow me to modulate via binops (Mul, Add, etc), I tried this, but didn't work..I'll test again tonight

var predatorsampler; var preysampler;

preysampler = new Gibberish.Sampler(
    {file:'resources/prey.wav'});   

   preysampler.onload = function() {
     preysampler.note(1);   
     predatorsampler = new Gibberish.Sampler(       
          {file:'resources/predator.wav'});      
     predatorsampler.onload = function() {
         predatorsampler.note(1);   
         mixed = Add(predatorsampler,predatortable).connect();  
   };             
};  

[metaphorz]

I have something working so far -- it sounds really noisy, but that is likely because I need to look more closely at the signals being generated (their amplitude levels) for the base digital audio signal as well as the modulation signal. Thanks for your feedback and fixes.

[charlieroberts]

Great! Glad to hear things are (almost) working. Please keep me posted on how it goes!