Support multichannel audio

[ssfrr]

Currently we're sending a mono signal out to PortAudio. We need to put some thought into how multichannel audio should be handled.

I think in general the audio should be passed around in NxM vectors, where N is the # of frames and M is the number of channels.

This is related to AudioNodes that might have multiple inputs and/or outputs. For instance, a Panner node would have 1 input and 2 outputs. Should it be a stereo output, or 2 separate outputs? The AudioMixer node has potentially many inputs, but they're likely from different nodes so I'd think of them as separate inputs, not a single multichannel input. Can AudioMixer mix multiple stereo inputs into a single stereo output?

needs more thought.

[jiahao]

The @async pattern I used in my karaoke notebook actually works great now in v0.1.1 - I don't get the weird sync issues anymore.

[ssfrr]

Awesome! In general the switch to PortAudio's read/write API seems to have cleaned things up a lot. I still seems to be getting some spurious underflow warnings on ALSA, but the audio seems much more solid.

Also now it should be easier to implement options on opening the audio stream (# of ins/outs, different devices, etc.).

[jiahao]

I tried updating my karaoke notebook to 0.3/0.4 and it throws errors. Code like

@sync begin 
    soprano = @async play(parsevoice("""
    f'#.4 f'#. g'. a'. a'. g'. f'#. e'. d'. d'. e'. f'#. e'.~ e' d'8 d'4~ d'2
    """, lyrics="Wir be- tre- ten feu- er- trun- ken, Himm- li- sche, dein Hei- - lig- thum!"))

    alto = @async play(parsevoice("""
    a.4 a. b. c'. c'. b. a. g. f#. f#. g. f#. g.~ g4 f#8 f#~ f#2 
    """))

    tenor = @async play(parsevoice("""
    d.4 d. d. d. d. d. d. d. d. d. c#. d. c#.~ c# d8 d d2  
    """))
    bass = @async play(parsevoice("""
    d.4 d. d. d. a,. a,. a,. a., a., a., a., a., a.,~ a, a,8 d, d,2
    """))
end

produces errors like

InexactError()
 in start_timer at ./stream.jl:507
 in sleep at ./stream.jl:529
 in sleep at In[3]:4
 in play at In[6]:12
 in play at In[6]:2
 in anonymous at task.jl:368

on 0.4.

I traced it down to Base.start_timer being started with a negative timeout parameter. Modifying Base.start_timer to show what's going on:

function start_timer(timer::Timer, timeout::Real, repeat::Real)
    assert(timeout ≥ 0, "negative timeout = $timeout in start_timer") #<--
    assert(repeat ≥ 0, "negative repeat = $repeat in start_timer") #<--
    associate_julia_struct(timer.handle, timer)
    preserve_handle(timer)
    ccall(:uv_update_time,Void,(Ptr{Void},),eventloop())
    ccall(:uv_timer_start,Cint,(Ptr{Void},Ptr{Void},UInt64,UInt64),
          timer.handle, uv_jl_asynccb::Ptr{Void}, UInt64(round(timeout*1000))+1, UInt64(round(repeat*1000)))
end

produces

ERROR (unhandled task failure): AssertionError: Negative timeout = -1.5525746741818183 in start_timer
 in assert at error.jl:37
 in start_timer at ./stream.jl:505
 in sleep at ./stream.jl:529
 in sleep at In[3]:4
 in play at In[6]:12
 in play at In[6]:2
 in anonymous at task.jl:368
ERROR (unhandled task failure): AssertionError: Negative timeout = -0.4382639541818184 in start_timer
 in assert at error.jl:37
 in start_timer at ./stream.jl:505
 in sleep at ./stream.jl:529
 in sleep at In[3]:4
 in play at In[6]:12
 in play at In[6]:2
 in anonymous at task.jl:368
r Task (failed) @0x000000010e404910
nothing
AssertionError("Negative timeout = -2.6962522031818184 in start_timer")

The problem appears to be the delay between when play() is called and the data generated by parsevoice is ready to be played. Code like

soprano = parsevoice("""
f'#. f'#. g'. a'. a'. g'. f'#. e'~ e'8 d.'4 d.' e.' f#'. f#'.~ f#' e'8 e'4~ e'2
""", lyrics="Freu- de, schö- ner Göt- ter- fun- ken, Toch- ter aus E- li- - si- um!"
)

alto = parsevoice("""
a. a. a. a.  a.  a. a. a~ g8 f#.4 a.  a.  a. a.~ a a8 a4~ a2
""")

tenor = parsevoice("""
d. d. e. f#. f#. e. d. d~ e8 f#.4 f#. a,. d. d.~ d c#8 c#4 c#2
""")

bass = parsevoice("""
d. d. d. d. a,. a,. a,. b,~ c8 d. a., a., a., a., a, a8, a,4 a,2
""")

@sync begin
    @async play(soprano)
    @async play(alto)
    @async play(tenor)
    @async play(bass)
end

or

@sync begin

@async begin
soprano = parsevoice("""
f'#. f'#. g'. a'. a'. g'. f'#. e'~ e'8 d.'4 d.' e.' f#'. f#'.~ f#' e'8 e'4~ e'2
""", lyrics="Freu- de, schö- ner Göt- ter- fun- ken, Toch- ter aus E- li- - si- um!"
)
play(soprano)
end

@async begin
alto = parsevoice("""
a. a. a. a.  a.  a. a. a~ g8 f#.4 a.  a.  a. a.~ a a8 a4~ a2
""")
play(alto)
end

@async begin
tenor = parsevoice("""
d. d. e. f#. f#. e. d. d~ e8 f#.4 f#. a,. d. d.~ d c#8 c#4 c#2
""")
end

@async begin
bass = parsevoice("""
d. d. d. d. a,. a,. a,. b,~ c8 d. a., a., a., a., a, a8, a,4 a,2
""")
play(bass)
end

end #sync

works, where the generation of the waveform is taken out of the play() call site.

[JeffBezanson]

Holy crap, that form of the assert function needs to be removed. It allocates the string argument eagerly, meaning code that uses it will allocate huge numbers of unused strings. We used to have such a method and I removed it when I started seeing numerical code spending all of its time allocating strings. I see it has crept back in. I will remove it again.

[jiahao]

@JeffBezanson I put those two lines in for debugging. It's not in the base library version.

[JeffBezanson]

No, the problem is the existence of the assert(x, msg) method that lets you write those. That method does exist in Base.

[ssfrr]

Just catching up on this now. So it looks like in 0.4 the current behavior will be to throw an error if start_timer is called with a negative wait time, so I should be handling that case more gracefully within AudioIO, correct? Just want to make sure I'm caught up before I dive in.

[jiahao]

Yep