Audio module

[funrep]

If Helm is going to be a complete game engine it probably needs a easy way to handle audio. I've looked around and I think the best option is either SDL_mixer(http://hackage.haskell.org/package/SDL-mixer) or as someone pointed out on #haskell OpenAL(http://hackage.haskell.org/package/ALUT).

My question is, is there a need for an audio module and if so, what will be the best library, and also, could this be abstracted in a nice way using FRP?

Take care, hope it's no problem that I submit it here since there's no mailing-list or IRC channel for this project atm.

[veryrandomname]

Audio would make me more able to use helm on game jams. I prefer OpenAL over SDL_mixer since SDL gives problems with building on mac and helm would be tightly bound to SDL as a result. But I don't know how to make imperative audio and FRP work together.

[funrep]

Not sure if this could be done or is a good way of using audio, but couldn't a currently playing sound be represented as a signal and inside it is the volume so you can then transform it into a new signal with less or more volume to change the volume?

[veryrandomname]

That sounds nice to me, but my personal problem lies in how load and play sounds at will with frp. As my knowledge goes you have to create a signal which, when sampled, runs the IO action to play the sound. But I'm not even absolutely certain about WHEN signals are sampled. edit: I actually don't even know how to sample a signal only sometimes.

[z0w0]

I think using SDL_mixer for the initial version of the audio API will be our best bet. The issues with SDL on OS X are to do with the initialization of the window manager API, so using SDL_mixer shouldn't cause problems. As for how the API should work, I'm still trying to think of a nice way to have "notified emissions" from signals with Elerea, like how Elm's FRP system works. For example, if you created a function called play that takes SignalGen (Signal Bool), it would turn the audio on or off based on the value within the signal.

But with Elerea, the only way to get that to work (as far as I know) would be to constantly sample the supplied signal until it changes, essentially blocking the thread. Maybe we could make it spawn a concurrent task that repeatedly samples the signal and then emits to the main thread when the signal has actually changed? Sounds like a horrible hack though.

I think we're going to end up having to make the play/pause and other similar functions just lifted IO inside of a signal gen monad, that take values, not signals. It's unfortunate but I don't think there's a way to do this nicely with Elerea.

[veryrandomname]

I don't think you can just lift IO functions in a signal gen, if you do 'start' on a signal gen, all the IO actions are done once, to "generate the signal". after that only the callbacks embedded in signals are called if sampled. I also think signals have to be either sampled always or never, because Signal(Signal a) seems to be impossible. Without Signal(Signal a), one could only do helper a b c = if c then a else b. where the signals a and b would be sampled always. (I don't know why, just tested it.)

[schell]

Have you looked at all at Euterpea? It's not traditional and is more of a synthesis library, but an audio module written in it may be an option...

https://github.com/dwincort/Euterpea

[z0w0]

Thanks for the link. I certainly think a DSL for audio state, in a similar manner that Element represents render state, will be our best bet. I don't think I'll use the library internally, but I'll certainly look to it for inspiration.

[funrep]

Haven't read it but skimmed through it and saw the title "Reactive Functional Reactive Sound Engine", might be relevant. http://ocharles.org.uk/blog/posts/2013-08-18-asteroids-in-netwire.html

[z0w0]

Looks pretty good actually. Still doesn't solve exactly how to support playing music instead of audio chunks.

[Smurf]

I've been playing a bit with music and sounds in Haskell and come to the conclusion that the SFML Audio module is probably the easiest way to quickly get audio playing in a nonblocking way. The SDL-Mixer requires its own thread via forkOS and lacks a lot of features.

The PortAudio implementation is buggy and gave me a hell of a lot of trouble often saying that files didn't exist, and required manipulating buffers directly which seems kinda iffy. Not to mention callback based playback is very inefficient and every call to poke from the buffer will block the sound playing.

It looks like the sfml-audio package in hackage is abandoned (the company of the maintainer is now defunct), but the bindings are complete! I've started working on porting some of the low level features of it (panning, getting duration, and seeking in a sound to name a few) as well as implementing a high-level mixer. There's also a complete SFML binding, but it's also abandoned as its maintainer and the maintainer(s) of SFML had some disagreements.

[z0w0]

We're probably going to need a thread for handling audio state changes anyway (unless the main engine tick does it, still haven't decided). I would prefer to use SDL solutions. I'm also not a fan of the SFML project and its developer in general, to be honest.

[Smurf]

 I'm also not a fan of the SFML project and its developer in general, to be honest.

Seems to be a common theme I'm encountering. :)

Just thought I would share my recent experiences over the past week or so on the state of audio in haskell. I do think SDL2 bindings can fix some/all of these issues and would gladly use them if bindings were available as I'm too inexperienced to make my own.

[veryrandomname]

Probably important, annoyed me enough already: With the current sdl bindings the loaded audio files get garbage collected, if not explicitly used inside haskell: http://stackoverflow.com/questions/12443843/sdl-mixer-audio-stops-upon-starting-reactive-banana-input-loop

[z0w0]

Thanks for the info. Looks like it'll definitely affect this if we end up using SDL_mixer (which I don't see why we won't). There'll probably have to be a concurrent thread handling audio anyway, so we could probably just touch the foreign pointers every tick like recommended. That is unless the SDL2 bindings end up fixing this.

[veryrandomname]

Meh. Is there a Haskell FFI of SDL_mixer for SDL2.0? I can't find one.

[z0w0]

Not yet I'd imagine. I'd like to see this done soon but I'm working full time until mid February, although I'm happy to put a bounty up for getting SDL2_mixer going and some concepts for the audio system being developed. Just really want it to be consistent with the current graphics API, i.e. composing some abstract Audio object.

[TheLostLambda]

I think the idea of having an abstract Audio object is a good idea, but that means it needs to be actually implemented in some sort of engine backend.

Should the audio be handled by the current Engine type (i.e. should we add a playSound function to the engine typeclass similar to render) or would it be preferred to separate the graphics and audio engines?

Should we force the incorporation of audio into the existing Engine typeclass, or should we create a new AudioEngine typeclass?

[TheLostLambda]

Also, I haven't played with it any yet, but at first glance it looks like the SDL2 package has a fairly robust audio implementation now (SDL.Audio).

Now we just need to figure out how to make it play nice with Elerea...

[z0w0]

I think the audio should be included in the typeclass. I also think we'd want to use SDL2 mixer

[TheLostLambda]

Do you mean this, as opposed to this? Also, if you don't mind me asking, what is the rationale for that choice?

[z0w0]

Correct me if I'm wrong - I don't think SDL.Audio supports loading audio files. It's just allows for raw audio device manipulation and then mixer lays additional functionality over the top, including loading audio formats.

[TheLostLambda]

Haha, yes you are right! I suppose that is what I get for just glancing over the Haddock... Okay, I'll look some into SDL mixer. Thanks for filling me in! I'll read over docs more carefully in the future!

[z0w0]

Don't worry, my first instinct would also be for those features to be part of the SDL audio library :)