[audio] Chiptunes support

[kd7tck]

Wondering if there was any support for this. I have been looking into it and am going this route: https://github.com/kd7tck/raylib/commit/8a16d76a6554bbf06969dc9d6bbae715c6f36a75

Wanted a little feedback, wondering if better libraries are out there.

[raysan5]

Hi @kd7tck, I really like chiptunes-based module music (MOD, XM, IT, S3M) and I already was looking for some way to add support for it to raylib.

I checked the following libraries:

• libxm (https://github.com/Artefact2/libxm)
• libxmp (http://xmp.sourceforge.net)
• DUMB (http://dumb.sourceforge.net/index.php)
• libmikmod (http://mikmod.sourceforge.net)
• libmodplug (http://modplug-xmms.sourceforge.net)

Actually, all them are quite good and can be integrated with raylib but one of the raylib key design objectives is keeping dependecies to the minimum, I don't want external libraries unless they can be directly integrated into raylib basecode (like stb libraries). For that reason I didn't integrate any of those libraries into raylib... yet.

The only one in this list that is quite small and I think could be ported to single-header to be added to raylib is libxm.

[kd7tck]

Both DUMB and libxm require one external lib. However DUMB is ANSI C(DOS), and libxm is C11. Are you planning to add support for the modern version of C?

I am thinking about changing over to XMP, it seems easier to port into yours. DUMB works too but it does have these weird UNIX, DOS commands bundled in to maximize backwards compatibility.

[kd7tck]

I want to make this my next project, I will make a stb_chiptunes header file. I plan to get a proto up and running in the next month or two. I hope to have an in game synth and a player that is incorporated directly into the playmusicstream function. The game I am working on is nearing completion and I totally forgot about music until the last minute.

I intend to make Raylib library the highlight of my game, the raylib logo will even show at the beginning. Maybe you could add a simple chiptune to the logo as it runs for 1.5.

By the way, thank you for making raylib.

[raysan5]

Hi @kd7tck, that's fantastic! I've been looking for a single-header chiptunes file for long time but I couldn't find anything! As I told you, the closest approach was porting libxm to single file...

About a chiptune in the logo animation, it would be great, I already added a discrete beep in some of my games: http://www.raylib.com/games/just_do.html Undoubtly, it can be improved with some catchy chiptune!

As always, glad to hear you are enjoying raylib! Please, keep me updated with your progress! :D

By the way, one of my side raylib-based-project is a port of DrPetter's sfxr tool to raylib, using raylib's module raygui. Here it is a screenshot:

I'm still working on it, wav output works but sound playing still has some problems. I will upload the code as soon as I make it functional; together with some other raylib-based tools.

[kd7tck]

Here is an early draft for the XM format lib. https://drive.google.com/file/d/0B2jDwEP2AbataWgyaWhHd0hYQU0/view?usp=sharing

[raysan5]

Hi Joshua, it looks great!

It could be integrated with raylib audio module in PlayMusicStream(). Really promising! :D

Keep up with the good work!

By the way, better rename it to jrxm.h, stb stands for Sean T. Barrett libs. :P

[kd7tck]

Wondering if it was possible to change audio api to allow playing wav structs.

Something like:

typedef struct AudioContext {
    unsigned char channels;          // 0=mono
    unsigned char bitsPerSample;     // 16 is default
    unsigned short sampleRate;       // default is 48000
} AudioContext;

void startRawAudioContext(AudioContext ctx);             // for playing raw audio samples
void stopRawAudioContext(void);                       // Stop raw audio context
void pauseRawAudioContext();                          // Feeds zero's into queue
void unPauseRawAudioContext();                     // Resumes raw audio context play
void pushRawAudioData(Wave* data);      // Pushes data to be played inside of raw audio context

The data would be copied onto a Queue, ensuring smooth playback with multiple pushes occurring back to back.

[kd7tck]

There is a slight problem with getting the lib to work. We need 32bit floating point support in wave. I tried loading the raw data into a giant array and converted it into a wave struct. When I loaded it into a sound it played nothing at all. When I reset the bps to 16 or 8 it played static.

[raysan5]

Hi @kd7tck, I don't undestand clearly what you mean with the raw audio data playing... Do you refer to streaming audio data in a similar way of what we do with OGG?

If that's the case, why not keep using the Music struct and PlayMusicStream() - UpdateMusicStream()?

About the 32bit floating point support, I think it should only be some configuration parameter for OpenAL Soft library, I'll try to take a look ASAP.

[kd7tck]

The play raw, will act as a plug and play system. It should allow anyone to attach third party sound libraries into raylib. The ability to push raw sound data into a sound queue is very helpful for testing new sound libraries.

[raysan5]

Sorry, my bad, still not understanding it clearly... Do you mean an streaming service for multiple audios? I mean, sample data loaded into the queue every certain time and then played by OpenAL system? That's the way PlayMusicStream() - UpdateMusicStream() work.

Right now, when calling PlayMusicStream(), it checks file format, opens the stream, init two internal buffers (the queue) and UpdateMusicStream() keeps feeding the queue with raw audio samples coming from the OGG file. Same process could be used with any other kind of audio source.

If that's the idea we are talking about, I like it... but I'd like to discuss a bit more the implementation way.

[kd7tck]

When prototyping new audio libs sometimes converting everything in real time over to ogg files or wav files is really time consuming. The ability to simply push raw audio data allows you to skip that other step and just jump right in with pushing the data to be played. For example I could push the raw output of the sin function, and it would produce a simple soundwave. By modulating the period of the sin wave I can change the pitch.

At the very least offer a muxing feature where you can mux 2-3 music/sound streams to be played concurrently. The clipping would need to be calibrated to the entire track's average volume. This of course being one of the hardest things to calculate.

[raysan5]

Ok, I understand. At this moment audio data is already streamed by chunks that are picked from the ogg file, those samples could be generated from a sine wave and injected into the buffer in the same way... but it's true that right now those functions are not prepared to consume custom raw samples. Neither to say about concurrent music playing, not supported at this moment.

Right now Music struct is directly linked to OpenAL playing system and OGG file. It would be desirable to have some general struct... I imagine that's what you propose with struct AudioContext, right?

[kd7tck]

The testing of the xm lib is nearing completion. There are still some problems with clipping and audio not outputting for certain xm files. I hope to have the final version out by next week.

[raysan5]

Great @kd7tck! It sounds fantastic! :D

[kd7tck]

Alright, been doing lots of testing and found out we can either resample to 16bit or use the openal extensions for 32 bit floating point support:

#define AL_FORMAT_MONO_FLOAT32                   0x10010 
#define AL_FORMAT_STEREO_FLOAT32                 0x10011

I will be posting a patch for the lib in the next couple of days; I just wanted to give a heads up. Where do you want the header files dropped?

[raysan5]

Hi @kd7tck ! Great to read that! Really expecting to see your progress! :D

How many new header files you have? For the moment, you can drop them into the main src directory. Later on we can look for a better destination (maybe an external directory inside src to compile all single-header libs used by raylib).

[kd7tck]

For now only one that fully works, it so happens it is the one format my game uses. XM seemed the logical choice since milky tracker was so easy to learn. You can find the lib under my jar folder on github. I plan to have the jar_MOD version ready in the next week, I am still having trouble with that one. There are too many versions of the file format, and only one of the specs can be found. I am still searching for the older 4/15 spec, any help on that would be appreciated.

[raysan5]

Great work @kd7tck! I agree that support for XM is the logical choice.

About MOD files, it's an old format and it has changed a lot with time. I tried to look for some specs on the format, don't know if any could help:

http://textfiles.com/programming/FORMATS/modform.txt - Amiga ProTracker MOD format http://www.textfiles.com/programming/FORMATS/mod-form.txt http://textfiles.com/programming/FORMATS/modulesg.txt https://greg-kennedy.com/tracker/modformat.html https://groups.google.com/forum/#!topic/comp.sys.acorn/_bQ_ZN9moxk

I also found a MOD player project in GitHub that could be useful for reference: https://github.com/jfdelnero/HxCModPlayer

[kd7tck]

I just updated my lib with a new remaining samples calculation. Hopefully it can be plugged in and work with your music system.

[raysan5]

Amazing work @kd7tck! Very clean integration with audio module, I like it! :)

[kd7tck]

The audio.c file is still not updated to reflect the new remaining samples calculation I just added. That new function should allow for the calculation of remaining number of samples.

[kd7tck]

Testing the MOD player is getting harder, the api for testing audio needs to be implemented.

Here is an updated audio api for testing samples:

typedef struct AudioContext {
    bool channels;                     // 0=mono, 1=stereo
    unsigned char mixChannel;       // 1-16, each mix channel can receive an audio stream
    unsigned char bitsPerSample;    // 16 is default
    unsigned short sampleRate;         // default is 48000
} AudioContext;

void startRawAudioContext(AudioContext* ctx);  // for playing raw audio samples
void stopRawAudioContext(AudioContext* ctx);  // Stop raw audio context
void pushRawAudioData(AudioContext* ctx, void* data, unsigned short dataLen);  // Pushes data to be played inside of raw audio context, raw audio queue auto pauses when no data is pushed.

I wanted some feedback on this new design.

[raysan5]

Hi @kd7tck! I think that audio module requires a redesign to adjust to new features and, in a future, find a replacement for OpenAL (probably libsound.io).

You can add the new struct and functions if they are required. Just for consistency with other raylib parts (loading and unloading structs), maybe functions signature could be:

void InitAudioContext(AudioContext *ctx);
void CloseAudioContext(AudioContext *ctx);
void UpdateAudioContext(AudioContext *ctx, void *data, unsigned short *dataLength);

or maybe, considering every AudioContext for an audio file:

AudioContext LoadAudioContext(void);
void UnloadAudioContext(AudioContext ctx);
void UpdateAudioContext(AudioContext *ctx, void *data, unsigned short *dataLength);

[kd7tck]

Now that the API is down, I will spend a week or two testing out how to fully implement it and wrap the Music system within it.

[raysan5]

Hi @kd7tck! Thank you very much for the effort and the hardwork! Those new addition are great! :)

[kd7tck]

I have been playing around with libsoundio the last couple of days and I don't like it very much. I prefer openal_soft/fmod. libsoundio literally is the most barebones setup possible. Antialiasing, mixing, effects, resampling, and other standard features you would expect from a professional audio lib are missing. I still think you should go ahead with it though, you just need to add in some of the missing functionality.

The game I am working on is audio driven, and the audio backend openAL is just fine for my current needs. If you ever need help with rewriting portions of the audio system over to libsoundio just give me a holler. I still intend to finish up the MOD player after I finish up audioContext. The audioContext system isn't needed for just testing out audio libs. It will allow me to plug in any front end audio system and get everything to work more easily. I also intend to rewrite the music and sound systems to run entirely on top of audioContext.

Just a heads up. I will be gone for a while this next month, so I plan to sprint my way through this last bit and get everything done on my end as fast as possible. I have to meet with my friends and finish up the game in an epic crunch. We are all bringing sleeping bags over and staying until it's done. Lastly I hope you don't mind letting me push all these patches to the library. If at any time you don't want anymore changes, it's OK. I can just fork the rest and use it in my game.

[raysan5]

Hi @kd7tck! Thank you very much for the update! No hurry on finishing the audio system, just take your time, don't worry! raylib is a hobby project and I expect everybody that collaborates on it just have fun doing it, no pressure at all.

Many thanks for the feedback on libsoundio! My idea was finding a small-portable audio library that could be directly integrated into raylib basecode, again, trying to avoid external dependencies but I think it's quite hard to get that, there are too many audio devices and platforms to take into account...

Any patch you send to the library due to the use of it in real productions is welcome, that's the way to improve raylib.

Have fun working on your game and let me see your results once finished! :D

[kd7tck]

Here is the fork where I am working on the new music system: newaudio

[kd7tck]

Wondering if you had any time to look over the fork I made. I wanted some feedback and recommendations.

[raysan5]

Hello @kd7tck!

Excuse me for my late response, I've been out for some days... Additionally, current academic year is finishing in about a month and I'm very busy reviewing projects and preparing exams for my students... A part from that, when I have some free time, I'm working on raylib support for Oculus Rift CV1 (almost ready! :D).

I've been taking a quick review to your additions to audio module, you really did a lot of work! Still some things are not clear to me... If I understand it correctly, now we have up to 4 MixChannels for audio mixing, right? We need those 4 MixChannels to allow MOD playing (4 instruments), right? Independently of MOD files, one Music stream can use up to 2 MixChannels for mixing, so, up to 2 Music streams could be playing at the same time, right? It looks a bit complicated to me (in the way it's designed) but I need to analyse it with more time...

A part from that, there are some details that I'd like to review more carefully (naming, formating, floats-management) but I think you are doing a good job; just continue this way and I'll give you more feedback as the update goes on and gets shape.

[kd7tck]

I have been experimenting with different ways of doing audio mixing in this fork. I have not yet achieved a result that I am pleased with. I hope to eventually hammer away at it until I get there. I will be gone later this week, so I hope to get as much done up until then.

Actually the four channels are needed so you can have any combination of midi,ogg,raw,wav playing at the same time. Up to four channels can be muxed with 16bit samples. If we used floating point it maxes out at around 32 channels. For some reason the OpenAL extensions for floating point are buggy and not working correctly. I am guessing it has to do with some unknown setting that I am not activating.

There is an easier alternative to all of this. We could just get rid of all globals and go to a semi object oriented way of doing this. Where all music objects are malloc'd into pointer types and the end user is required to do all the required safety checks themselves. There would only be about a dozen functions that the end user would have to juggle and use to play a song themselves. It would be easier for me to code that vs this static global system. I went along with it under the impression you wanted the end user to never have to worry about; when to unload the music pointer from memory, ensuring you are not playing too many channels at the same time or check for errors at every step.

[raysan5]

Ok, I see, thanks for the explanation. About OpenAL Soft, probably I'm using an old versión of the library, I haven't updated it in a while. Maybe you can try with latest version but every platform uses its own implementation, so, Android or Raspberry Pi version could not match.

You're right, I prefer the second path, keep things simpler for the user. Despite raylib has grow quite a lot in the last three years, it was primary designed for students with zero experience on programming, to allow them to program games easily, still coding. I prefer to keep middle layers inside the module as static functions to be used by upper layers exposed to the user, even though some globals are required with some internal management; actually, it's the way you're doing it, that's great.

Again, thanks for your effort on the project.

[raysan5]

Hi @kd7tck, do you think we can close this issue?

[kd7tck]

I will be doing one more update to jar_mod in another day. Close it after that, we are pretty much done with chiptunes.

[raysan5]

Closing issue with commit https://github.com/raysan5/raylib/pull/126 after a bunch of updates and redesign of the audio module.

Probably still work to do for future updates and reviews...