caf audio containing Mpeg4Aac

[mistydemeo]

Describe the feature or issue.

I'm looking at a game which uses AAC in a CAF container. I see that AAC in the standard container is supported, by decoding it to PCM ahead of time. CAF looks like it could be a little more complicated within that model because of how it's set up. I'd be up for taking a look at implementing something, even just something basic, but I'm wondering if you'd looked into this yet or had any ideas/preferences for how it should look.

My goal was to see if I could get Critter Crunch to run. This small patch got it from crashing on boot to reaching the title screen, but then it crashes from the panic in the audio code from encountering an unhandled file format in the caf path.

By submitting this issue, I certify that…

• [X] I understand that linking to, discussing or asking for pirated content is not permitted, and will lead to an immediate ban.
• [X] I understand that “Abandonware” and illegal copies of free games are also pirated content.

[hikari-no-yume]

Hi! The audio format code currently isn't as “orthogonal” as it should be. The original formats implemented were linear PCM in WAV and ima4 (Apple-style) ADPCM in CAF (both for Super Monkey Ball), then MP3 was added for Touch & Go, then AAC in MP4 was added for some other title I don't remember. But each of them is implemented differently:

• WAV files can be opened and the linear PCM data read directly. The hound crate is used for this.
• CAF files can be opened and either linear PCM (IIRC?) or ima4 ADPCM can be read. The caf crate is used for this part. ADPCM is then decoded only once it is placed in an audio queue. My own implementation is used for this part.
• MP3 files can be opened, but the entire file is decoded to linear PCM when it is opened, and then this data is presented to the app as if it had opened some file that was in a linear PCM format. The dr_mp3 crate is used for this.
• The situation for AAC is the same as for MP3, except the symphonia crate is used.

You will notice then that we are using a different library for each container parsing and/or audio decoding task. This was due to my suspicion towards large dependencies and wanting to keep things lean. But for AAC I eventually ended up using symphonia, which is quite a big and versatile library that supports many different container formats and many different audio codecs. And now that we have that, using so many other libraries is an unnecessary complication. There's already a patch in Gerrit (by @LennyKappa) that would move to using symphonia for the MP3, dropping the dr_mp3 dependency.

You will also notice that MP3 and AAC are handled specially. This decoding-to-linear-PCM behaviour is (I believe) not what iOS is supposed to do. It was done to avoid implementing variable-length packet reading and decoding, and it works for most apps, but we've already encountered some that are broken by this.

Getting back to the specific case you're interested in: I know symphonia supports CAF and AAC, though I haven't checked if it supports them in combination. So the easiest way to fix this might be to add a special case that falls through to the decode-everything-to-linear-PCM path if a CAF file contains AAC, though this is not guaranteed to work properly for your particular app. The “correct” long-term solution is for us to support variable-length packets in audio file reading, and decoding MP3 and AAC in audio queues, but as always this is limited by bandwidth and interest of available contributors.

[mistydemeo]

Thanks for the quick response! I really appreciate it.

I took a look at symphonia for aac-in-caf, but unfortunately no dice here - symphonia reports "adts: only 1 aac frame per adts packet is supported" when trying to open the CAF in question.

[hikari-no-yume]

Ah I see, that's unfortunate.

One trick I've done before when trying to get a game working is to use a tool like ffmpeg to convert all the audio files to e.g. WAV, but without changing the file extension. Obviously that's not a proper solution, but it may get you past this hurdle for the moment.

[mistydemeo]

Oh, that's a good idea. I was also looking at the code that ignores unknown audio formats, so I could add the minimal detection and move on. Does look like there's one interesting quirk though - the caf crate is passing bytes_per_packet as 0 for the AAC, which blows up some of the later assumptions. I've hacked around it by setting it to a fake value on the understanding that it'll never be used for anything real, but I'm not sure exactly how hacky you'd like the "ignore this" handling to be.

[hikari-no-yume]

0 is the correct value for “bytes per packet” in the case of AAC, it indicates the packet size is variable. We don't have any support for variable packet sizes right now though. I'm not sure how to handle this.

[mistydemeo]

Hmmmm right, that's tricky...

I guess the other possibility would be to extend the "unknown format, do nothing" logic and use that in places where more things need to be skipped over?

I mean I guess the other possibility is just actually implementing variable size packets but I think I'd want to do that later, haha.

[ciciplusplus]

FYI this format and container is also used for some PopCap games (PvZ, Bookworm, etc.)