Figure out the sound playing mechanics

[tonihele]

Figure out the sound playing mechanics... Yes.. Currently we have the means to play MP2. We know how to extract them from the SDT files. But it is not as simple as that.

In game data many things have property Sound Category. I think it points to a pair of MAP files; BANK.map and SFX.map. Some kind of playback events. The SDT is just an archive. It doesn't even seem to be a playlist in itself.

We can currently also load BANK.map files, but not fully understand them. SFX is quite a pain...

We could convert these to SPIFF format. Special events could be just special URLs that we understand in application. This would also be still a valid SPIFF.

[tonihele]

We play all the MPEGs used in the game without any problems. Also we read all the mentioned .MAP files. But the contents of those aren't fully understood.

[belgianguy]

I remember seeing some references to SFX in DKII.exe e.g.: SFX.map.BANK.map........SFManager...SFMAN32.DLL.SoundFont

It's called the SoundFont Master Manager, and is made by Creative Technologies.

Digging deeper, assuming the .map files are related to the SoundFont Manager, I found this: https://github.com/kxproject/kx-audio-driver/blob/master/h/sfman/SFMAN.H

Maybe this could shed some light on further unknown variables.

Maybe we can open them with a tool? http://support.creative.com/downloads/searchdownloads.aspx?strstring=sfbm_pcapp

So I found a SoundFont.sf2 file in the DKII Sound directory, and investigated it with Swami (I'm on Linux, they don't seem to have a Windows version).

The sounds herein are limited and I don't know if they are used in DKII, I at least don't recognize them, maybe they're leftovers or just never used.

[belgianguy]

Ugh, an update, so I set off to dig around in the patches (extract self extracting EXE with 7-zip and dig about the cab files with unshield), but while I do see some reference to the creature salamander SFX and BANK, the contents after it don't make sense to me, so that foray was abandoned.

Then I thought of loading the demo, which should be different enough from the actual released version. Turns out the demo doesn't even contain a salamander character. So no luck there.

Now I'm sorting out the different values each part of the SFX can hold, so I can create custom SFX files and feed them to DKII, and use a button of some sort to test the change in effect against.

As we're probably looking from some sound parameters, maybe the QMixer interface might yield some ideas:

http://svn.gna.org/svn/warzone/tags/1.10a/lib/sound/qmixer.h

[belgianguy]

The demo comes with an a3dapi.dll, seems to be Aureal 3D sound, yet another pathway to explore, anyone know if a3dapi.DLL was succeeded by AWE32MAN.DLL ?

http://forums.steampowered.com/forums/showthread.php?t=3217448

[belgianguy]

Ingame Audio options on the Homescreen:

• Sound quality: Low/Medium/High
• Speaker configuration: (Headphones/Stereo)
• QSound checkbox (yes/no)
• Volume master
• Volume voice
• Volume music
• Volume SFX --> this one in particular is interesting, and is what I have been digging for in IDA.

In IDA the switch the GUI uses is called "SFXSwtich" (yes, it's misspelled as Swtich).

This might give some extra pointers for what to look for in IDA: https://wiki.mumble.info/wiki/HackPositionalAudio e.g.: what calls .idata:0066C03

I've generated a gdl file with all references to DirectSoundCreate, but GitHub won't let me attach it.

graph-easy --from gdl --input=graph.gdl --png --output=graph.png

http://timgolden.me.uk/pywin32-docs/directsound__DirectSoundCreate_meth.html

Until I can convert the graph, a screenshot might help: http://imgur.com/a/vZJhk something that points to the SFX dir: http://imgur.com/a/OVxwx

From a first look, the nodes of sub_53BCD0 and sub_5306F0 seem interesting, as a lot of functions point to it, the former even looking like a startpoint of a pipeline of some kind.

[belgianguy]

Maybe it's an idea to look for the CreateFileA Import function, as that can also be used to read files, as a lot of audio functions seem to be located in the "sub_5* region, a CreateFileA nearby there might provide some info. (sub_5FFEC0 ?)

[ArchDemons]

https://cloud.mail.ru/public/CoLJ/LgDzRkXnw my reverse engineering

[belgianguy]

@ArchDemons : I'm still looking around, don't yet understand too much of it :), but thanks!

I was looking at the file mapping part that starts at sub_628C70, and went on to see what File mapping is about. Seems it's done to be able to obtain faster file I/O on Windows than when using the regular file I/O functions. It looks like it maps the files to memory.

dwNumberOfBytesToMap: how many bytes you want to map. Set it to zero and the entire file is mapped. It also has to be a multiple of the granularity value.

http://imgur.com/a/tHKZs

Source: http://www.flipcode.com/archives/Faster_File_Access_With_File_Mapping.shtml

Still trying to get a breakpoint or trace working in IDA 5, so far no luck.

[belgianguy]

And we're getting somewhere, I had some succes with setting breakpoints and attaching the debugger when the game was launching. I found out where the BANK and SFX files are read, now to figure out where that data is stored and what it is used for. http://imgur.com/a/2htxl

[belgianguy]

Switching from the file load route for a moment to better understand the QMIXER sound mixing system.

QMIXER headers are here (can't import it into IDA though): http://svn.gna.org/svn/warzone/tags/1.10a/lib/sound/qmixer.h is no longer reachable https://github.com/Warzone2100/warzone2100/blob/v1.10a/lib/sound/qmixer.h is the new location it can be found at.

And I stumbled upon another C++ application that used the QMIXER API, could be useful to see parameters used: http://read.pudn.com/downloads106/sourcecode/graph/436093/SRC/H_WIN/SOUND.QMX__.htm

[belgianguy]

Results so far of stepping through the QMIXER stuff: http://pastebin.com/1PTEiZXs

It has 12 channels it can mix, its samplerate is 22050, it can enable and disable the channels, it can set volume and panning, it sets source position (x,y,z) and listener position(x,y,z) and listener orientation and distance mapping (how far from the source the sound can be heard)

For the moment I think the SFX files mostly contain float values, often in a QMIXER struct.

for float conversion: http://www.h-schmidt.net/FloatConverter/IEEE754.html

e.g. the sequence 3F 80 00 00 translates to 1.0

Other sets: 3F000000 = 0.5 40400000 = 3.0 40000000 = 2.0

[belgianguy]

Tinkering on, http://proger.i-forge.net/Hacking/Stage%20Once/Part%203.html how to debug a CreateFile call in IDA and see where its contents goes.

[ArchDemons]

some interesting links

• https://sourceforge.net/p/mplayer-ww/ffmpeg-ww/ci/1ae2c5f2e7f3b74827f705811c87b22fd59bd0af/tree/libavformat/electronicarts.c
• http://forum.xentax.com/viewtopic.php?t=2041
• https://code.videolan.org/libav/libav/blob/efec3bc65a5a1ea92de2ba6229c6bef43887d094/libavformat/electronicarts.c

*.sdt - Electronic Arts Sound Data File. I don`t found structure, but upper urls can help us. Extracted Wav files not playable at all. sdt file can be easily extracted by DragonUnPACKer. Maybe there full structure of file

DK2 sound has file SoundFont.sf2 - SoundFont 2 Sound Bank. Opens by Awave Studio e.g.

[belgianguy]

Hi Arch, I could not yet find a suitable structure for the sound file mechanics.

I do however, have figured out where it stores them in memory using http://proger.i-forge.net/Hacking/Stage%20Once/files/Stage%20Once.pdf

CreateFile (this takes a human readable string of the filename) and after executing (next line breakpoint) it will have saved a file handle in the EAX register. Write both filename and the assigned handle down. The application will refer to that file using that handle, not its name.

Then you move on to the ReadFile functions, where you'll see the handle used, and also a parameter of how many bytes it should read. the lpBuffer is then the actual memory location where this data resides.

Now it's off to hardware breakpoints to see what tries to read that data that was stored there.

SFX reads as follows: 20 bytes, 8 bytes, rest until EOF. The first 28 bytes are the header, the rest is the interesting data, which is where the breakpoint should be I assume.

[ArchDemons]

All that we know.

https://github.com/tonihele/OpenKeeper/blob/master/src/toniarts/openkeeper/tools/convert/sound/SfxMapFile.java

https://github.com/tonihele/OpenKeeper/blob/master/src/toniarts/openkeeper/tools/convert/sound/BankMapFile.java

But what does all bytes is mean ?

[belgianguy]

Hi @ArchDemons, that's not clear for the moment, IDA allows me to see where the data gets loaded and when another function tries to load that data, but the operations accessing it are very vague, and I can't see what the game uses it for (yet).

So, to maybe get some insight, I started comparing the smallest of SFX files (containing only the bare essentials in my mind), and they are quite alike. http://imgur.com/a/gY0on

I was thinking of maybe changing some of the red marked values by those of another file, and then load the game with those and actually hear a difference.

[belgianguy]

After some internet sleuthing, I stumbled on a CGDC document of 1998, which stated that at one point in time QSound released a free version of their mixer software, called QMDX, their own claims are that QMDX and QMixer share identical APIs. It used to be available at http://www.qsound.com/gamedev, sadly this part of the site has long been decommissioned.

But that doesn't mean it wasn't saved somewhere, WayBackMachine to the rescue!

https://web.archive.org/web/19990508143218/http://www.qsound.com/gamedev/download.htm#disclaim2

The green "Download the FREE SDK" points to a file qm413sdk.exe, and it'll only work on 32-bits Windows (or WINE in my case).

It contains a PDF file with its API explained, this might provide some insight in ranges of parameters etc, something to look for in the binary SFX files.

It also contains sample C++ code samples and the QMixer.h file.

PS: I had to download it two times, the first time it would not install correctly.

PPS: Initially I had the 5.0 version of the EXE, but DKII uses 4.13.

[ufdada]

That's some nice research you did there.

[belgianguy]

Right, not much new to share, but I think I at least identified 3 float sequences of 4 bytes each.

The "CD CC CC 3D" sequence got my attention, as it looked so peculiar. I entered it into a convert website to convert it to a float: http://www.h-schmidt.net/FloatConverter/IEEE754.html, but that gave me a non-sensical number, so I just googled that sequence.

That pointed me to https://forum.juce.com/t/coreaudio-i-o-rounding-float-32-vs-fixed-24-bit/15094, where I then got the idea that I probably needed to reverse the bytes if the poster was right.

Turns out, the float representation of its inverse (the way the CPU reads it) is 3DCCCCCD, which maps to the float value of 0.1, immediately following it is 00000040 (40000000) which translates to float 2.0, and following that is 0000003F (3F000000), which maps to 0.5.

So we have (0.1, 2.0, 0.5) (or 0.5, 2.0, 0.1 ?), at least I think it's too much of a coincidence for now :).

Other values found at the CD CC CC 3D position e.g. door_barricadeSFX.map, (9A 99 99 3E) 0x3e99999a (== 0.3), ambienceSFX.map (00 00 80 3F) 0x3f800000 (== 1.0), terrain_claimed_pathSFX.map (CD CC 4C 3E) 0x3e4ccccd (== 0.2) and creature_firefly (00 00 00 3F) 0x3F000000 (== 0.5)

Update:Maybe I identified something after all, see this:

UINT QSWaveMixSetDistanceMapping(HQMIXER hQMixer, int iChannel, DWORD dwFlags, const QMIX_DISTANCES* lpDistances);

typedef struct { UINT cbSize; // size of structure (needed for future versions) float minDistance; // sounds are at full volume if closer than this float maxDistance; // sounds are muted if further away than this float scale; // relative amount to adjust rolloff } QMIX_DISTANCES;

At first I dismissed it because I didn't see the cbSize defined in the binary file, but I remember seeing it hardcoded in IDA, so they just don't set in this file because it never changes (but used a constant instead). From the few times I've seen the distance mapping values pass, the scale always was 0.5.

If we assume the first parameter is minDistance and the second one maxDistance, then we must assert that minDistance <= maxDistance for each SFX file.

Maybe it's the DirectSound equivalent or QMixer's own QSWaveMixSetDistanceMapping.

Another thing we can test is to set the 0.2 value for minDistance in terrain_claimed_pathSFX.map to 1.0 (like ambienceSFX.map has) and hear if imps claiming land are now audible from much farther away.

[tonihele]

All the values and files in DK II are little endian. Maybe the struct can be mapped to some of the entries: https://github.com/tonihele/OpenKeeper/blob/master/src/toniarts/openkeeper/tools/convert/sound/SfxMapFile.java#L29

[ArchDemons]

@belgianguy, can you upload QSound SDK 5 anywhere and send link?

[ArchDemons]

@belgianguy, you found this values as I understand https://github.com/tonihele/OpenKeeper/blob/master/src/toniarts/openkeeper/tools/convert/sound/SfxMapFile.java#L69

SfxMapHeaderEntryHeader {
    ...
    protected byte[] unknowns = new byte[12];
    ...
}

[ufdada]

@ArchDemons The qmixer in dk2 was actually version 4.13 (in some cases) or 4.14 (at least in mine). You can find 4.13 here: https://web.archive.org/web/19990508143218/http://www.qsound.com/gamedev/qmdx-files/qm413sdk.exe

[ArchDemons]

@ufdada, download broken at 1024 kb

[ufdada]

Worked fine for me

[belgianguy]

@ArchDemons

PS: I had to download it two times, the first time it would not install correctly.

[ArchDemons]

@tonihele, thanks. Just leave it here https://cloud.mail.ru/public/7fib/3FKp2RSnb

[belgianguy]

Here is the list of all (X,Y,Z) floats in the *SFX.map files in the sfx directory for the 12 bytes I recognized:

http://pastebin.com/Ps0ccxxm

The program that created it:

http://pastebin.com/nLDVQdx7

Ugh, time for sleep, X=minDistance, Y=maxDistance and Z=scale

Another function that might yield some insights is QSWaveMixSetChannelParams.

[belgianguy]

Could it be the 4 bytes starting at decimal pos 64 in the SFX files has something to do with their priority? These are the outcomes for said sequence http://pastebin.com/dv7CPVEU The value's range is [0, 6000]

There exists QSWaveMixSetChannelPriority, so we could check if it's calls are all inside the range defined (or see if any peculiar values pass along, like 3999)

I extended my earlier program: http://pastebin.com/anENPeKr

[ArchDemons]

templates.zip

Here templates for 010 Editor. Strongly recommended use its. SFX.map not fully have real structure, but parses fully.

@belgianguy, Use code of our project to find real structure. I made some re-factor sound part of code soon.

[ufdada]

@ArchDemons I run a repo for 010 templates already: https://github.com/ufdada/dk2-tools/tree/master/Formats Do you mind if i update my templates with your changes?

[ArchDemons]

@ufdada, if my files is better - of course yes!

[ArchDemons]

If I correctly understood, I found a connection between the SDT, SFX and BANK files. I lay out my achievements tomorrow, if the assumptions are confirmed

[belgianguy]

Exciting! Very curious!

[ArchDemons]

SfxMapEEntry.index - global index of sound part in game.

SfxMapEEEEntry.index - 1-based entry id in *.SDT file

SfxMapEEEEntry.archiveId - 1-based archive id in *BANK.map file

Others undecoded values in this structures may sets sound volume, starting and ending time to play, repeating...

[tonihele]

global index of sound part in game.

^what this means exactly? Could this be the event id? Enum?

So one would start by finding the _sound_category_SFX.map and _sound_category_BANK.map files. When wanting to play a sound from the index (SfxMapEEntry.index), get the entry from BANK.map (SfxMapEEEEntry.archiveId) to find out the actual sound archive name. Then play the sound (SfxMapEEEEntry.index) from the SDT archive. Am I even close?

[ArchDemons]

what this means exactly? Could this be the event id? Enum?

Maybe event id. This index unique (not exactly unique.... it`s repeated in some files, but sound data identical. I am not compare this them, but I think. We should check it).

This index used to check whether it is already loaded into the game e.g. Maybe something else. Maybe it in some undecoded values in kwd/kld files.

• sound category show us what *bank.map file we must load.
• Bank.map tell us what SDT file we must load
• Sfx.map tell us what sound we must play in SDT file and how we do that.
• By SfxMapEEntry.index we can find any sound (or group of sounds) loaded in game.

@tonihele, find "user-defined id" in qm413api.pdf. I think this is it

[tonihele]

Maybe it in some undecoded values in kwd/kld files.

This would make a lot of sense to embed these to ArtResources, even more so to the animations itself (the model data). And with static objects/tiles like HeroGateFrontEnd it could be in the map record.

[tonihele]

Here, just for fun & debugging: https://github.com/tonihele/OpenKeeper/tree/noisy-dungeon

We now have main menu "music" :)

[ufdada]

ArchDemons did some great work on this

247

284

what's still open here?

[tonihele]

Probably HW & HD sounds difference, usage-wise. And I would think that the animations are synced with the sounds... Maybe, so answer lies in KMF or the sounds have build in delay or some delay parameter etc.

[ufdada]

Well the artresource should have values for left footstep frame and right footstep frame, so they might be synced with that?

[tonihele]

Possibly. Wet feet also leave footprints so that might at least be synced... We could study maybe the imp digging anim, I can't imagine its sounds being synced with those, at least literally as that description.

[tonihele]

This is my favorite suspect... (KMF's animControls):

[werkt]

I realize this is very old at this point, but http://www.synthfont.com/sfspec24.pdf has the full specification for the soundfont behavior, and all the assumptions I saw in this thread are correct: banks contain pcm samples, SFX contains the use of those banks in effects.

[belgianguy]

From my earlier research, I think the http://pastebin.com/Ps0ccxxm results mapped to the 'distance mapping' concept of sound, meaning the amount of volume a listener would experience depending on their location relative to the source and the specific properties of the source of the sound.

A detailed explanation of the QMixer distance mapping can be found in the QM413API.doc, (which can be found after installing qm413sdk.exe) from pages 23 up and until page 27

As I thought these were positions earlier, the results have a non-sensical prefix, as they all three denote float values: X: minDistance Y: maxDistance Z: scale

minDistance:

As far as I understand it, the minDistance denotes the difference in "sound energy". The value relates to the normalization of sounds (digital range of computer speakers), where sounds with a large (e.g. 1.0) minDistance would be considered "louder" in mixing, whereas smaller minDistance (0, 0.1, 0.2, ...) values would be considered "softer" in mixing. The documentation says to compare a jet engine with a mosquito.

minDistance affects the volume in the following two ways:

• minDistance defines the boundary in which the channel volume is unaffected by the range (range = distance between source and listener). For sound sources with high energy: These get their maximum volume at a reasonable distance away from the listener For sound sources with lower energy: You'll have to get close to hear these sources at their maximum volume

• minDistance affects the roll-off distance Moving away from a source with a high energy should be audible longer than moving away from a source with lower energy, so minDistance also affects the slope of the attenuation (together with scale) of the sound.

maxDistance:

The volume would be 0 if range > maxDistance, eg, the source is not audible past maxDistance.

The problem is that there indeed are *sfx.map files with a maxDistance (Y) of 0.0 in the results, so that either means that they do not denote maxDistance, or these sources are silent/muted/unused or perhaps set dynamically by the game.

e.g.

creature_jackinboxSFX.map X:0000803F 1.0 Y:00000000 0.0 Z:00000000 0.0

creature_maidenSFX.map X:0000803F 1.0 Y:00000000 0.0 Z:00000000 0.0

effectsSFX.map X:CDCCCC3D 0.1 Y:00000000 0.0 Z:00000000 0.0

object_fe_mapSFX.map X:0000803F 1.0 Y:00000000 0.0 Z:00000000 0.0

object_bedSFX.map X:CDCC4C3E 0.2 Y:00000000 0.0 Z:00000000 0.0

options_musicSFX.map X:0000803F 1.0 Y:00000000 0.0 Z:00000000 0.0

options_speechSFX.map X:0000803F 1.0 Y:00000000 0.0 Z:00000000 0.0

trap_jackinboxSFX.map X:0000803F 1.0 Y:00000000 0.0 Z:00000000 0.0

scale:

Affects the rate of rolloff/attenuation, eg how fast the volume (curve) drops between minDistance and maxDistance, scale > 1 would cause the volume to fall sharply, whereas scale < 1 would cause the volume to fall gradually

Formula that is applied:

applied_volume = volume / 1 + ( range / minDistance - 1) * scale, where minDistance < range < maxDistance

following from earlier means that for range < minDistance applied_volume = volume, and range > maxDistance means that applied_volume = 0

Other peculiar results:

Some sfx.map files have a minDistance of 0 (X:00000000 0.0), which would declare them as very soft sounds but a maxDistance and scale of 1.0: These seem to be all (level) speech (presenter/Horny) related, therefor it is possible they are not seen as sound sources that appear the game world on the same level as the rooms/traps are, and thus are affected by different rules: perhaps these are used by the UI (map screen/level selection) where no extra mixing is needed or they have their own channel:

speech_level11cSFX.map X:00000000 0.0 Y:0000803F 1.0 Z:0000803F 1.0

(and a lot of others, left out for readability)

Interesting tidbit, which might detect a 'volume' value for an 'unknwown' perhaps:

• Volume range in QMixer: https://doxygen.scummvm.org/d9/d41/qmixer_8cpp_source.html, here it states that QMixer volumes go from 0-32767

Questions that arose during this research:

• Are the animations that creatures/traps/UI play separated into frames? Could it therefor be the *sfx.map file contain timing data so the game engine will start to play a certain sound that aligns with the visible movement (denoted by a specific number of a frame of the animation) ?

[tonihele]

Are the animations that creatures/traps/UI play separated into frames? Could it therefor be the *sfx.map file contain timing data so the game engine will start to play a certain sound that aligns with the visible movement (denoted by a specific number of a frame of the animation) ?

This... Yeah... With this we could really start to implement the sounds. I think it could be either way. The frame info is stored on the animations, or on the sounds... This would be my guess.

[belgianguy]

Well the artresource should have values for left footstep frame and right footstep frame, so they might be synced with that?

I digged through the SDT files using MAStudio2002's SDT Editor subprogram and found out that some creatures have their own 'walking sounds', always one or or more files containing 'ft' (for foot I suppose) and ending in mp2. But there is also a generic 'walking sound' that is located in MiscCreatures.sdt, plus water related movement seems also to be saved in the same manner.

Creature SDT with "dedicated" 'walking' sounds (filenames containing 'ft' and which sound like a step):

• Skeleton
• Imp
• Black Knight
• Giant
• Mistress
• Reaper
• Salamander

As the screenshot refers to a frame of the animation, is there a way to render say the imp's walking animation as a sequence of images, and then see if we can find references to the frames where one of their feet touches the ground?

Generic walking sound (MiscCreatures SDT):

• files containing 'genfoot'
• files containing 'WADING'
• files containing 'FT IN WATER LA' (where LA I think stands for Large, as it sounds as if something heavy steps in water)
• files containing 'FT IN WATER SMA' (where SMA stands for Small, as it sounds as if something light steps in water)
• files containing 'maifoot' (although I could not play these)

1. Are these 'walking' sounds only heard when in 'Possession' mode? Then maybe they are not animation related, but player input related? (Moving a distance/pressing a move button). I loaded the game and for all creatures except for the Reaper the walking sounds on solid terrain are only heard when in Possession mode. When standing still the sound is not played, when keeping walking forward into a wall the walking sound is still played. So I wonder whether tapping the movement key would also trigger this sound, or if you have to keep a movement key pressed for x seconds for the sound effect to play.

2. Is there a sound of which we are sure that it does not start at the same time as the animations? Then maybe we could 'poison' its SFX.map file with some other values and see whether this affects it, to zone in on it (eg making it start earlier or later). I will investigate if the Reaper has its footsteps played when used by the player in-game (seeing the end-game crystal pickup as separate and possibly hardcoded.)

As the Black Knight has a walking sound a a regular Knight does not, perhaps comparing both *SFX.map files might yield some insights (as I hope the developers were lazy and reused the same structure). If I read the SFXMap code right, The Black Knight has 37 SFX mappings, and the Knight has 36.

3. Also, animation wise, for the Reaper 'melted terrain/flames' appear under his feet when he walks/stands. And I do think creatures walking through water and blood also leave blue or red footprints.