Ultimately, I'd like to create a modern game engine that uses unmodified game files from Origin+Looking Glass's Ultima Underworld to replicate the original gameplay. This is comparable to projects like ScummVm and various Doom engines. It's also a great deal of work, and there are some things that I haven't figured out yet. I intend this to be a long-term project, probably taking a few years of my spare time to complete. The end result will be an open source game engine for playing Ultima Underworld 1 + 2 natively on modern computers, optionally with improved graphics.
I'm currently developing under Linux. There isn't anything technically stopping it from working on Windows, but I haven't gotten around to getting it compiled there. If you've got make, a modern C++ compiler, OpenGL/GLU, SFML 2.x (I think 2.2 or 2.3 would be the minimum, but I'm not positive), GLM, and one called libtimidity that can be found on SourceForge (but you only need if you're interested in MIDI playback of the Ultima music files).
When the requirements are satisfied, you can build the main engine with make sfml-fixed-engine. There are a ton of other file viewers (for textures, audio, music, creature images, cutscenes, and so on). Generally, any file with a main() wrapped by #ifdef STAND_ALONE_SOMETHING can also be built and used to view files. Those have been more debug+testing utilities, but it can be cool to fire up the AdLib-emulating music player (cd audio; make opl_music; ./opl_music <path_to_uw.ad> <path_to_xmi>) or something.
Everything else having gone smoothly (snerk), sfml-fixed-engine <path_to_uw_dir> will launch the basic level viewer. Most of the other file viewers will tell you what their arguments are. A few were written in a hurry, and might not have usage info (example: opl_music exits silently).
Separate the game into a Model-View-Controller paradigm. Import files to provide the model of the game state and the assets for graphics and audio, to be used in the view.
I can convert the XMI files to MIDI, read the ad-lib instrument definition files, and sequence music into register writes for an emulated YMF262 synthesizer. This also opens the door for sending the music to an emulated Roland CM-32L (MT-32) or to a modern MIDI server, like Timidity. And actually, I can decode the music into MIDI and send that to Timidity (that's the use of the libtimidity library). It sometimes sounds great, and sometimes not. It depends heavily on your patchsets. I think I'll need to do more tweaking and find a way to compare (for example) MT-32 patches with the ones from some specific soundfont, so that I can find the ones with the least-grating differences, to build my instrument mappings (MT-32 didn't use the General MIDI instrument numbers, so I have to remap).
Note: Ouch, I'll make a note that the uw* songs sound pretty bad through the OPL/Ad-lib emulator. Use the aw* files for that.
These are defined in the instrument definition files, but don't follow any format that I've been able to find documentation for. I contacted John Miles, who wrote the Audio Interface Library (AIL) that the game uses, and was informed that sound effects are defined in terms of twiddling OPL2 registers, but he didn't have any further details. The format is refered to as OSI ALE or OSI TVFX (OSI = Origin Systems, Inc), and it is a proprietary format made by Origin themselves.
Dosbox can record register writes to an Ad-Lib card into a file format called DRO, and I've included some code to interpret those. It's been a while since I've looked at it (a couple of years), and it seems like it needs some love to get into a properly-working state.
Having looked at some of the audio code, I've identified the routines that handle the TVFX sound effects, and I've looked at them some. I haven't completely figured out their operation. Here's what I think I've found, though:
Look at audio/kail/ALE.INC. It's an open implementation of OSI_ALE TVFX (the library used for adlib sound effects). The "TVFX" struct shows the structure of the first 0x36 or 0x3e bytes of a TVFX timbre (the larger ones override the default attack/decay and sustain/release values used by the library).
There are 8 sets of values that can be time-varied: frequency, carrier + modulator volumes, feedback levels, carrier + modulator AVEKM, and modulator + carrier wave selection. Each of those basically has a for-loop: initial value, number of increments to do, ending value, and a pointer into a variable-size data table for the next set of these values (or looping back to previous ones, etc). The whole ASDR note life-cycle is covered. It's possible to build the effects to work as "instruments", but I think the Ultima games only use them as sound effects (one-shot things, no separate note-off message, hardcoded amount of time to run)
I can load most of the graphics in the game, including cut scenes, wall, floor and 3d object textures, sprites for in-game items, etc. Foreground/in-hand weapon graphics use a variant on the other graphic formats, with variable-size images and a separate file of offsets for where to render them onscreen, but I've got most of that working (except some of the mace animations; they don't fit the patterns of the other weapons).
I can read the map, load item/character locations, etc. The function of the minimap needs to be documented.
I've got a fly-through view, level-switching, and a few other things written. 3D objects are imported and rendered (the Ankh, barrels+furniture, pillars, etc), but some of the complex shapes are only partially rendered. In addition, there are still some shading, coloring, and object placement bugs with those objects. Some things use placeholder graphics from the game's level editor and haven't been properly replaced and/or hidden from view (traps+triggers, animated overlays, etc). The current engine is based on fixed pipeline OpenGL (1.x). It runs nicely on an old netbook I've got around, which I couldn't claim a few weeks ago. The next iteration will use OpenGL ES 2 and be developed to run on a Raspberry Pi. I'm not using glBegin/glEnd-style immediate mode functions, but I am currently using client-side vertex arrays, but I think it'd be pretty easy to switch to VBOs.
These are actually contained in structs in the executable itself. I can interpret them and output .obj 3D models. The engine's handling is still buggy (see description under the "engine" heading), but basic rendering is working.
Function 0: 2 arguments. The first argument is a palette index from the current nXX file. The second argument is an index into the game's string table (strings.pak is also documented elsewhere). It reads flag bit0, and returns immediately if it isn't set.
Function 1: 0 arguments. I don't know what it does.
Function 2: 2 arguments. Seems to be a no-op.
Function 3: 1 argument. The argument is a number of half-seconds to pause, displaying the current frame. Current frame is used as a kind of hidden second argument. Clears flag bit1.
Function 4: 2 arguments. The first seems to be a frame number. The second seems to be a time in seconds(maybe? I ended up ignoring it in my code and playing at a constant 5fps). It seems to play the frames from current to the first argument, probably over the course of arg2 seconds.
Function 5: 1 argument. I'm not sure what it does, but the argument isn't used in the function. It just clears a bit in an 8-bit flag that controls flow of the cutscene. On subsequent investigation, I think it has to do with "static" vs "animating" cutscenes (a space-saving option in the game config).
Function 6: 0 arguments. Marks the end of a cutscene.
Function 7: 1 argument. The argument is the number of times to repeat from the beginning of the file to this point (the start point is speculation on my part, but cs011 is the only example)
Function 8: 2 arguments. The first argument is a cutscene number, and the second is an animation file number. It instructs the file to load the given cutscene file.
Function 9: 1 argument. The argument is a rate to fade to black at. A higher number is a slower rate (1 fades in 8 steps, 2 fades in 16, 0 is instant-black, etc)
Function A: 1 argument. The argument is a rate to fade from black to the current frame. Same rules apply as in the fade-out function.
Function B: 1 argument. I'm not sure what it does. Argument is a frame number that shouldn't be the "current" frame. This clears flag bit0 and sets flag bit1.
Function C: 1 argument. Not sure what it does. The argument is used as a boolean value. Odd numbers are true. Even are false (i.e. it just looks at bit0 of the value). It sets/unsets bit4 of the flag.
Function D: 3 arguments. First arg is a palette index. Second is text (these two are the same as function 0). Third arg is the number of a .voc sound file to play.
Function E: 2 arguments. Pauses on current frame for different amounts of time, depending on whether audio is enabled. Current frame is used as a frame number arg. arg0 is used as a time if flag bit5 is clear, arg1 is used if flag bit5 is set. I think bit5 is related to whether digital sound is active or not. I chose to use the same timing as in function 3, and it seems to be about right.
Function F: 0 arguments. Plays the "Klang" sound effect (MIDI bank 1, patch 3, using 0-based numbering)
Function 10+: Only available in UW2, and I haven't begun my investigations of that binary.
The file format is straight-forward, but not necessarily only raw WAV data. It's used in its simplest form in UW 1+2, though. Open the file, seek to byte 32 to skip the header. The remaining data is 8-bit unsigned mono PCM (except the final "00" byte that marks the end of the file). In UW1, they're recorded at 12048Hz. In UW2, they're recorded at 11111Hz. The actual details of the format are published elsewhere, so I'm not going to go into detail.
This one isn't documented in the uwformats file, so I decided to figure it out. It's used in the character generation process. There are a number of important indexes that the values refer to, and these are reflected by the orders of some strings in strings.pak.
0: Fighter 1: Mage 2: Bard 3: Tinker 4: Druid 5: Paladin 6: Ranger 7: Shepherd
0: Strength 1: Dexterity 2: Intelligence 3: Vitality
00: Attack 01: Defense 02: Unarmed 03: Sword 04: Axe 05: Mace 06: Missile 07: Mana 08: Lore 09: Casting 0A: Traps 0B: Search 0D: Sneak 0E: Repair 0F: Charm 10: Picklock 11: Acrobat 12: Appraise 13: Swimming
The first 32 bytes are divided among the 8 classes, with 4 bytes per class. These describe the ranges for the character attributes, with 0C, 0E, 10, 12, and 14 as the possible values. These aren't used directly, but as range inputs for random number generation, and possibly involving other calculation. For example, Vitality usually comes in values of 33, 34, and 35, when the range values seem to be 0C and 14. I did 5 character generations each for a fighter and a mage. The fighter's Strength value ranged between 24 and 28, while the mage's was between 13 and 19. This at least correlates with the classes' values of 0x14 and 0x0C for the base strength values, but is clearly calculated using different formulae than the one for vitality. I can either re-generate a large number of characters until a pattern emerges or find the character generation code. The former would be easier, but the latter would give a clearer answer.
The remaining bytes (starting from the 33rd and going to the end of the file) are a series of records, 5 per class, which provide the skills lists for that class during character generation. Each record is in the format of a byte (len) to show the length of the record, and then "len" bytes of data showing the available choices. A record with a length of 1 is auto-selected (no choice is presented to the player). The classes are in order identified above, and the skill numbers are the same as identified above.
Here's an example, for what I might expect for a "battle mage" class (keeping with my policy of not including actual game data in any of my materials):
02 00 01 02 00 01 01 07 01 09 04 03 04 05 06
First: 02 - 2 bytes, 00 - Attack, 01 - Defense Second: 02 - 2 bytes, 00 - Attack, 01 - Defense Third: 01 - 1 bytes, 07 - Mana (Automatically selected, because it's the only choice) Fourth: 01 - 1 bytes, 09 - Casting (Ditto) Fifth: 04 - 4 bytes, 02 - Unarmed, 03 - Sword, 04 - Axe, 05 - Mace, 06 - Missile
Similar to the character attributes, I suspect that the way to calculate how the skills are actually applied to the character are contained in the game executable itself, rather than encoded anywhere as data.
I encourage you to go to gog.com (or a similar site) and buy Ultima Underworld. It's cheap, and it includes the sequel (which I plan to support eventually anyhow). Effort will be taken to allow the demo files to work as well, but I'm sure that EA would like to see sales of the original games. I can't give anyone copies of any game files, and this project will never distribute actual game data or information that could be used to reconstruct game data (just information about how to interpret game data). I'm also wary of distributing files directly representing any efforts at analysis of the files, so I don't plan to post anything like hex dumps, program traces, disassemblies, etc.
Hank Morgan has an Underworld exporter to the Unity game engine that's more advanced than this project in many ways, currently. One difference between our projects is that he looks like he's building an exporter and an implementation of the game logic. I'm doing that, but also using it as an excuse to teach myself OpenGL and game engine design+implementation.