Big Endian support

[walkero-gr]

Hello there, Congrats for the excellent job you have done with this engine. I am working on a port for AmigaOS 4 and later for MorphOS. Both of them are operating systems running on big endian systems. So, I am working on converting the needed parts of the code to support them. I already managed to have some results, but a lot are missing.

My question is, is this something you are working as well? This is necessary for me to know, so we do not do the same work both of us.

in Shape.cpp you have the followin `#define BE_Cross_Swap16(x) ((uint16_t)(((uint16_t)(x)<<8)|((uint16_t)(x)>>8)))

define BE_Cross_Swap32(x) ((uint32_t)(((uint32_t)(x)<<24)|(((uint32_t)(x)<<8)&0x00FF0000)|(((uint32_t)(x)>>8)&0x0000FF00)|((uint32_t)(x)>>24)))

` Can you tell me if this is related, and what is the purpose of this code? It is weird for me since the whole code is for LE, and I do not understand if this code is for making the switch.

Thank you in advance for your time and help.

[ArnoAnsems]

Hello and thank you! No worries, I'm not working on Big Endian support and also wasn't planning to. I'm actually quite unfamiliar with AmigaOS. I just did some googling, since I wasn't even aware that it supports OpenGL. Nice!

You're probably already aware that Catacomb 3-D was ported to AmigaOS in the past? I think that port is based on the ReflectionHLE source port, which does support Big Endian. https://www.lemonamiga.com/games/details.php?id=4017

The function BE_Cross_Swap16 actually originates from the ReflectionHLE source code. I don't know why the function is named like that. The shape loading code in Shape::LoadFromFile is based on the function ext_LoadShape in ReflectionHLE, see https://github.com/ReflectionHLE/ReflectionHLE/blob/master/src/id91_11/cat_all/external_restored/ext_soft.c

Good luck and have fun!

[walkero-gr]

@ArnoAnsems Thank you so much for taking the time to reply to this issue.

So, I will continue on working on the Big Endian support and as soon as I have it finished I will make a PR and you can decide whether you want to merge it or not.

I'm actually quite unfamiliar with AmigaOS. I just did some googling, since I wasn't even aware that it supports OpenGL. Nice!

The work I am doing is for AmigaOS 4, which is a more modern version of the old AmigaOS developed back in the 80s. Fun fact, even for the old version there are new releases happening, but it targets the old 68K CPU clocked in less than 100MHz.

You're probably already aware that Catacomb 3-D was ported to AmigaOS in the past? I think that port is based on the ReflectionHLE source port, which does support Big Endian. https://www.lemonamiga.com/games/details.php?id=4017

Yeap, that is for the classic line of Amiga's which are not my target. A port of an engine like yours would crawl to death on these systems.

Thank you again for your help. Please allow me to come back here with questions if/when I get stuck. I find it difficult sometimes to work with code that is not mine fast.

[walkero-gr]

@ArnoAnsems I managed to have the game running pretty well on my AmigaOS 4 system. I can see the intro images, menu options, enter a game and fight the enemies just fine.

But I have a problem that I am afraid I am a little bit stuck. And your guidance can be crucial.

If you check the screenshot above, this is the first level of the Abyss Shareware game. The map shows the placement of the walls in the level, close to where the player starts. The first 22 lines of tiles seem to be in the right place, but then they are misplaced and broken. The level still works, but it is not accurate.

I am trying to figure out which parts of the code are involved, and which methods I should investigate. If you could share some info on that, it would be much appreciated.

[walkero-gr]

Here is a second screenshot of the lower-left side of the same level, showing the problem more clearly.

Thank you for your help in advance

[ArnoAnsems]

I'm quite impressed by your progress! It's nice to see the game up and running already on that OpenGL 1.3 driver. I still recognize the lower part of the first level in those screenshots, but the tiles are shifted one or more places to the left. It looks to me like a few individual tiles were skipped while decompressing the map data. You may want to look into the function GameMaps::GetLevelFromStart(). Especially the sub-function Decompressor::CarmackExpand() looks suspicious, since it has bit operations that might be vulnerable to Big Endian.

If you're eventually able to make a Pull Request, I'm certainly willing to have a look at it. In the mean time, further questions or help request are welcome.

[walkero-gr]

@ArnoAnsems Thank you so much for your kind words and your prompt reply, with hints where to look.

I have done a lot of work with the Decompressor::CarmackExpand(), which made possible the game to even start. Last night I created a commit at my fork, which you can see at: https://github.com/walkero-gr/CatacombGL/commit/1bb2fc146a18d3d9e3e66de36e3c8d9849bd6c0f

I tested all the values used in this method with the Linux version and the numbers much. But still, I might missed something.

Have in mind that this code is still a WIP. So, I haven't made the code clean to become a PR yet.

Now, the weird thing is how the majority of the level is fine (first 22 rows), and the collecting items are in the right positions, even on the broken part of the map.

I wonder if FARTAG and NEARTAG are involved in that. But I am not getting what they mean and the logic behind them. The following lines look interesting

 const uint16_t chhigh = ch >> 8;
 uint16_t count = ch & 0xff;

[ArnoAnsems]

No worries, I'm well aware that you're in the getting-it-to-work phase, and that it will take time to get towards a fully functional PR. I didn't notice it at first, but it indeed looks like the collectable items are in the correct positions in the broken part of the map. For example, the blue key in the first screenshot is 4 tiles below the health powerup from the unbroken part of the map, as it should be. That would mean that plane2 with all the items got correctly decompressed, while plane0 with the walls and floor data didn't.

The meaning behind FARTAG and NEARTAG is explained on this wiki page: https://moddingwiki.shikadi.net/wiki/Carmack_compression

[walkero-gr]

@ArnoAnsems Problem Solved.

The problem was in the CarmackExpand() only for the FARTAG, which needed a switch in the offset value. I did a new commit at https://github.com/walkero-gr/CatacombGL/commit/d97b45a8da6136c5f8ffab844d404ec4a30631c7

Delighted that I managed to find and fix it. But I couldn't be able to do that without your guidance. Thank you so much for that.

Are all the problems solved now? Not really, since I have the file saving and loading to fix, and a few more. But it is getting better now.

[ArnoAnsems]

You're welcome! Good to hear that you managed to tackle the problem; good job!

[walkero-gr]

Thanks. I did some more fixes here and there, but the one that I can't find a solution is the music playback. The soundfx using adlib and pc cound are working well, and they sound correctly. But the music using adlib is not there. Actually, you can hear some bells and low ham, but not the whole track. Of course I use the full version from the GOG package, which I also use on my Linux machine.

Any debugging I did so far didn't reveal any issues. And the files seem to be found and loaded just fine. I wonder if you ever had the same issue. I could probably record a video that shows clearly the problem.

[walkero-gr]

I created a small video that shows how it works, and especially how it sounds. https://youtu.be/1FMLyF0OBS0 If you have any idea where the problem might be, that would help a lot. I am a bit lost with this one.

[ArnoAnsems]

I had a look at the video and it indeed sounds like the software is attempting to play a music track, yet misses a lot of data. It is pretty cool to see so much other functionality already working, though. I mean, there's gameplay and it seems to run smooth.

The sound code in CatacombGL was taken from ReflectionHLE (be_st_sdl_audio_timer.cpp, id_sd.cpp, dbopl.cpp). I checked the original files in the ReflectionHLE repository, but there's no big endian specific code in there (usually marked with REFKEEN_ARCH_BIG_ENDIAN). So I assume I didn't "clean up" any big endian support there.

I can imagine that loading the music from disk can be problematic on a big endian architecture. But the sound FX are loaded from the same data file and decompressed in the same way as the music track. So it puzzles me as well that the sounds FX are working but the music is not. The music in Catacomb Abyss is experimental, though. The game data contains a music track, but it was never played in the original DOS game. Only Catacomb 3-D (The Descent) had music. So you may want to double check how it sounds in Catacomb 3-D.

[walkero-gr]

@ArnoAnsems Thank you for your reply and for checking that issue.

I tried to print out different values of the music data on Linux and the AmigaOS 4, and they match 100%. But if there is something you would recommend me to check, please let me know because I might miss something.

I also checked the Catacomb-3D and plays the music in the exact same way.

I think that even the sound-fx are not so clear. Furthermore, I believe there is some kind of noise in them, although they seem to be the correct ones. This might be a different problem, though.

I continue investigating this issue. If you have any idea of a way to log the music data and make sure they are loaded in the right way, please let me know.

[ArnoAnsems]

Regarding the sound-fx quality, I was quite recently informed by NY00123 that the music can actually influence the sound-fx. There's more info in his post on Doomworld. The key take away is that sound-fx are best tested when CatacombGL is started with music disabled.

It's been a while (years actually) since the last time I've looked at the sound code. I just did a quick comparison with the latest ReflectionHLE code base and there appears to be some differences. I will do a sync in the coming days. I'm not sure if that will help, because I don't know how often ReflectionHLE is run on big endian systems. But maybe I spot something.

[walkero-gr]

Thank you for looking into this, and sorry for dragging you into that. As soon as you make your changes, I will merge them with mine and will test it.

[ArnoAnsems]

I've just pushed a commit that integrates the latest audio code from ReflectionHLE into CatacombGL. It looks like NY00123 redesigned the audio backend quite a bit the last few years. The ThirdParty folder looks entirely different now. This time I left all the compiler switches in, so maybe that helps. I tested it on Windows 11 and Ubuntu 24.04. Music, adlib and PC speaker sounds still worked as expected.

[walkero-gr]

@ArnoAnsems Thank you so much. I will work tomorrow on it and see if that helps my port. I will let you know how it goes.

[walkero-gr]

@ArnoAnsems Unfortunately I get the same result in the sound. Now, I have seen you left some output info.

On my Linux system I get the following

Initializing audio subsystem, requested spec: freq 48000, format 32784, channels 2, samples 1024 Audio subsystem initialized, received spec: freq 48000, format 32784, channels 2, samples 1024, size 4096

But on my AmigaOS 4 I get:

Initializing audio subsystem, requested spec: freq 48000, format 36880, channels 2, samples 1024 Audio subsystem initialized, received spec: freq 48000, format 36880, channels 2, samples 1024, size 4096

You will see that every value is the same, except the format. I will investigate that and see if I can figure out anything useful.

[walkero-gr]

I do not believe that format has anything to do with the problem. It is a number set by the AUDIO_S16SYS, on both the AmigaOS and Linux. it just seems to have a difference in number for these systems.

I am wondering if the problem is in my SDL2 port. What kind of format is the audio of the game? Is there one? I would like to create a test example and see if that works.

[ArnoAnsems]

It's unfortunate that the merge with the latest ReflectionHLE code didn't bring a solution, but at least we can now rule out any bug fixes that I might have missed in the past few years. The game uses different audio formats for PC speaker sound effects, adlib sound effects and music. All three are described here. The format for the PC speaker sound effects is by far the simplest and would also be the easiest to test. Do the PC speaker sound effects work?

[walkero-gr]

Do the PC speaker sound effects work?

Yeah, they are working. Although I think they have more noise than on my Linux machine, but this might not be accurate. But they seem to work fine. I will double-check that

[NY00123]

Been reminded of this thread. I've made some checks, inspecting only Catacomb 3-D for music playback. As far as I can tell, the last version of ReflectionHLE should play music in Catacomb 3-D as expected on Big-Endian targets, more-or-less.

To elaborate, I had this old qemu-based PowerPC VM with Debian Wheezy v7.3 set up. I had been curious to see how well may it fare with the last version of ReflectionHLE. The VM's disk image had the timestamp of "2017-01-22 22:32:26.129476100 +0200".

This setup had already had a few old builds, including ones with working sounds and music in Catacomb 3-D. As expected, the VM was running significantly slower than the host PC, probably slower than many 20+ years old PCs as well. Despite this, older build was still working, at least in part.

After making adjustments for building an updated version of ReflectionHLE (details below), it was still ok, albeit sound and music were stuttering. Removing the definition of BE_ST_MANAGE_INT_CALLS_SEPARATELY_FROM_AUDIO didn't assist. There may be multiple reasons, but one possibility is my migration from DBOPL to Nuked OPL3. While Nuked OPL3 is considered to be the more accurate emulator, DBOPL is known to be faster.

Writing about the trajectory I had went through, until I could test an updated ReflectionHLE build:

• Recalling which user accounts were present in the Debian installation.
• Mouse clicks not working as expected for an unknown (to me) reason, albeit I could move the mouse cursor and use a keyboard.
• Figuring out (again) how to enable sound in the PowerPC VM. I eventually used the QEMU command-line arguments -soundhw es1370 -audiodev alsa,id=default (from an x86-64 Linux host).
• Finding a way to download DEB packages for a version of Debian released in 2016. Luckily, there was an archive URL I could add to the package manager. I used this for installing CMake, and also the SpeexDSP library.
• The version of CMake in question was 2.8.9, lacking support for target_include_directories. After checking if I could change ReflectionHLE's CMakeFiles.txt, I eventually downloaded the CMake 2.8.12.2 sources and bootstrapped that version with CMake 2.8.9.
• Call to SDL_GameControllerAddMappingsFromRW from BE_Init_Common, the former being introduced in SDL 2.0.2, had been disabled beforehand; In fact, not just in 2017. It had already been disabled in an earlier tree from the beginning of 2016, a modification of Reflection Keen v0.13.0 (20151120).
• Now in 2024, I disabled the launcher function EL_ST_Launcher_Handler_ImportControllerMappingsFromSteam, also due to use of SDL_GameControllerAddMappingsFromRW.
• Had problems with using macros like PRIu32 in be_cross.h. I tried investigating a bit, but eventually just used, e.g., %u (assuming unsigned type is 32-bit here) as-is. I similarly adjusted uses in wolf3d/id_heads.h.
• Due to conflicting extern declarations of g_be_emulatedMemSpace, the static inline definition of BE_Cross_GetPtrXmsOff was moved from be_cross_xmm.h into be_cross_xmm.c.
• Like SDL_RENDER_DEVICE_RESET being explicitly defined in ReflectionHLE if building with SDL < v2.0.4, I added a similar conditional definition of SDL_RENDER_TARGETS_RESET for SDL v2.0.0-1.
• The macro __STDC_CONSTANT_MACROS was defined in nukedopl/opl3.h before including inttypes.h, so it'd be possible to use UINT64_C.

[walkero-gr]

@NY00123 If I understood correctly you are saying that the game music and sounds work fine on this PowerPC emulated linux based system you tested?

[NY00123]

@NY00123 If I understood correctly you are saying that the game music and sounds work fine on this PowerPC emulated linux based system you tested?

Outside of the aforementioned stutters, yes, they work well (at least in Catacomb 3-D).

[walkero-gr]

@ArnoAnsems I was reading your post at https://www.doomworld.com/forum/post/2858768 I wonder if this is responsible for me that the music in not working. I mean, maybe the calculations on how many bytes are removed are wrong at the system I am porting. Can you please let me know where exactly this is happening? I could take a look and compare my port results against the linux version.

[ArnoAnsems]

In the function AudioRepository::GetMusicTrack(), I first decompress the music data. Then I store the first two bytes (uint16_t) of that decompressed data as the musicTrackLength. I can imagine that would need some conversion in case of big endian. Then the decompressed music data is copied to the m_musicTracks array with a memcpy. The memcpy is made with an offset of two bytes by adding sizeof(uint16_t) to the source pointer.