SDL2 port (Linux)

[hifi]

Replace all winapi stuff after #19 is done so we can build a native Linux executable.

[vanfanel]

@hifi I am closely looking at this possibility! SDL2 on GNU/Linux has KMSDRM support, so it will be possible to have hardware-scaled graphics on the console, no X-server running will be needed :) Oh man, how badly I want this feature.. :D

[hifi]

It's not actually that far off (without sound and networking). I had TD menu partly rendering on SDL2 a few weeks ago. I just got swamped with pays-the-bills kind of work and got lost in that rabbit hole.

If you find someone interested in getting that part further please get everyone involved. We do welcome contributors.

[hifi]

SDL2 builds of TD work good enough on 64-bit Linux. RA is building but it's RA specific that it has not gotten some glue code updates so it runs properly. Closing this as fixed.

[vanfanel]

@hifi So, is it now possible to build and run RA against SDL2 on aarch64? I don't understand the previous message very well...

[hifi]

It builds but doesn't yet run properly. TD is currently the "fully" working one.

EDIT: Yes, I don't understand what I wrote either, uh. It builds but doesn't run!

[vanfanel]

@hifi Thanks for the detailed response! So you guys ALMOST there! Soon, it will be possible to run RA on a Pi natively without X (SDL2 runs on KMSDRM as I said). Will be awesome and dos-like! :dancers:

[hifi]

@vanfanel #205 will fix Linux SDL2 port of RA. You can also pre-scale the game into your display native resolution with OutputWidth and OutputHeight in [Options] section of REDALERT.INI.

[d10sfan]

Does this mean that it's possible to start playing the remastered version on Linux through the sdl2 port? Or only the original cd based version currently?

[hifi]

@d10sfan Unless someone implements a glyphx compatible frontend for the remastered builds of vanilla there will be no remastered on Linux natively.

Implementing that is out of scope for this project so we'll only refine the original interface with original data. At least for now.

The dlls would be buildable on Linux and they have a stable ABI so it's definitely possible to do that with some effort.

[d10sfan]

Ok, thanks for the explanation.

[vanfanel]

@hifi

I was delighted to read two weeks ago that it's now working on SDL2 standalone!

So today I had time to try to make TD to work with this engine. However, I can't make it work.

I built the engine on GNU/Linux X86_64, latest SDL2 (2.0.14, released today) resulting in both vanillatd and vanillara executables. I copied over the executables to my install directory, which contains a simple TD/CnC95 install, and tried to run "vanilllatd" but I got this an the game window never shows up:

manuel@hp15db0:~/cc$ ./vanillatd 
C&C95 - Starting up.
Run SETUP program first.

According to GDB, the engine is somewhere in main.cpp

7  0x00005555556d3e00 in WWKeyboardClass::Fill_Buffer_From_System (this=0x555555a4e270)
    at /home/manuel/src/Vanilla-Conquer/common/wwkeyboard.cpp:536
#8  0x00005555556d3834 in WWKeyboardClass::Check (this=0x555555a4e270) at /home/manuel/src/Vanilla-Conquer/common/wwkeyboard.cpp:152
#9  0x00005555556d3876 in WWKeyboardClass::Get (this=0x555555a4e270) at /home/manuel/src/Vanilla-Conquer/common/wwkeyboard.cpp:172
#10 0x0000555555690014 in main (argc=1, argv=0x7fffffffdfb8) at /home/manuel/src/Vanilla-Conquer/redalert/startup.cpp:621

Is this to be expected currently?

Setup.exe won't run on WINE, anyway, but using WINE to run the game via the C&C95.exe of the installation, the game runs without problems.

[hifi]

You may be hitting two issues here.

First you are trying to run the RA executable in TD directory. This would cause the game to look for REDALERT.INI instead of CONQUER.INI and will display that text if the file is not found or is empty.

Second all filenames need to be uppercase at this point. We haven't yet merged mixed case support for case sensitive filesystems. Though now that I'm reminded of this I'll clean it up and submit a PR.

Anyway, the games should run reasonably well albeit some issues. You may also want to set ScreenWidth and ScreenHeight in CONQUER.INI and REDALERT.INI to 1280 and 800 respectively to get 2x scaling.

[vanfanel]

@hifi Sorry for taking long to answer. SDL2 KMS/DRM has taken my programming time lately (so it's indirectly in benefit of this project, too!)

-I have just build latest GIT code, and I am trying to run "vanillatd" in the Tiberian Dawn (orginal C&C) directory. -I have also made all filenames uppercase.

The engine tries to start, but I get this:

Thread 1 "vanillatd" received signal SIGSEGV, Segmentation fault.
0x00007ffff7a0bd4d in clearerr (fp=0x0) at clearerr.c:25
25      clearerr.c: No such file or directory.
(gdb) bt
#0  0x00007ffff7a0bd4d in clearerr (fp=0x0) at clearerr.c:25
#1  0x000055555568df0b in RawFileClass::Raw_Seek (this=0x7fffffffd960, pos=0, dir=1)
    at /home/manuel/src/Vanilla-Conquer/common/rawfile.cpp:871
#2  0x000055555568da20 in RawFileClass::Seek (this=0x7fffffffd960, pos=0, dir=1)
    at /home/manuel/src/Vanilla-Conquer/common/rawfile.cpp:623
#3  0x0000555555676a28 in BufferIOFileClass::Read (this=0x7fffffffd960, buffer=0x55555602b1f0, size=768)
    at /home/manuel/src/Vanilla-Conquer/common/bfiofile.cpp:715
#4  0x0000555555677ae5 in CCFileClass::Read (this=0x7fffffffd960, buffer=0x55555602b1f0, size=768)
    at /home/manuel/src/Vanilla-Conquer/common/ccfile.cpp:202
#5  0x00005555555f14dd in Init_Game () at /home/manuel/src/Vanilla-Conquer/tiberiandawn/init.cpp:273
#6  0x00005555555a771f in Main_Game (argc=1, argv=0x7fffffffdfb8) at /home/manuel/src/Vanilla-Conquer/tiberiandawn/conquer.cpp:135
#7  0x000055555564621d in main (argc=1, argv=0x7fffffffdfb8) at /home/manuel/src/Vanilla-Conquer/tiberiandawn/startup.cpp:454

Any idea on what's going on?

[hifi]

That's where it's reading the palette from your disk. Possible reason is that your game directory does not contain all the mix files that are required to run or you're running with data that are not from the retail game or the executable does not have read permission to the file. It's not very graceful when it comes to IO errors right now.

TD data is a bit different than RA and I'm more familiar with latter but basically it should work somewhat correctly if you copy all mix files in the game directory that are on the CD after installing. We have a CD swapping no-cd system now in place that works with sub directories so you should be able to create "nod" and "gdi" directories with their respective main.mix files in them.

If you're working on SDL2/KMS stuff I've been testing Vanilla on Raspberry Pi 1B and 3B+ successfully with the VC4 GL driver enabled and full KMS. Hardware cursor causes a crash, though.

The OpenGL ES PR makes the final surface rendering pipeline slightly faster but it's still on the edge of being playable on the original Pi without any scaling. 3B+ runs it pretty well even with the current pure SDL2 surface method but if you find performance issues with the implementation we have that'd be great help!

[vanfanel]

@hifi Thanks for the explanation. Is there a list of the needed .MIX files for each game? So I can verify I have them all.

Also, the HW cursor crash... does it happen with latest SDL2 sources from mercurial? You can get them with: hg clone http://hg.libsdl.org/SDL

My goal is to have Vanilla-Conquer on Pi3/4 working right on SDL2-KMS/DRM. HW cursors should work, too. Are you using a somewhat special cursor format?

[hifi]

Don't have a list yet. If you want something simple to test pull main.mix and redalert.mix from RA retail or demo zip and those are pretty much enough for the game to boot up. You also need a non-empty redalert.ini (just put anything in it) so that it doesn't complain about running the SETUP program.

I promise to write a wiki page how to set up the data correctly :sweat_smile:

The SDL2 KMS cursor crash happens in the DRM code so it's unlikely to be an SDL2 issue, though. Haven't tried with a nightly build of SDL2 yet. Looking at SDL code it converts anything into ARGB8888 anyway if the format doesn't match when you call SDL_CreateColorCursor. I tried a lot of different things but I guess the size of the cursor when using scaling makes it an issue.

[hifi]

And for the record if you want to discuss the Discord server linked in the README is the best place but I do understand not everyone are on Discord or want to.

[Gerwin2k]

This mix-file setup works for VanillaTD.exe. It is a bit of a compacted set, without the movies and without the Covert Ops music tracks. File size in bytes is behind the name.

cclocal.mix 121.305 deseicnh.mix 120.296 local.mix 114.387 / 4 (Bigger one is from Covert Ops) movies.mix 3.562.660 (This is a modded file with only the intro animation) sc-000.mix 185.937 (Covert Ops) sc-001.mix 490.318 (Covert Ops) scores.mix 39.114.329 (These are the base game tracks only, not Covert Ops) speech.mix 947.949 / 594.297 (Bigger one is from Covert Ops) tempicnh.mix 119.935 transit.mix 4.105.646 update.mix 10.233.756 updatec.mix 990.901 winticnh.mix 119.935

sc-n64.mix 206.536 (optional extra missions) sc-psx.mix 112.932 (optional extra missions)

GDI\aud.mix 1.890.646
GDI\conquer.mix 2.039.435 GDI\desert.mix 677.061 GDI\general.mix 2.584.388 GDI\sounds.mix 1.228.794 GDI\temperat.mix 639.607 GDI\winter.mix 667.385

NOD\aud.mix 1.890.646 NOD\conquer.mix 2.039.435 NOD\desert.mix 677.061 NOD\general.mix 2.549.589 NOD\sounds.mix 1.228.794 NOD\temperat.mix 639.607 NOD\winter.mix 667.385

CovertOps\aud.mix 1.890.646 CovertOps\conquer.mix 2.270.373 CovertOps\desert.mix 677.061 CovertOps\general.mix 3.409.189 CovertOps\sounds.mix 2.121.045 CovertOps\temperat.mix 639.607 CovertOps\winter.mix 667.385

[vanfanel]

@Gerwin2k I have tried to replicate that exact fileset, but I am still getting the same segfault.

It happens just after the "C&C95 - About to load the language file" message. What language file is that?

@hifi I got RA to work. HW cursor doesn't work on X11 either. What cursor size are you trying to set? HW cursors are drawn in a special HW plane that only supports some sizes depending on the graphics system. So, for example, AMDGPU (what I have here) doesn't support the size that Vanilla-Conquer is trying to set, it seems. Ah! I don't like Discord that much, because what we talk isn't available for public help and information, but I may join anyway :)

[hifi]

Does https://github.com/TheAssemblyArmada/Vanilla-Conquer/pull/418 fix the the cursor issue on X11?

Is your C&C install the English version btw.? We don't support non-English variants yet.

[Gerwin2k]

Edited; served its purpose.

[vanfanel]

@Gerwin2k That works, thanks a lot! :+1:

@hifi I will be glad to test your PR, but can you please tell me how to merge to my local repo a PR that is not yet merged? Because I am not ashamed to admit I have no idea on how to do that :P

[vanfanel]

Another thing that comes to my mind: I believe C&C and RA assets are free by now, aren't they? Maybe you could simply upload proper data packs ready to use.

[hifi]

They are kind of freeware but that doesn't mean that we are free to repackage and redistribute the data as we see fit. We do plan to create an installer that will use any official source to build the game data from but for now we settling with instructions for testers.

To test a GitHub PR (#418 in this case) in your local checkout assuming you have no changes to the repository (git status is clean):

git fetch origin pull/418/head:pr-418
git checkout pr-418

You can go back to the vanilla branch with git checkout vanilla. To update your local branch you can always run the same git git fetch with -f to force an update if the remote has been rebased.

Hope these instructions work for you. Thanks.

[vanfanel]

@hifi Thanks a lot for the easy instructions :+1: I have built with these changes and yes, the hw cursor works on AMDGPU inside X11 now!

Another suggestion: Why don't you add SDL_SetHint(SDL_HINT_RENDER_SCALE_QUALITY, "linear"); before the SDL_CreateRenderer() call? Default scaling is nearest, and pixels look uneven on modern displays... Bilinear is far more appropiate. Also, adding SDL_RENDERER_PRESENTVSYNC to the SDL_CreateRenderer() flags is good because it makes tearing go away in screen movement.

As for performance, since you asked me earlier: are you doing some form of software scaling? Software scaling when using SDL2 is a BAD idea, since SDL2 uses OpenGL in the renderer to scale without using the CPU. Other that that, I don't see what performance bottlenecks could be on the Pi. Software scaling on the Pi platforms is ALWAYS a no-no. Simply let SDL2 do it's job using the GPU and I think that is optimal enough :)

[Gerwin2k]

They are kind of freeware but that doesn't mean that we are free to repackage and redistribute the data as we see fit. We do plan to create an installer that will use any official source to build the game data from but for now we settling with instructions for testers.

Got it. Edited my post. Some contradictions and difficulties with that come to mind, but that is for another time.

[vanfanel]

@Gerwin2k I have overwritten movies.mix with a full 400MB one. It works, but after the initial video, the second video (preview) appears in grayscale. Is that expected or I have a corrupt/bad movies.mix?

[Gerwin2k]

@vanfanel That small substitute movies.mix was in the root folder. But if you want to add those full movies.mix files you have to place them in the right subfolder: NOD/GDI/CovertOps. (Then maybe also delete the small movies.mix from the root?)

[vanfanel]

@Gerwin2k Ahh! Understood! Thanks!

[vanfanel]

@Gerwin2k Do you have a file list for Red Alert, too? I know I must have:

main.mix redalert.mix redalert.ini

Where do the RED/ALLIES data go? In RED / ALLIES dirs? What are the files needed in each dir?

[Gerwin2k]

Well, I compacted it again, and used XCC mixer to do that. So I am not entirely sure how it was before I started tweaking things.

FULL SIZE (If I recall correctly):

VanillaRA.exe
Aftrmath.ini
redalert.ini
redalert.mix 25.046.328 expand.mix 458.242 expand2.mix 469.922 hires1.mix 488.204 (In this case: This is the small infantry modded file) allied\main.mix (CD1) soviet\main.mix (CD2) counterstrike\main.mix aftermath\main.mix

---

COMPACTED (No movies):

VanillaRA.exe
Aftrmath.ini
redalert.ini
redalert.mix 25.046.328 expand.mix 458.242 expand2.mix 469.922 hires1.mix 488.204 (In this case: This is the small infantry modded file) allies.mix 309.406 (Extracted from aftermath main mix) conquer.mix 2.183.781 (Extracted from aftermath main mix) interior.mix 247.425 (Extracted from aftermath main mix) russian.mix 266.077 (Extracted from aftermath main mix) snow.mix 1.030.861 (Extracted from aftermath main mix) sounds.mix 1.376.627 (Extracted from aftermath main mix) temperat.mix 1.038.859 (Extracted from aftermath main mix) scores.mix 28.070.921 (Modded file with a small subset of the music tracks) allied\main.mix 17.172.260 (CD1, modded mix file, only retained general.mix in there) soviet\main.mix 17.172.260 (CD2, modded mix file, only retained general.mix in there) counterstrike\main.mix 17.890.388 (modded mix file, only retained general.mix in there)
aftermath\main.mix 18.968.011 (modded mix file, only retained general.mix in there)

These files just appear when running the game: intr_vtx.pal, snow_vtx.pal, temp_vtx.pal

[vanfanel]

@Gerwin2k That was it! Have it all running now :) Thanks a lot! :)

[hifi]

Another suggestion: Why don't you add SDL_SetHint(SDL_HINT_RENDER_SCALE_QUALITY, "linear"); before the SDL_CreateRenderer() call? Default scaling is nearest, and pixels look uneven on modern displays... Bilinear is far more appropiate.

I guess that's a preference thing. I do like the blockiness of the original pixels scaled up but a compromise would be that we do integer scaling with nearest by hand or with a shader and then bilinear for the rest of the space. I don't know how that ends up looking but probably a lot better than full bilinear.

Also, adding SDL_RENDERER_PRESENTVSYNC to the SDL_CreateRenderer() flags is good because it makes tearing go away in screen movement.

That would be okay with the hardware cursor enabled but without it it's going to add input delay. Because we default to hardware cursor I guess it would be an acceptable default when we fix game ticks being somewhat tied to framerate (which needs to be higher than 60 right now for best experience).

As for performance, since you asked me earlier: are you doing some form of software scaling? Software scaling when using SDL2 is a BAD idea, since SDL2 uses OpenGL in the renderer to scale without using the CPU. Other that that, I don't see what performance bottlenecks could be on the Pi. Software scaling on the Pi platforms is ALWAYS a no-no. Simply let SDL2 do it's job using the GPU and I think that is optimal enough :)

Not doing software scaling except for the VQA movies but it's fast enough. Pushing full 32-bit 640x400 frames seems to be a bit of a bottleneck even with hardware scaling. The GLSL variant of the SDL renderer improves that but some margin as it's then only pushing 8-bit frames and a shader does palette conversion.

It's a bit odd that the games ran on a potato originally but the og Pi seems to be a hard target to get it running smoothly. Granted we don't have any assembly optimizations anymore but the Pi should be plenty more powerful than what the original recommended system specs were for RA on Win95 (Pentium @ 75Mhz) even with the added overhead from SDL2.

[capsterx]

Another suggestion: Why don't you add SDL_SetHint(SDL_HINT_RENDER_SCALE_QUALITY, "linear"); before the SDL_CreateRenderer() call? Default scaling is nearest, and pixels look uneven on modern displays... Bilinear is far more appropiate.

I guess that's a preference thing. I do like the blockiness of the original pixels scaled up but a compromise would be that we do integer scaling with nearest by hand or with a shader and then bilinear for the rest of the space. I don't know how that ends up looking but probably a lot better than full bilinear.

I'd prefer this to be switchable. I grew up with the blockyness, but I like the look of the more modern filters

Also, adding SDL_RENDERER_PRESENTVSYNC to the SDL_CreateRenderer() flags is good because it makes tearing go away in screen movement.

That would be okay with the hardware cursor enabled but without it it's going to add input delay. Because we default to hardware cursor I guess it would be an acceptable default when we fix game ticks being somewhat tied to framerate (which needs to be higher than 60 right now for best experience).

As for performance, since you asked me earlier: are you doing some form of software scaling? Software scaling when using SDL2 is a BAD idea, since SDL2 uses OpenGL in the renderer to scale without using the CPU. Other that that, I don't see what performance bottlenecks could be on the Pi. Software scaling on the Pi platforms is ALWAYS a no-no. Simply let SDL2 do it's job using the GPU and I think that is optimal enough :)

Not doing software scaling except for the VQA movies but it's fast enough. Pushing full 32-bit 640x400 frames seems to be a bit of a bottleneck even with hardware scaling. The GLSL variant of the SDL renderer improves that but some margin as it's then only pushing 8-bit frames and a shader does palette conversion.

It's a bit odd that the games ran on a potato originally but the og Pi seems to be a hard target to get it running smoothly. Granted we don't have any assembly optimizations anymore but the Pi should be plenty more powerful than what the original recommended system specs were for RA on Win95 (Pentium @ 75Mhz) even with the added overhead from SDL2.

I hope these changes will allow them to be compiled out. Porting things to the switch I've had to deal with projects that assumed a specific level of opengl-ness. Im not an expert with opengl, but I know the switch uses the ES subset, which I've ran into issues with some games assuming it's a "full" *nix distro. The switch only fakes posix/nixness for a few things for ease of development, but it is not the same.

[hifi]

I'd prefer this to be switchable. I grew up with the blockyness, but I like the look of the more modern filters

Yes, of course. Everything will always be optional.

I hope these changes will allow them to be compiled out. Porting things to the switch I've had to deal with projects that assumed a specific level of opengl-ness. Im not an expert with opengl, but I know the switch uses the ES subset, which I've ran into issues with some games assuming it's a "full" *nix distro. The switch only fakes posix/nixness for a few things for ease of development, but it is not the same.

Including the GLSL renderer if it ever gets merged. It targets OpenGL ES 2.0 which runs on the Pi and desktop Linux and I would assume the Switch is ES 2.0 compatible and if not and it makes a performance difference it should be rather trivial to make it ES 3.0 compatible as well.

Though, if you want filters then you do need the GLSL renderer so getting it running on the Switch is equally important than the current one.

[vanfanel]

I guess that's a preference thing. I do like the blockiness of the original pixels scaled up but a compromise would be that we do integer scaling with nearest by hand or with a shader and then bilinear for the rest of the space. I don't know how that ends up looking but probably a lot better than full bilinear.

If you ever do integer scaling with nearest by hand (or with a shader) please make it optional. The thing with that technique (crispy-doom used to do that) is that it requires intermiediate surfaces, and the corresponding copy-around. That makes CPU usage on Pi very high. And once that kind of stuff is in place, single-surface mode tends to be forgotten and authors don't go back to offer it. So please, do it with optional GLSL shaders, and/or make it optional, but don't force the use of intermediate surfaces in all cases.

What I mean is that, not only do the "blocky scaling" optional, but also make optional the internal intermediate surfaces that are used for that. Many devs obviate this, and then people wonder why an old game engine only works well on overpowered X86 computers.

[hifi]

As I said in the previous comment, everything will be optional and selectable. The defaults may change according to what we think fit the best for the games style.

However even with a single surface it needs the conversion before rendering on screen so that is unavoidable without shaders or native paletted mode which we don't support right now because no modern system really has that. The optimization in the GLSL renderer does avoid the 8->32 bit conversion on the CPU so technically on any modern system it probably outperforms the native SDL2 surface.

It would be nice if SDL2 had a palette emulation mode for the renderer but that's an edge case with very little expected use.