ELKS version of doomtd3

[toncho11]

Very excited that you have started working on the ELKS port! Here is a how graphics has been initialized in ELKS: https://github.com/ghaerr/elks/wiki/Coding-games-for-ELKS in function set_mode(byte mode). This is a version that has been tested to work on ELKS.

I see export CPU=i286, so it will run only on 286 for now?

[FrenkelS]

Compiling fails with the error message

elf2elks: error: stray SHT_PROGBITS SHF_ALLOC section 0x8 `.fardata.f.r_draw.c.20005$'

and I don't know what that means.

Is the program too big? Compiling for 286 should create a smaller executable. Setting CPU to i8088 results in the same error message.

Putting __far in front of finetangentTable_part_3 and finetangentTable_part_4 in r_draw.c results in the error message

elf2elks: error: stray SHT_PROGBITS SHF_ALLOC section 0x4 `.farrodata.f.r_draw.c.73135$'

[toncho11]

@ghaerr can you please assist?

[ghaerr]

Hi @FrenkelS,

Nice work trying to get Doom compiled and running on ELKS!

Compiling fails with the error message elf2elks: error: stray SHT_PROGBITS SHF_ALLOC section 0x8 `.fardata.f.r_draw.c.20005$'

I'm not entirely sure what that message means either, it is coming from the linker post-processor that converts ELF files to ELKS a.out format. Try adding the following lines to your CFLAGS in compelks.sh:

-melks -melks-libc -mtune=i8086 -mcmodel=medium -msegment-relocation-stuff -ffunction-sections

and remove -march=.

I can also see what a mess it is that ELKS header files export sector_t and seg_t. I think I can do something about that, but might as well leave what you have for the moment.

[toncho11]

Try this to see if compilation will improve:

//uint16_t _fmemalloc(uint16_t __size, uint16_t *__seg);

segment_t I_ZoneBase(uint32_t *size)
{
    uint16_t max, segment;
    //_fmemalloc(0xffff, &max);
    //_fmemalloc(max, &segment);
    //*size = (uint32_t)max * PARAGRAPH_SIZE;
    return segment;
} 

Maybe this declared version of _fmemalloc interferes with the real one declared in ELKS libc or something like that? Also I am not sure if it is _fmemalloc or fmemalloc without the "_".

[ghaerr]

Maybe this declared version of _fmemalloc interferes

Don't use _fmemalloc, that's the system call. Use the C library version, which is defined in <malloc.h> as

void __far *fmemalloc(unsigned long size);

Just pass the size in bytes desired and a far pointer will be returned to the memory, or 0 if unable. Of course, only 64k bytes can be accessed at a time without adjusting the segment portion of the far pointer.

[ghaerr]

Hi @FrenkelS,

I can also see what a mess it is that ELKS header files export sector_t and seg_t.

I fixed the problem of sector_t and seg_t being exported in the ELKS C library header files in https://github.com/ghaerr/elks/pull/1970. You can now remove the klugery of renaming sector_t and seg_t you had to add in order to get doomtd3 to compile for ELKS if you like. Sorry about that.

Thank you!

[FrenkelS]

using CFLAGS="-march=i286 -mcmodel=medium -melks -Os" and CFLAGS="-Os -melks -melks-libc -mtune=i8086 -mcmodel=medium -msegment-relocation-stuff -ffunction-sections" results in

elf2elks: error: stray SHT_PROGBITS SHF_ALLOC section 0x8 `.fardata.f.r_draw.c.20005$'

using CFLAGS="-march=i286 -mcmodel=medium -melks -O0" results in

/usr/lib/x86_64-linux-gnu/gcc/ia16-elf/6.3.0/../../../../../ia16-elf/bin/ld.gold: error: address of section '.fartext' moves backward from 0x28b30 to 0x20000
/usr/lib/x86_64-linux-gnu/gcc/ia16-elf/6.3.0/../../../../../ia16-elf/bin/ld.gold: error: address of section '.fardata.f.r_draw.c.20005!' moves backward from 0x37330 to 0x28b30
/usr/lib/x86_64-linux-gnu/gcc/ia16-elf/6.3.0/../../../../../ia16-elf/bin/ld.gold: error: address of section '.fardata.f.r_draw.c.20005&' moves backward from 0x37330 to 0x28b30
/usr/lib/x86_64-linux-gnu/gcc/ia16-elf/6.3.0/../../../../../ia16-elf/bin/ld.gold: error: address of section '.fardata.f.z_zone.c.00712!' moves backward from 0x37330 to 0x28b30
/usr/lib/x86_64-linux-gnu/gcc/ia16-elf/6.3.0/../../../../../ia16-elf/bin/ld.gold: error: address of section '.fardata.f.z_zone.c.00712&' moves backward from 0x37330 to 0x28b30
/usr/lib/x86_64-linux-gnu/gcc/ia16-elf/6.3.0/../../../../../ia16-elf/bin/ld.gold: error: load segment overlap [0x10000 -> 0x37330] and [0x30000 -> 0x42d70]
collect2: error: ld returned 1 exit status

at least, that's different

Texture mapping can be replaced by rendering one color per wall by using -DFLAT_WALL. This removes a table of 2048 bytes and a table of 4096 bytes. CFLAGS="-march=i286 -mcmodel=medium -melks -Os -DFLAT_WALL", results in

elf2elks: error: stray SHT_PROGBITS SHF_ALLOC section 0x6 `.fardata.f.z_zone.c.00712$'

[toncho11]

So ld says it is something in r_draw.c and z_zone.c and they both contain far pointers. I see that you use the original z_zone.c that worked when compilation is for DOS.

[toncho11]

I checked here for address of section '.fartext' moves backward from and @ghaerr says

This error means that the code segment is too large, and that you'll have to use medium model (which could also easily be too small).

But it is already medium model, right?

[ghaerr]

Yes, it seems the executable is too large. You might try -O3, -O0 generates lots of code.

Another option would be to switch to the OpenWatcom compiler large model, which is now supported.

[toncho11]

If switching to OW then the assembler format is different I think and this will require changing the init graphics procedure to the format of OW.

Also the same compiler was able to compile for DOS in medium model, but now it hits limitations when the target is ELKS.

[toncho11]

No large model in gcc-ia16: https://github.com/tkchia/gcc-ia16/issues/152

[ghaerr]

this will require changing the init graphics procedure to the format of OW.

That should be very straightforward, as it looks there's only ~15 lines ASM required. See here for an example of how to write ASM within C for OpenWatcom.

Also the same compiler was able to compile for DOS in medium model, but now it hits limitations when the target is ELKS.

In that case, it would be worth looking in to why the DOS version was able to fit, and ELKS not. I would not think there would be that much difference in "library" code between the systems. Are there particular options or functions that are commented out for DOS but included in the ELKS build? Is the screen driver much different?

[ghaerr]

Also I notice that doomtd3 supports an OpenWatcom build in medium model using makefile.w16. This should mean large model shouldn't be required for an OW build for ELKS as well. If the problem is that the screen driver for ELKS is too large, perhaps the same one that is used for DOS could work - it writes directly to the EGA/VGA, doesn't it?

[FrenkelS]

It writes directly to CGA. i_elks.c and i_ibm.c are pretty similar.

[toncho11]

I see you are switching to OpenWatcom. Do you have #include somewhere for the fmemalloc to be used from ELKS libc?

[ghaerr]

@FrenkelS,

I can see you had no problem rewriting I_SetScreenMode for Watcom :)

For building, at this point it is easiest to use the ewcc and ewlink shell scripts from ELKS for wrappers around OpenWatcom's wcc and wlink. If you need special -Wc,xxx options for the underlying tool, you can pass them to the scripts and they'll be passed along to wcc. If you think you only need medium model rather than large, just change libc/watcom.model to read MODEL=m. That will also also you to build a matching libc/libc.lib for OpenWatcom by running make -f watcom.mk from elks/libc/. More instructions are at https://github.com/ghaerr/elks/wiki/Using-OpenWatcom-C-with-ELKS.

The source for ewcc and ewlink are in elks/elks/tools/objtools/.

Do you have #include somewhere for the fmemalloc

Including <malloc.h> explicitly isn't needed, as stdlib.h already includes it.

[toncho11]

Does it compile with this last commit?

[FrenkelS]

Watcom gives a linking error:

Error! E3173: default data segment exceeds maximum size by 3008 bytes
Error: Linker returned a bad status

But it does compile when I change -Wl,stack=0x1000 and -Wl,heapsize=0x1000 to 0xa00.

This is with the medium memory model. I haven't tried the large memory model.

[toncho11]

The wiki page https://github.com/ghaerr/elks/wiki/Using-OpenWatcom-C-with-ELKS by @ghaerr says

When compact or large models are built, ELKS supports the loading of multiple code and data segments, with a current adjustable default limit of five segments, allowing programs of 320k+ size to run.

So it means more than 320kb can be allocated.

[toncho11]

But it does compile when I change -Wl,stack=0x1000 and -Wl,heapsize=0x1000 to 0xa00.

So you set both to 2560 bytes?

[FrenkelS]

Yes, it's just a number to make it compile and link.

The next problem is running it in ELKS. file doomtd3.os2 says it's a shell script, so when I do ./doomtd3.os2 I get

MZÇ☺♦  ╕@Ç♫▼║♫┤: not found                                                      
./doomtd3.os2: 2: Syntax error: ")" unexpected                                  

[ghaerr]

Error! E3173: default data segment exceeds maximum size by 3008 bytes

For any Watcom, OS/2 or ELKS executable, in any memory model, the "default" data segment is the "static data" segment where all C strings are put, as well as the stack, heap from malloc, etc must be. This message means that the combination of the Doom data declarations that are not __far or over specified minimum amount (there's an OW option for this, I forget its name), is 3008 bytes > 64K, the max allowed.

But it does compile when I change -Wl,stack=0x1000 and -Wl,heapsize=0x1000 to 0xa00.

I'm not sure if Doom can operate with a 2560 byte stack, nor whether it needs any malloc heap at all. You'll have to determine that and adjust either up or down accordingly. The default data segment does now fit into 64k, which is great, the executable can run to be initially tested.

The next problem is running it in ELKS. file doomtd3.os2 says it's a shell script

The reason for this is the ELKS by default doesn't run OS/2 executables. In order to configure that, set CONFIG_EXEC_OS2 in your .config file, or set "Support OS/2 Executables" on in the Filesystem config screen. Then make kclean and make and it should run it. Good luck! :)

[ghaerr]

So it means more than 320kb can be allocated.

Yes, but only in additional data or code segments. Code is automatically put in another segment when compiling in medium or large model, and data declarations over a certain size passed to the Watcom compiler are also put into another data segment (instead of the default data segment) when compiling in compact or large models. The wcc option to control this is -zt<number>, where <number> is 32767 if the option is not used, or 256 if not specified. Using this option may be necessary to move certain data declarations to their own segment, which will free up space in the default data segment.

If using owcc, then I think you have to use -Wc,-zt256 for instance.

Here's the manual page for the -zt option:

[toncho11]

• You reduced (I suppose the previous default value was higher) the stack and heap size to 2560 so that it can fit in the "default" data segment which is 64 kb. But this means that Doom might not run with this lower level of stack and heap.
• You need to recompile the ELKS distribution with OS2 executables enabled. @ghaerr I thought this was by default now enabled.
• the zt option can be used to place certain objects larger than N bytes to another data segment by implicitly declaring some data objects as far. Interestingly it does not mention "medium model", so you need to switch to "large" in order to try it.

[FrenkelS]

I took the latests master build and I had to change fopen("DOOMTD3L.WAD", "rb") to fopen("doomtd3l.wad", "rb"), but now I got something running doomelks.zip

Too bad it fails with an out of memory error: Z_Malloc: failed to allocate 14676 B, max free block 9264 B, total free 13104

[toncho11]

Bravo! Awesome job!!!!

Now there will be a process of tuning I suppose:

• reduce memory usage in code
• change compiler settings until it fits

But the fact that it compiles (and starts) on a new OS with a rather new compiler (new in ELKS) is a huge feat!!!

[toncho11]

So the https://github.com/FrenkelS/doomtd3/blob/9770b50c99e585307c8aa0ca9c091a73c1c506b6/z_zone.c#L319 is unable to reserve enough memory. It is the way the memory is reserved or the OS can not provide a large enough block of memory? I think it is the second?

[FrenkelS]

The memory manager isn't that different from the one in vanilla Doom. That one is described in Chapter 5.7 of the Game Engine Black Book DOOM.

doomtd3 tries to malloc 550 kB in I_ZoneBase(). If that fails it decreases the amount with 16 bytes and tries again until it's successful. In my case it can successfully malloc 178 kB. In other words, the largest block of contiguous memory that ELKS can supply in my case has a size of 178 kB. Doom manages this block and divides it into smaller blocks when the game needs memory. In my case it fails because there isn't a free block that's at least 14676 bytes long to load the Doom level data (sidedefs) into.

[ghaerr]

doomtd3 tries to malloc 550 kB in I_ZoneBase(). Doom manages this block and divides it into smaller blocks when the game needs memory.

I see. So the game should be started directly after boot, and possibly optimize/configure ELKS to use as little main memory as possible (I'll take a look at how to do that). After grabbing the max contiguous RAM, does Doom need to be compiled in compact or large model in order to manage that memory, or does it use internal far pointers? If it uses internal far pointers, then perhaps it can be compiled small model.

Which model is being used for compilation now?

Does Doom use malloc, or only use its own internal allocator after having grabbed all the main memory it can? This will help me think about how to best get to the next step moving forward on this port.

If Doom is code segment constrained, use medium model. If default data segment constrained, use compact. If both, then large model is possibly required. It sounds like it might be possible to use small model unless code segment restrained, and do all the memory management/allocation internally with __far pointers.

Can Doom do another allocation using TryZMalloc? I'm wondering whether it could use an extra 64k bytes from UMB-configured memory.

[ghaerr]

need to recompile the ELKS distribution with OS2 executables enabled. I thought this was by default now enabled.

That's on my list and will be enhanced shortly, thanks for the reminder!

[toncho11]

It is -mcmodel=m so medium model.

[ghaerr]

It is -mcmodel=m so medium model.

Interesting why ia16-elf-gcc wouldn't work, but I think OW's a better choice here, and it allows more enhancements if necessary.

It sounds like Doom is lots of code, tight in the default data segment, but using internal far pointers. Staying with medium model will save the most data space, since it sounds like its already coded to handle far memory outside the data segment.

Possibly not running /etc/rc.sys and logging in as 'toor' (to run the much smaller /bin/sash shell) may allow for a larger block of contiguous memory. Run meminfo to learn what main memory fragmentation looks like.

[FrenkelS]

doomtd3 only supports map e1m7. E1m1 is a smaller level and Doom8088 can run it with only 178 kB of memory...

I'm wondering whether it could use an extra 64k bytes from UMB-configured memory.

Yes that is possible. I_ZoneAdditional() is used for that purpose. In the DOS version that function is implemented by using 64k of EMS without bank switching.

[FrenkelS]

Right now it uses the medium memory model. It can compile using the small memory model and 286 instructions. 8086 instructions results in Error! E2021: size of segment _TEXT exceeds 64k by 1483 bytes.

[ghaerr]

Yes that is possible. I_ZoneAdditional() is used for that purpose. In the DOS version that function is implemented by using 64k of EMS without bank switching.

That's good then, ELKS can use UMB directly w/o bank switching as well. If doomtd3 needs more memory in order to run on ELKS, perhaps try enabling up to 64k UMB memory using by uncommenting the umb= line in /bootopts, then try grabbing another 64k using fmemalloc and hand it off to I_ZoneAdditional. This may not always work depending on how you're got ELKS setup for system buffers, as that may be grabbed out of the UMB area during the boot process, depending on how many L1 or L2 buffers you have set in /bootopts as well. meminfo will show the memory layout before running doomtd3.

In order to reduce L2 system buffers and L1 cache buffers radically (1K each), you can use buf=8 and cache=4 in /bootopts respectively. You won't want any xmsbuf= set. Setting init=/bin/sash in /bootopts will skip running /bin/init and the large /bin/sh and /etc/rc.sys processing. This should open up more memory and reduce any boot-time fragmentation.

I'm thinking of adding a heap= enhancement to /bootopts, this would reduce the size of the ELKS kernel near heap, we might be able to gain 30K additional by doing that.

[toncho11]

I used

buf=16
cache=16
init=/bin/sash

to get 30 kb more. I am testing in this online emulator. With UMB mapped as shown in /bootopts you get 95kb more. To use this emulator you need to load the latest hdd image of ELKS to have the OS2 execution enabled. New images of any OS are loaded here: https://copy.sh/v86/. Scroll down for the image uploading.

So the final result checking with meminfo is 505kb free and before it was 331kb. Not bad!!!

Also another option is to set: init=/your_folder/doomtd3 and this will skip sash as well. This will reduce further memory and fragmentation, but you won't be able to issue commands such as edit/vi/meminfo after exiting doom.

[FrenkelS]

I'll probably abandon this branch and continue work on https://github.com/FrenkelS/elksdoom Next issues: how to implement keyboard input?

[toncho11]

I thought the first step was to make sure this time demo works. This will help the ELKS port of Doom8088. Is this the latest binary version of the timedemo https://github.com/FrenkelS/doomtd3/pull/12#issuecomment-2314672915

[ghaerr]

how to implement keyboard input?

Pretty straightforward, with the possible exception of ANSI keyboard sequence ESC handling: use struct termios to set the keyboard (file descriptor 0) into RAW mode, then just read(0, &c, 1) to read a character, UNIX style. Look at a Linux version of Doom, this should port right over.

buf=16 cache=16

With the updated L2 cache code added some time ago, you can go quite lower than this to buf=8 and cache=4 and still get decent performance if you're not doing much disk I/O. init=/path/to/doom is a nice idea to save even more ram.

[toncho11]

I lowered the memory to 517KB free. I execute it with ./doomtd3.os2 and nothing happens even after 3 minutes. ?????

[ghaerr]

I lowered the memory to 517KB free.

Run meminfo with your setup and lets see the main memory layout.

[toncho11]

In the emulator is it clear that the CPU goes and stays high. Maybe it is related to https://github.com/FrenkelS/doomtd3/issues/13

FAT: total 31M, fat16 format                                                    
VFS: Mounted root device 0300 (msdos filesystem).                               
# meminfo                                                                       
  HEAP   TYPE  SIZE    SEG   TYPE    SIZE  CNT  NAME                            
  41be   SEG     16   26ac   free  495936    0                                  
  41da   SEG     16   c000   BUF    16384    1                                  
  41f6   SEG     16   d000   CSEG   29904    1  /bin/sash                       
  4212   TASK 14112                                                             
  793e   INOD  7488                                                             
  968a   FILE   896                                                             
  9a16   BUFH 16384                                                             
  da22   BUFH   320                                                             
  db6e   SEG     16   c400   free   16384    0                                  
  db8a   SEG     16   1464   CSEG    3984    1  meminfo                         
  dba6   TTY     80                                                             
  dc02   TTY     80                                                             
  dc5e   SEG     16   d74d   DSEG   20448    1  /bin/sash                       
  dc7a   SEG     16   dc4b   free   15184    0                                  
  dc96   free    16                                                             
  dcb2   SEG     16   155d   DSEG    3280    1  meminfo                         
  dcce   SEG     16   162a   free    2080    0                                  
  dcea   free  8982                                                             
  Heap/free   48718/ 8998 Total mem  603584                                     
  Memory usage  589KB total,   72KB used,  517KB free                           
#                                

[toncho11]

This is my hdd image with doom in /bin hd32-fat_doom.zip

[toncho11]

We have to try it on qemu 386 for example.

[ghaerr]

Thanks for the meminfo. Unfortunately it doesn't show the size of the unallocated main memory, only free segments previously allocated in the kernel near heap. I guess I should add an option to show the main memory usage list directly. I think doomtd3 as written only uses the largest contiguous memory segment.

[toncho11]

Maybe I am not getting anything because it is either:

• a 286 build on a 8086 emulator (I think the emulator is not restricted 8086 instructions)
• it is a 8086 build but affected by https://github.com/FrenkelS/doomtd3/issues/13
• a 286 build that still hangs up because of memory issue or other in the emulator that I use

[toncho11]

Thanks for the meminfo. Unfortunately it doesn't show the size of the unallocated main memory, only free segments previously allocated in the kernel near heap. I guess I should add an option to show the main memory usage list directly. I think doomtd3 as written only uses the largest contiguous memory segment.

Maybe you can even calculate the largest current free memory block as an extra info (which will be quite useful here).

[toncho11]

Even after 20 minutes the CPU is still working on something, but just see the command line and nothing happened.