Automate historical makefiles generation

lhsazevedo commented 3 years ago

Here is a simple test python script for generating historical makefiles using a template. Only amiga makefiles for now, but should be easy to add msdos and smake.

It still needs some adjustments to output makefiles equivalent to the current ones, see the diff between the current makefile and the generated one.

I'll get back to this soon.

lhsazevedo commented 3 years ago

Might help #356

cr1901 commented 3 years ago

We already require Python to generate docs. I'm fine with this change- it can be the CMake equivalent for the historical architectures.

Maybe this script could even evolve the ability to zip up the contents for xfer to an old machine (Ville, what do you do to xfer the source tree to Amiga, seeing that I doubt git runs there)?

vhelin commented 3 years ago

This sounds like a great idea!

@cr1901 - I sold my Amiga as I wasn't using it much, using WinUAE now.

vhelin commented 3 years ago

Also, having such script in GitHub would be super nice, but we still need to keep on having the (generated) historical makefiles there as well as for example people who use Amiga/MSDOS might not have python executables etc...

lhsazevedo commented 3 years ago

We can add a step to the CI pipeline that, for every commit on master, run the script and commit the new makefiles if there are changes. Something like:

python script.py

if git diff --no-patch --exit-code; then
  git commit -am "Update historical makefiles"
  git push
fi

Finishing the script

Now I just need to figure out the patterns to the object dependencies, if there is one. Each object depends just on i'ts own source and headers, defines.h, and makefile:

main.o: main.c defines.h main.h makefile
parse.o: parse.c defines.h parse.h makefile
include_file.o: include_file.c defines.h include_file.h makefile

But printf.o does not depends on defines.h

printf.o: printf.c printf.h makefile

Opcodes objects depends only on the .c source:

opcodes_65c02_tables.o: opcodes_65c02_tables.c
opcodes_65c02.o: opcodes_65c02.c

Would removing defines.h from vendor objects be a problem? If not, we can use tree groups for generation:

WLA objects (main, parse, pass_1 etc): @file.o: @file.c defines.h @file.h makefile
Opcodes object: @file.o: @file.c
Other objects (crc32, printf, mersenne_twister etc): @file.o: @file.c @file.h makefile

vhelin commented 3 years ago

Would removing defines.h from vendor objects be a problem?

I'm not 100% sure, but you can try it. Personally I have nothing against it.

lhsazevedo commented 3 years ago

So, I was able to compile on MS-DOS 7.1 with DJGPP running on DOSBox-X! Note that I was still using historical makefiles, instead of generated. I'm doing this to try to understand makefiles and MS-DOS.

Unfortunally 65816, 65c02, 65ce02, huc6280, spc700 were not built, but I suspect it's because the 8dot3 short name limitation, as their output binaries have long names. They are in debug mode, sorry :) Download here

How to build

As there is little documentation and I don't know if I did things correctly, I'll list what I did:

Host (Linux) steps:

Setup DOSBox-X, DJGPP, WLA
Copy .\historical\* to .\ (repo root)
Change -m486 flag to -mcpu=i486 in all makefiles (gcc doesn't recoginizes)
Add NDEBUG=1 definition to all makefiles to avoid building in Debug mode (forgot this)
Patch line 32 of printf.c from: #ifdef AMIGA to #if defined(AMIGA) || defined(MSDOS) so PRINTF_DISABLE_SUPPORT_EXPONENTIAL is set to 1
Copy previously built opcodes tables to repo root cp build/op_tbl_gen/opcodes_* ./
Remove huc6280 trailing make clean at msdos.bot, to keep crc32.o, hashmap.o and printf.o objects for wlalink build

MS-DOS steps:

Navigate to repo root and run msdos.bat :smiley: You might want to put CPU cycles at max, activate turbo mode and frame skipping here haha

Warnings

Only the following warnings were given during the build:

gcc.exe: warning: '-mcpu=' is deprecated; use '-mtune=' or '-march=' instead

gcc -c -O3 -ansi -pedantic -Wall -DMSDOS -DGB hashmap.c
hashmap.c: In function 'hashmap_hash_int':
hashmap.c:75:3: warning: this decimal constant is unsigned only in ISO C90
   75 |   key = (key >> 3) * 2654435761;
      |   ^~~

Dependency graph

Generated using makefile2graph. I'm almost sure that historical makefiles aren't optimal but it works :shrug: make -Bnd | make2graph | sed '2i ranksep=2' | dot -Tpng -o my_graph.png

msdos-z80

Results

Now I should be able to create a working makefile template to genenate MS-DOS makefiles.

lhsazevedo commented 3 years ago

So, gcc can generate a cmake compatible dependency file for automatic dependency tracking. This should work on Amiga and MSDOS when using gcc.

It apears that the SAS/C compiler also can generate this list with the List and ListInclude options, see page 110 here.

vhelin commented 3 years ago

It apears that the SAS/C compiler also can generate this list with the List and ListInclude options, see page 110 here.

Next time I'm booting WinUAE I'll check this out!

vhelin commented 1 year ago

I have to admit that I haven't tried out List and ListInclude options for SAS/C, but I've used the Amiga smakefiles your Python script has generated and they've been working really well.

lhsazevedo commented 1 year ago

Thanks for the feedback! I'm glad I was able to contribute with the project in some way, and I hope continue to do so in the future.

PS: Should we consider this issue as resolved?

vhelin commented 1 year ago

I haven't been able to test the MSDOS makefiles and perhaps wouldn't know how to compile WLA DX under MSDOS, but you had some experience there.

Hmm... I'm now kind of thinking that we should rename all *.c files so they fit into 8.3 chars so compiling under MSDOS can be done 100%. What do you think of that?

vhelin commented 1 year ago

What do you think, do these two commits enable WLA DX to be compiled on MSDOS? If not, what else should be done?

Should folders like byte_tester to be renamed to 8.3 as well?

lhsazevedo commented 1 year ago

From what I recall, I was able to successfully compile compile on MSDOS except for the binaries with long filenames.

At the time, they were:

88888888.333
wla-65816.exe
wla-65c02.exe
wla-65ce02.exe
wla-huc6280.exe
wla-spc700.exe

But I guess it's safer to make the sources comply with the 8.3 rule, just in case.

I'll try again next sunday (8) and report back here!

vhelin commented 1 year ago

You mean the number of characters on the final executable name, if they go over 8.3, failed the building process? Hmm, why not make MSDOS build bat output

88888888.333
w65816.exe
w65c02.exe
w65ce02.exe
whuc6280.exe
wspc700.exe

?

lhsazevedo commented 1 year ago

Hmm, looks like we already did that back in https://github.com/vhelin/wla-dx/commit/fad4cec2639e51e39427e35677ef279a261580ae

I've built successfully in DOXBox-X:

I didn't enough have time to test in FreeDOS, but I can try next weekend.

vhelin commented 1 year ago

The time difference in those exes is kinda big. Did you build those exes one by one by hand for some reason? Long filename problems somewhere else still?

lhsazevedo commented 1 year ago

Yes, I built the targets manually (using the -t flag e.g. make -t wla-z80). No issues with long file names, but I would like to build all of the targets on a real DOS VM just to be on the safe side. I might as well record a video of the process for future reference.

vhelin commented 1 year ago

That sounds really good, the video thing I mean!

lhsazevedo commented 1 year ago

Managed to build all variants, on a VM running FreeDOS:

Indeed, long file names were a problem when using the msdos.bat script, but it was an easy fix. The Instruction Table Generator BAT script also needed some tweaks and its MAKEFILE was missing (didn't noticed before as I was reusing the tables generated on my host machine).

Unfortunately I ran out of time for this weekend, but I'll open an PR with the fixes + video by the next sunday.

Later, I plan to test with an OS and compiler that doesn't support LFN (MS-DOS 6 and Open Watcom).

vhelin commented 1 year ago

Great work! :)

cr1901 commented 1 year ago

MS-DOS 6 and OpenWatcom

OpenWatcom requires a 386 or above to run, but can target the 8088. I'm curious what the binary size will look like when opts are enabled.

The Windows version of OpenWatcom historically could compile WLA-DX just fine using CMake targeting 32-bit Windows (cross compiling w/ CMake is "eh..."), but it should be possible to target OpenWatcom with a new Makefile.

vhelin commented 1 year ago

Can WLA even run on MSDOS? I mean didn't it have some sort of 640KB memory limit and to go beyond that the program needs to access extended memory via some kind of a mechanism? Can the C compiler that was used to build WLA DX on MSDOS take care of that by itself?

lhsazevedo commented 1 year ago

From what I read this past week, the program needs an DPMI server/host installed to access extended memory. It seems that DJGPP handles this for us. Maybe Open Wacom does it as well?

vhelin / wla-dx