Configure Builds with Lua

Lethja commented 5 months ago

This pull request has been added in a draft state for feedback purposes...

This pull request adds a Lua script that can determine and address the differences between different types of build platforms. The script is inspired by GNU Autotools but thanks to Lua is completely platform agnostic.

The script aims to address in the current build system like:

72
The need to run/emulating pre-compiled binaries to configure successfully
Makefiles being generated with the wrong platform specific commands (rm vs del)
Certain compiler flags not being available on older versions of the compiler (-fdiagnostics-color=never)
Single source (configur.lua for everyone instead of configur.bat vs configur.sh)

Note: While FreeDOS has shipped Lua for some time on their bonus CD this version uses a DOS extender and could freeze if another DOS extender is started at the same time. https://github.com/Lethja/lua-watcom allows Lua to be built for real mode to avoid this problem. Luas official makefile is excellent at building non-DOS 32-bit targets.

gvanem commented 5 months ago

I do not see that lua configur.lua [ -h | --help ] is handled. Users would like to know what this script does before using it.

Lethja commented 5 months ago

I do not see that lua configur.lua [ -h | --help ] is handled. Users would like to know what this script does before using it.

Excellent point. b5eff3fa560a17a650a3e2ab1d9836ca286eb15f has added this

$ lua configur.lua
Specify a makefile to generate, options are:
    cc  - Handle everything yourself with CC,CFLAGS,LD enviroment variables
    clang   - A GCC compatible alternative compiler
    djggp   - A port of GCC for 80386+ DOS systems
    gcc - GNU Compiler Collection intended for Posix systems
    mingw   - A port of GCC for Win32 based systems
    watcom  - A C/C++ toolchain that can build 16-bit DOS programs

Use '/?' or '-h' to get full help

$ lua configur.lua /?
configur.lua [TARGET] [OPTIONS]
Configures and generates Watt-32 makefiles regardless of host enviroment using Lua
A platform agnostic alternative to `src/configur.sh` and `SRC\CONFIGUR.BAT`

[TARGET]:
    cc  - Handle everything yourself with CC,CFLAGS,LD enviroment variables
    clang   - A GCC compatible alternative compiler
    djggp   - A port of GCC for 80386+ DOS systems
    gcc - GNU Compiler Collection intended for Posix systems
    mingw   - A port of GCC for Win32 based systems
    watcom  - A C/C++ toolchain that can build 16-bit DOS programs

[OPTIONS]:
    -h, --help, /? - Show this help
    -s, /s         - Skip compiler checks

gvanem commented 5 months ago

Does it need normal stock Lua 5.x or does it work with LuaJIT too?

Lethja commented 5 months ago

Does it need normal stock Lua 5.x or does it work with LuaJIT too?

Up until reading this comment this script has been tested using the official Lua 5.4.6 on x86_64 Linux and DOS Real Mode (via this build).

I just built and tested the script running under LuaJIT, initial results: looks like it works without issue

$ ~/Git/Repositories/luajit/src/luajit configur.lua gcc
Determining operating system family... Unix
Checking 'mkdir' works... Yes
Checking 'rm' works... Yes
Checking 'rm -R' works... Yes
Checking 'inc' contains required files... Yes
Checking 'src' contains required files... Yes
Checking 'CC' enviroment variable... None
Guessing compiler... GNU Compiler Collection or compatible
Checking 'gcc' is available... Yes
Checking if 'gcc' understands '-fdiagnostics-color=never'... Yes
Checking size of 'int' C type in bits... 32
Checking size of 'long' C type in bits... 64
Checking if Lua configuration script is finished and ready to use... No

Lethja commented 4 months ago

I'm not 100% on what cflagsbf.c output is suppose to look like. It'll take time to fully decipher and replicate all the moving parts of makefile.all and the util binaries.

For the time being I've pushed ef743bf which will generate a rough mockup (probably not working) of a Watcom makefile when running lua configur.lua wcc -s. Watcom doesn't need to be installed when -s is used

jwt27 commented 4 months ago

This is a neat idea, I'm all for it. However, I don't see how it fully solves:

compiling on linux #72

The need to run/emulating pre-compiled binaries to configure successfully

Specifically, generating the error code macros and sys_errlist[]. This needs access to the target libc's <errno.h> to put them in the right order.

The solution, I think, is to preprocess this with the target compiler, and then make a native (build-time) binary that generates the headers. This implies that the configure script must distinguish between native and target toolchains, eg via autoconf-style variables such as CC_FOR_BUILD/CC_FOR_HOST. It gets complicated quick.

edit: Also, if your intention is to eventually merge #89 into this, the following may be worth considering beforehand:

Ability to do out-of-tree builds (like autotools)
Separate targets for each of Watcom's 16-bit memory models

gvanem commented 4 months ago

I'm not 100% on what cflagsbf.c output is suppose to look like.

It's cflagsbf.h.

E.g. build\clang\64bit\release\cflags.h is:

const char *w32_cflags = "-nologo -Zi -Zo -GS- -I. -I../inc -DWATT32_BUILD -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_OBSOLETE_NO_WARNINGS -D_WIN32_WINNT=0x0601 -D_WINSOCK_DEPRECATED_NO_WARNINGS -DBUILD_WINDOWS -Wno-#pragma-messages -MD -Ox -fms-compatibility";
const char *w32_cc     = ""f:\ProgramFiler\LLVM-17.0\win64/bin/clang-cl.exe"";

Hence those 3 \ (= 0x5C) above must be hex-ified into build\clang\64bit\release\cflagsbf.h. A hex-view of build\clang\64bit\release\cflags.h:

00000000  63 6F 6E 73 74 20 63 68 61 72 20 2A 77 33 32 5F    const char *w32_
00000010  63 66 6C 61 67 73 20 3D 20 22 2D 6E 6F 6C 6F 67    cflags = "-nolog
00000020  6F 20 2D 5A 69 20 2D 5A 6F 20 2D 47 53 2D 20 2D    o -Zi -Zo -GS- -
00000030  49 2E 20 2D 49 2E 2E 2F 69 6E 63 20 2D 44 57 41    I. -I../inc -DWA
00000040  54 54 33 32 5F 42 55 49 4C 44 20 2D 44 5F 43 52    TT32_BUILD -D_CR
00000050  54 5F 53 45 43 55 52 45 5F 4E 4F 5F 57 41 52 4E    T_SECURE_NO_WARN
00000060  49 4E 47 53 20 2D 44 5F 43 52 54 5F 4E 4F 4E 53    INGS -D_CRT_NONS
00000070  54 44 43 5F 4E 4F 5F 57 41 52 4E 49 4E 47 53 20    TDC_NO_WARNINGS 
00000080  2D 44 5F 43 52 54 5F 4F 42 53 4F 4C 45 54 45 5F    -D_CRT_OBSOLETE_
00000090  4E 4F 5F 57 41 52 4E 49 4E 47 53 20 2D 44 5F 57    NO_WARNINGS -D_W
000000A0  49 4E 33 32 5F 57 49 4E 4E 54 3D 30 78 30 36 30    IN32_WINNT=0x060
000000B0  31 20 2D 44 5F 57 49 4E 53 4F 43 4B 5F 44 45 50    1 -D_WINSOCK_DEP
000000C0  52 45 43 41 54 45 44 5F 4E 4F 5F 57 41 52 4E 49    RECATED_NO_WARNI
000000D0  4E 47 53 20 2D 44 42 55 49 4C 44 5F 57 49 4E 44    NGS -DBUILD_WIND
000000E0  4F 57 53 20 2D 57 6E 6F 2D 23 70 72 61 67 6D 61    OWS -Wno-#pragma
000000F0  2D 6D 65 73 73 61 67 65 73 20 2D 4D 44 20 2D 4F    -messages -MD -O
00000100  78 20 2D 66 6D 73 2D 63 6F 6D 70 61 74 69 62 69    x -fms-compatibi
00000110  6C 69 74 79 22 3B 0A 63 6F 6E 73 74 20 63 68 61    lity";.const cha
00000120  72 20 2A 77 33 32 5F 63 63 20 20 20 20 20 3D 20    r *w32_cc     = 
00000130  22 22 66 3A 5C 50 72 6F 67 72 61 6D 46 69 6C 65    ""f:\ProgramFile
00000140  72 5C 4C 4C 56 4D 2D 31 37 2E 30 5C 77 69 6E 36    r\LLVM-17.0\win6
00000150  34 2F 62 69 6E 2F 63 6C 61 6E 67 2D 63 6C 2E 65    4/bin/clang-cl.e
00000160  78 65 22 22 3B 0A                                  xe"";.          ```

Otherwise the C pre-processor would complain. See version.c for it's purpose. Was that what you was uncertain of?

Lethja commented 4 months ago

Specifically, generating the error code macros and sys_errlist[]. This needs access to the target libc's <errno.h> to put them in the right order.

The solution, I think, is to preprocess this with the target compiler, and then make a native (build-time) binary that generates the headers. This implies that the configure script must distinguish between native and target toolchains, eg via autoconf-style variables such as CC_FOR_BUILD/CC_FOR_HOST. It gets complicated quick.

Indeed! I had not thought of that, that's a tough problem to solve. Worst still CheckCompilerIntSize() and CheckCompilerLongSize() depend on target system execution to give the correct result. Unless I missed something a pre-processor check isn't good enough for CheckCompilerIntSize() since wcc pre-processor lies about how big int is and others might to.

The immediate workaround to these cross-compiling problems would be:

Use the /s or -s flag (which will skip tests and fill in the values with guestimates)
Setup Lua and a native C toolchain in a emulator/VM enviroment
- Everywhere to DOS: DOSBox-X
- Everywhere to everywhere: QEMU, Virtualbox
- Linux to DOS: dosemu
- Linux to Windows: Wine (wine cmd)
- Windows to Linux: WSL2

It's cflagsbf.h. Was that what you was uncertain of?

Yes, I assumed it was more complex then a direct string conversion. It should be trivial to write a function in the script that converts cflags.h into it's hex array counterpart for cflagsbf.h.

Lethja commented 2 months ago

Apologies for lack of update on this. I hope to make some time to work on this again in the future but I'll close PR until such time

Lethja commented 1 month ago

I've started working on this PR again but I'm keeping it closed to prevent appveyor from doing irrelevant builds every time I push a commit. I assume that it cost money and/or take computing power to run and the CI scripts currently don't test the Lua scripts at all.

In its current state, configur.lua can test Open Watcom and GCC-like compilers and OS capabilities. It can also generate some of a wmake style makefile. The current output watcom_l.mak looks like this:

#
# NB! THIS MAKEFILE WAS AUTOMATICALLY GENERATED.
# After making manual edits, be sure to save with a different name
# otherwise your changes might be overwriten.
#

# This makefile builds Watt-32 for large model 16-bit DOS
#
#  WCC-flags used:
#   -bt=dos   target system - DOS
#   -m{s,l}   memory model; small or large
#   -0        optimise for 8086, register call convention
#   -s        no stack checking
#   -zq       quiet compiling
#   -zc       place const data into the code segment
#   -d{1,2,3} generate debug info
#   -os       optimization flags
#     s:      favour code size over execution time
#

# Build Binaries
AS  = wasm
AR  = wlib
CC  = wcc
LUA = C:\DOS\DEVEL\LUA\BIN\LUA16.EXE

# Build Flags
AFLAGS   = -bt=dos -zq -w3 -d1 -I"../inc"
CFLAGS   = -bt=dos -ml -0 -os -s -zc -zm -zlf -DWATT32_STATIC -zq -wx -DWATT32_BUILD -I. -I"../inc" -d1

# Project Paths
BINPATH = ..\bin\
LIBPATH = ..\lib\
LUAPATH = ..\lua\
OBJPATH = build\watcom\large\

# Watcom Arguments
C_ARGS    = $(OBJPATH)wcc.arg
LIB_ARGS  = $(OBJPATH)wlib.arg
LINK_ARGS = $(OBJPATH)wlink.arg

# Output library
STAT_LIB = $(LIBPATH)wattcpwl.lib

# Source code
ASM_SOURCE = asmpkt.asm cpumodel.asm
BIND_SOURCE = res_comp.c res_data.c res_debu.c ...
BSD_SOURCE = accept.c bind.c bsddbug.c ...
CORE_SOURCE = bsdname.c btree.c chksum.c ...

# Objects to build
OBJS = $(OBJPATH)cpumodel.obj $(OBJPATH)accept.obj $(OBJPATH)bind.obj ...

When combined with the make rules from the old generator it builds as expected with two caveats. BIN2C.EXE and WC_ERR.EXE still need to be run first. BIN2C.EXE can easily be replaced by a lua script hence the LUA and LUAPATH variables in the new output. The change to a generated makefile may look something like this:

-W32_BIN2C = ../util/linux/bin2c
+LUA = lua
+LUAPATH = ../lua/

 $(OBJPATH)cflagsbf.h: $(C_ARGS)
-   $(W32_BIN2C) $(C_ARGS) > $^@
+   $(LUA) $(LUAPATH)bin2c.lua $(C_ARGS) > $^@

As jwt27 mentioned earlier replacing WC_ERR.EXE will be far more tricky. Their suggestion to run the targets pre-processor and compile the processed code on the build systems native compiler is clever and I can't think of a better solution. configur.lua already runs a test to check if the targets compiler is native to the build system and knows about wcc -P and gcc -E -P but doesn't make use of them yet.

Lethja commented 1 month ago

As of 5debd9c81a1cb56e082d2ffa66279f2d3ccfae72 configur.lua can generate makefiles which will build the library for DOS successfully.

Tested successfully

Makefile	Built from	Built for	Comment
`DJGPP.MAK`	DOSX	DOSX	`NASM` required
`DJGPP.MAK`	Linux	DOSX	`DOSEMU` & `NASM` required
`WATCOM_L.MAK`	DOSX	DOS
`WATCOM_F.MAK`	DOSX	DOSX

Untested but assumed to work

Makefile	Comment
`WATCOM_3.MAK`	Should work, identical to `WATCOM_F.MAK` besides `CFLAGS`
`WATCOM_S.MAK`	Should work, identical to `WATCOM_L.MAK` besides `CFLAGS`
`WATCOM_W.MAK`	Watcom tools specific WIN32 untested

Now that there's something worth testing I will re-open this PR

gvanem / Watt-32

Configure Builds with Lua #114

72

Tested successfully

Untested but assumed to work