build on osx arm64 host is not working

[DelilahHoare]

Hi. Trying to build Open Watcom V2 on the arm64 version of macos fails with

wmake bootstrap build error

build/binbuild/bootx.log ends with

../c/mglob.c:117:10: error: Unknown CPU architecture
        #error Unknown CPU architecture
         ^
1 error generated.
make: *** [mglob.o] Error 1

It would be nice to be able to build for this host.

[jmalak]

The main reason why it is not supported is that we don't have such build platform to build it. Generally it should be possible , but it requires to add new build host architecture osxarm. It already exists in main configuration files, but need to add binary build subdirectory with appropriate makefile.

Anyway I checked definition for OSX and there is defined ARM architecture in wmake.

#elif defined( __OSX__ ) || defined( __APPLE__ )
    "__OSX__=\n"
    "__UNIX__=\n"
    #if defined( _M_X64 ) || defined( __x86_64__ ) || defined( __amd64__ ) || defined( __amd64 )
        "__OSXX64__=\n"
    #elif defined( _M_IX86 ) || defined( __i386 ) || defined( __i386__ )
        "__OSX386__=\n"
    #elif defined( __PPC__ ) || defined( __ppc__ ) || defined( __powerpc__ )
        "__OSXPPC__=\n"
    #elif defined( _M_ARM ) || defined( __ARM__ ) || defined( __arm__ )
        "__OSXARM__=\n"
    #else
        #error Unknown CPU architecture
    #endif

It looks like you don't have properly configured OW build environment.

[jmalak]

Please what you get if you run uname command on ARM system?

[jmalak]

I don't know how you configure OW build system.

Below is important section about creating OW build system setup script from setvars.sh script.

Start by modifying your copy of setvars.bat (DOS/Windows), setvars.cmd (OS/2), or setvars.sh (Linux) to reflect your system. Copy appropriate setvars script outside OW directory tree and rename it to something useful (you can have multiple files with different configuration). See the comments in the file for additional information. Of particular interest is the OWROOT variable that specifies the location of the OW source tree and the OWTOOLS variable that specifies which build compiler you intend to use.

[DelilahHoare]

uname prints Darwin. (This is macos 12.4) I copied setvars.sh and set OWTOOLS=clang and OWDOSBOX=dosbox-x, left the rest the same and sourced it and ran build.sh. I've now also set OWVERBOSE=1.

Those macros being checked for __OSXARM__ are not predefined on this system. cc -dM -E -x c /dev/null lists __aarch64__, __arm64, and __arm64__. If I patch mglob.c to check for __arm64__ there, the build gets far, but eventually fails.

bwasm -q -D_BLDVER=1300 -D_CYEAR=2022 ../../../a/fdmth086.asm -I"../../../h" -I"/Users/delilah/open-watcom-v2/bld/clib/h"  -I"/Users/delilah/open-watcom-v2/bld/watcom/h" -I"/Users/delilah/open-watcom-v2/bld/lib_misc/h" -I"/Users/delilah/open-watcom-v2/bld/comp_cfg/h"  -we -fo=.obj -mm-nt=_TEXT -fpc-w8-e-2  -bt=os2  -fpi87 -DNDEBUG
Abort
Error(E42): Last command making (fdmth086.obj) returned a bad status
Error(E02): Make execution terminated
<pmake -d build          -h> => non-zero return: 512
Build failed

I ran the failing command under lldb to get a backtrace (without -q here, although with it is exactly the same traceback except the address of argv)

delilah@Delilahs-Mini open-watcom-v2 % cd /Users/delilah/open-watcom-v2/bld/clib/cgsupp/library/os2.286/mm
delilah@Delilahs-Mini mm % lldb bwasm
(lldb) target create "bwasm"
Current executable set to 'bwasm' (arm64).
(lldb) run -D_BLDVER=1300 -D_CYEAR=2022 ../../../a/fdmth086.asm -I"../../../h" -I"/Users/delilah/open-watcom-v2/bld/clib/h"  -I"/Users/delilah/open-watcom-v2/bld/watcom/h" -I"/Users/delilah/open-watcom-v2/bld/lib_misc/h" -I"/Users/delilah/open-watcom-v2/bld/comp_cfg/h"  -we -fo=.obj -mm-nt=_TEXT -fpc-w8-e-2  -bt=os2  -fpi87 -DNDEBUG
Process 87981 launched: '/Users/delilah/open-watcom-v2/build/binbuild/bwasm' (arm64)
Open Watcom Assembler Version 2.0 beta Jul  2 2022 10:46:20
Copyright (c) 2002-2022 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1992-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See http://www.openwatcom.org/ for details.
../../../a/fdmth086.asm: 1055 lines, 0 warnings, 0 errors
Process 87981 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00000001b7262d98 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`__pthread_kill:
->  0x1b7262d98 <+8>:  b.lo   0x1b7262db8               ; <+40>
    0x1b7262d9c <+12>: pacibsp 
    0x1b7262da0 <+16>: stp    x29, x30, [sp, #-0x10]!
    0x1b7262da4 <+20>: mov    x29, sp
Target 0: (bwasm) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00000001b7262d98 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001b7297ee0 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x00000001b71d23cc libsystem_c.dylib`__abort + 128
    frame #3: 0x00000001b71c3d48 libsystem_c.dylib`__stack_chk_fail + 96
    frame #4: 0x0000000100002b20 bwasm`WriteObjModule at write.c:0:28 [opt]
    frame #5: 0x00000001000006dc bwasm`main(argc=<unavailable>, argv=0x000000016fdff468) at main.c:1347:5 [opt]
    frame #6: 0x000000010005908c dyld`start + 520
(lldb) 

[DelilahHoare]

I've rerun the command a lot of times now while poking at it, and it has succeeded once. The build was able to continue for a while longer before reaching a similar failure.

======== 11:15:21 /Users/delilah/open-watcom-v2/bld/fpuemu/386/dos386 =========
bwasm -q -D_BLDVER=1300 -D_CYEAR=2022 -we -d1 -DNDEBUG -fo=emu387.obj -3-w8 -bt=dos -fpc  -ms -D_OS=_PLDT -DSEGMENTED  -I"../inc" -I"../../inc" -I"/Users/delilah/open-watcom-v2/bld/mathlib/h"   -I"/Users/delilah/open-watcom-v2/bld/hdr/dos/h"   -I"/Users/delilah/open-watcom-v2/bld/watcom/h" ../asm/emu387.asm
Abort
Error(E42): Last command making (emu387.obj) returned a bad status
Error(E02): Make execution terminated
<pmake -d build          -h> => non-zero return: 512
Build failed

This was all without the new change you pushed adding more files for osx arm, but that hasn't affected the failures

[DelilahHoare]

I set OWDEBUGBUILD=1, but it didn't make the traceback any more useful

bwasm -q -D_BLDVER=1300 -D_CYEAR=2022 ../../../a/fdmth086.asm -I"../../../h" -I"/Users/delilah/open-watcom-v2/bld/clib/h"  -I"/Users/delilah/open-watcom-v2/bld/watcom/h" -I"/Users/delilah/open-watcom-v2/bld/lib_misc/h" -I"/Users/delilah/open-watcom-v2/bld/comp_cfg/h"  -we -fo=.obj -mm-nt=_TEXT -fpc-w8-e-2  -bt=os2  -fpi87 -d1
Abort
Error(E42): Last command making (fdmth086.obj) returned a bad status
Error(E02): Make execution terminated
<pmake -d build          -h> => non-zero return: 512
Build failed
Current usage: 0000000000001c88 bytes; Peak usage: 0000000000020337 bytes
  Who      Addr     Size     Call     Contents
======== ======== ======== ======== ===========================================
0000000000000000 000000013380a800 00000828 00000000000000c2 0082003401000000 180f16100200...4..........
0000000000000000 0000000134008200 00000828 00000000000000bf 008e803301000000 e80d16100200...3..........
0000000000000000 0000000133808e00 00000828 0000000000000002 0000000000000000 500d16100200........P.....
0000000000000000 0000000133808800 00000410 0000000000000001 0000000000000000 2f5573657273......../Users
00000004 chunks (0000000000001c88 bytes) unfreed

delilah@Delilahs-Mini mm % lldb bwasm 
(lldb) target create "bwasm"
Current executable set to 'bwasm' (arm64).
(lldb) run -q -D_BLDVER=1300 -D_CYEAR=2022 ../../../a/fdmth086.asm -I"../../../h" -I"/Users/delilah/open-watcom-v2/bld/clib/h"  -I"/Users/delilah/open-watcom-v2/bld/watcom/h" -I"/Users/delilah/open-watcom-v2/bld/lib_misc/h" -I"/Users/delilah/open-watcom-v2/bld/comp_cfg/h"  -we -fo=.obj -mm-nt=_TEXT -fpc-w8-e-2  -bt=os2  -fpi87 -d1
Process 98975 launched: '/Users/delilah/open-watcom-v2/build/binbuild/bwasm' (arm64)
Process 98975 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00000001b7262d98 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`__pthread_kill:
->  0x1b7262d98 <+8>:  b.lo   0x1b7262db8               ; <+40>
    0x1b7262d9c <+12>: pacibsp 
    0x1b7262da0 <+16>: stp    x29, x30, [sp, #-0x10]!
    0x1b7262da4 <+20>: mov    x29, sp
Target 0: (bwasm) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00000001b7262d98 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001b7297ee0 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x00000001b71d23cc libsystem_c.dylib`__abort + 128
    frame #3: 0x00000001b71c3d48 libsystem_c.dylib`__stack_chk_fail + 96
    frame #4: 0x00000001000060b4 bwasm`WriteObjModule at write.c:0
    frame #5: 0x0000000100002674 bwasm`main(argc=17, argv=0x000000016fdff448) at main.c:1347:5
    frame #6: 0x000000010007d08c dyld`start + 520

[jmalak]

Sorry, first of all you need to get proper bootstrap wmake and builder tools. It has no sense to continue without it because it doesn't build corectly anything. First you should determine what default compiler is used by your host system. If it is GCC or CLANG or other. Only GCC and CLANG are supported. If you know compiler then you setup it in your OW setup script, it is line (default is GCC) export OWTOOLS=GCC if you use CLANG then change it to export OWTOOLS=CLANG It has no sense to build anything other then wmake and builder tools. Initialize OW build environment by your setup script, you should get message about compile type and version. If it is OK then you can run next command sh build.sh preboot It should build only wmake and builder tools. Output is in bootx.log file. Don't change any build make file untill you fix this bootstrap build. If any problem then add here your bootx.log file to get info why it is not build correctly.

[jmalak]

I submited fix 307ce31d7f16b4bd9b0f8daf689aaf1e656414ae which should resolve your issue. Please refresh your source tree and check. As soon as you get proper wmake and builder tools all should be going OK. Next we need to add binary build for OSX ARM target. Below is list of new subdirectories for OSX ARM target which must be add with single make file makefile. ./bld/as/axp/osxarm ./bld/as/mps/osxarm ./bld/as/ppc/osxarm ./bld/aui/osxarm ./bld/cc/386/osxarm ./bld/cc/axp/osxarm ./bld/cc/i86/osxarm ./bld/cc/mps/osxarm ./bld/cc/ppc/osxarm ./bld/cfloat/osxarm ./bld/cg/intel/386/osxarm ./bld/cg/intel/i86/osxarm ./bld/cg/risc/axp/osxarm ./bld/cg/risc/mps/osxarm ./bld/cg/risc/ppc/osxarm ./bld/cpp/osxarm ./bld/dmpobj/osxarm ./bld/dwarf/dr/osxarm ./bld/dwarf/dw/osxarm ./bld/exedump/osxarm ./bld/gui/ui/osxarm ./bld/ncurses/osxarm ./bld/ndisasm/osxarm ./bld/nwlib/osxarm ./bld/orl/osxarm ./bld/owl/osxarm ./bld/plusplus/386/osxarm ./bld/plusplus/axp/osxarm ./bld/plusplus/i86/osxarm ./bld/pmake/osxarm ./bld/rc/rc/osxarm ./bld/rcsdll/osxarm ./bld/setupgui/osxarm ./bld/setupgui/ziplib/osxarm ./bld/setupgui/zlib/osxarm ./bld/ui/unix/osxarm ./bld/vi/bind/osxarm ./bld/vi/ctags/osxarm ./bld/vi/osxarm ./bld/wasm/osxarm ./bld/wcl/386/osxarm ./bld/wcl/axp/osxarm ./bld/wcl/i86/osxarm ./bld/wcl/mps/osxarm ./bld/wcl/owcc/osxarm ./bld/wcl/ppc/osxarm ./bld/wl/exe2bin/osxarm ./bld/wl/fcenable/osxarm ./bld/wl/osxarm ./bld/wl/lnk/osxarm ./bld/wl/ms2wlink/osxarm ./bld/wres/osxarm ./bld/wstrip/osxarm ./bld/wtouch/osxarm

Make file makefile content is

#pmake: build os_osx cpu_arm

host_os  = osx
host_cpu = arm

!include ../master.mif

As reference you can use Linux ARM build

[DelilahHoare]

Yes, preboot works fine from upstream without modifying it now. wmake and builder seem to be fine.

delilah@Delilahs-Mini open-watcom-v2 % wmake '-?'
Open Watcom Make Version 2.0 beta Jul  2 2022 15:43:38
Copyright (c) 2002-2022 The Open Watcom Contributors. All Rights Reserved.
Portions Copyright (c) 1988-2002 Sybase, Inc. All Rights Reserved.
Source code is available under the Sybase Open Watcom Public License.
See http://www.openwatcom.org/ for details.
Usage  :  wmake [options]* [macro=text]* [target]*
Options:  ('/' may be substituted for '-', and '#' for '=')
-a make all targs (ignoring time stamp)  -b  block/ignore all implicit rules
-c do not check existence of files made  -d  debug mode (echo progress of work)
-e erase files after error (no prompt)
-f <filename>  process filename instead of MAKEFILE ('-f -' means stdin)
-h do not print program header           -i  ignore command return status
-k continue after an error               -l <logfile>   output to logfile
-m do not read MAKEINIT file             -ms Microsoft NMAKE mode
-n print commands without executing      -o  take advantage of circular path
-p print the dependency tree, macro definitions, and extension information
-q query mode (commands not executed)    -r  ignore built-in definitions
-s silent mode (do not print commands)   -sn noisy mode (print all commands)
-t touch files (commands not executed)
-u  UNIX compatibility mode
-v verbose listing of inline files       -y  show why a target will be updated
-z do not erase files (no prompt)
macro=text      overrides the definition of 'macro' with the value 'text'
target          specifies a target to make (default is first target)
delilah@Delilahs-Mini open-watcom-v2 % builder '-?'
Usage: builder [-c <ctl>] [-l <log>] [-b <bak>] [-i] [-v] [-u] [-q] [--] <parm>
    See builder.doc for more information

I created all of those directories and added that makefile to all of them, but the build still failed on fdmth086.obj.

[DelilahHoare]

To clarify, I already had wmake running, by patching mglob.c myself. The other error logs above are from running build.sh after that.

[jmalak]

I submitted all build make files it is more variants. After source refresh cleanup build by sh clean.sh and start new build by sh build.sh boot Now real build problems must be fixed.

Anyway now in porting phase is best to setup environment variable OWVERBOSE=1 to get log with full commands to get info if all is OK or some mistakes in build make files for OSX ARM host.

[jmalak]

Don't use OWDEBUGBUILD=1 for full build, it is not supported. It is for use with simple project only or part of project build if debug is necessary.

[DelilahHoare]

./build.sh boot also finishes with no error message. running simply ./build.sh still fails in the same spot

== 16:27:20 /Users/delilah/open-watcom-v2/bld/clib/cgsupp/library/os2.286/mm ==
bwasm -q -D_BLDVER=1300 -D_CYEAR=2022 ../../../a/fdmth086.asm -I"../../../h" -I"/Users/delilah/open-watcom-v2/bld/clib/h"  -I"/Users/delilah/open-watcom-v2/bld/watcom/h" -I"/Users/delilah/open-watcom-v2/bld/lib_misc/h" -I"/Users/delilah/open-watcom-v2/bld/comp_cfg/h"  -we -fo=.obj -mm-nt=_TEXT -fpc-w8-e-2  -bt=os2  -fpi87 -d1
Abort
Error(E42): Last command making (fdmth086.obj) returned a bad status
Error(E02): Make execution terminated
<pmake -d build          -h> => non-zero return: 512
Build failed
Current usage: 0000000000001c88 bytes; Peak usage: 0000000000020337 bytes
  Who      Addr     Size     Call     Contents
======== ======== ======== ======== ===========================================
0000000000000000 0000000123009c00 00000828 00000000000000c2 0082002301000000 180f16100200...#..........
0000000000000000 0000000123008200 00000828 00000000000000bf 008e802201000000 e80d16100200..."..........
0000000000000000 0000000122808e00 00000828 0000000000000002 0000000000000000 500d16100200........P.....
0000000000000000 0000000122808800 00000410 0000000000000001 0000000000000000 2f5573657273......../Users
00000004 chunks (0000000000001c88 bytes) unfreed

[DelilahHoare]

Don't use OWDEBUGBUILD=1 for full build, it is not supported. It is for use with simple project only or part of project build if debug is necessary.

how can I do a partial build? I've tried running ./build.sh wasm and it doesn't do anything.

[jmalak]

it is design to use different way simply go to the project directory which you need to rebuild, by example for wasm to bld/wasm and run builder it rebuild this project. if you need rebuild bootstrap version then run builder boot

If you need to create bootstrap version with debug info then run builder boot OWDEBUGBUILD=1

similar for cleanup project builder clean and for bootstrap version builder bootclean

[jmalak]

There is other possibility to selective rebuild only for some achitecture or operating system. By example you wan rebuild only OSX ARM version then you can use builder build os_osx cpu_arm In this case you must specify build word as first (it is default if nothing is specified) and next parameters are used for filtering expression in our case builder traverse project tree and find makefile files and check "#pmake: ....." line and search for os_osx and cpu_arm words with AND operator and process only proper makefile files, not all All make files have #pmake: line with minimum os_... and cpu_... words.

[DelilahHoare]

Thanks for the help. I was able to recompile wasm with -fsanitize=address and track down the memory errors, which allows wasm to always complete and the build to get a lot further. See #916

However, the build still eventually fails:

clang -pipe -D_BLDVER=1300 -D_CYEAR=2022 -static  -O -g -o wasm.exe  -Wl,-map,wasm.map  main.obj write.obj fatal.obj direct.obj posndir.obj asmeval.obj asmalloc.obj asmmsg.obj asmerr.obj wasmmsg.obj macro.obj condasm.obj breakout.obj asmstruc.obj asmexpnd.obj mangle.obj for.obj asmins.obj asmscan.obj asmline.obj asmsym.obj tbyte.obj asmlabel.obj asmfixup.obj asmmatch.obj asmdata.obj asmjump.obj autodept.obj dostimet.obj queues.obj pathlist.obj omfcarve.obj omfobjre.obj omfqueue.obj omffixup.obj omflifix.obj omfgen.obj omfgenio.obj  /Users/delilah/open-watcom-v2/bld/wres/linuxarm/_s/wres.lib /Users/delilah/open-watcom-v2/bld/watcom/binbuild/clibext.lib
ld: library not found for -lcrt0.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error(E42): Last command making (wasm.exe) returned a bad status
Error(E02): Make execution terminated
<pmake -d build          -h> => non-zero return: 512
Build failed

I'm not sure why it seems to think it is on linux.

[jmalak]

I will fix it and review other "client" make file which select library to work with osx arm.

[jmalak]

It is fixed now by commits 26cc4f7dda79448fded96a2451db143a0aed0cbe f04613d9c8b89cd3ba975fa32eab8332dfa23b22

[DelilahHoare]

I pulled these changes but the outcome is the same. I should have provided more context but I didn't notice before, I think it's trying to cross-compile everything for linuxarm, as well as the targets that it can use watcom for. The bld/wres/linuxarm/_s/wres.lib file it's using exists, and contains Mach-O objects. The build in linux386 seems to have succeeded, and produced a static 386 linux binary.

=========== 20:16:16 /Users/delilah/open-watcom-v2/bld/wasm/dos386 ============
=========== 20:16:16 /Users/delilah/open-watcom-v2/bld/wasm/dosi86 ============
========== 20:16:16 /Users/delilah/open-watcom-v2/bld/wasm/linux386 ===========
========== 20:16:16 /Users/delilah/open-watcom-v2/bld/wasm/linuxarm ===========
clang -pipe -D_BLDVER=1300 -D_CYEAR=2022 -static  -O -g -o wasm.exe  -Wl,-map,wasm.map  main.obj write.obj fatal.obj direct.obj posndir.obj asmeval.obj asmalloc.obj asmmsg.obj asmerr.obj wasmmsg.obj macro.obj condasm.obj breakout.obj asmstruc.obj asmexpnd.obj mangle.obj for.obj asmins.obj asmscan.obj asmline.obj asmsym.obj tbyte.obj asmlabel.obj asmfixup.obj asmmatch.obj asmdata.obj asmjump.obj autodept.obj dostimet.obj queues.obj pathlist.obj omfcarve.obj omfobjre.obj omfqueue.obj omffixup.obj omflifix.obj omfgen.obj omfgenio.obj  /Users/delilah/open-watcom-v2/bld/wres/linuxarm/_s/wres.lib /Users/delilah/open-watcom-v2/bld/watcom/binbuild/clibext.lib
ld: library not found for -lcrt0.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error(E42): Last command making (wasm.exe) returned a bad status
Error(E02): Make execution terminated
<pmake -d build          -h> => non-zero return: 512
Build failed

[jmalak]

Oh sorry, I fixed inclusion of incorrect library only. Trying to build any arm target is wrong, linux arm target should be excluded on osx system. I will fix it.

[jmalak]

This should be fixed now by last commit

[jmalak]

commit 99e742126f45f9505b458469653a7929214a6361 correct setup of builder to properly copy native OSX OW tools to appropriate location I added arm binaries and correct location of x64 binaries If you run sh build.sh rel it build OW and copy binaries to rel sub-folder in OW run-time tree structure

[DelilahHoare]

It now fails with

clang -pipe -c -D_BLDVER=1300 -D_CYEAR=2022 -fno-asm -fno-common -fsigned-char -Wall -Wno-switch -Wno-missing-braces -Wno-parentheses -Wpragma-pack -Wno-implicit-exception-spec-mismatch -ferror-limit=200  -O -g  -o omfgenio.obj  -DNDEBUG -D_M_ARM -D__OSX_ARM__ -D__UNIX__ -D__FLAT__ -D__OSX__ -D_STANDALONE_       -I. -I"../h" -I"/Users/delilah/open-watcom-v2/bld/fe_misc/h"  -I"/Users/delilah/open-watcom-v2/bld/trmem" -I"/Users/delilah/open-watcom-v2/bld/watcom/h" ../c/omfgenio.c
In file included from ../c/omfgenio.c:39:
In file included from ../h/omfobjre.h:36:
/Users/delilah/open-watcom-v2/bld/watcom/h/pcobj.h:39:10: warning: the current #pragma pack alignment value is modified in the included file [-Wpragma-pack]
#include "pushpck1.h"
         ^
/Users/delilah/open-watcom-v2/bld/watcom/h/pushpck1.h:49:13: note: previous '#pragma pack' directive that modifies alignment is here
    #pragma pack(push,1)
            ^
In file included from ../c/omfgenio.c:39:
In file included from ../h/omfobjre.h:36:
/Users/delilah/open-watcom-v2/bld/watcom/h/pcobj.h:421:10: warning: the current #pragma pack alignment value is modified in the included file [-Wpragma-pack]
#include "poppck.h"
         ^
note: previous '#pragma pack' directive that modifies alignment is here
2 warnings generated.
Error(F38): (/Users/delilah/open-watcom-v2/bld/wres/osxarm/_s/wres.lib) does not exist and cannot be made from existing files
Error(E02): Make execution terminated
<pmake -d build          -h> => non-zero return: 1024
Build failed

[jmalak]

the commit ecb16cea9d773607e932befe76e4023f2b163f90 should fix it, source of problem was broken build of /Users/delilah/open-watcom-v2/bld/wres/osxarm/_s/wres.lib on the beginning of OW build process

[DelilahHoare]

there've been a few more build errors. I needed to make these changes to the makefiles:

--- a/bld/cc/axp/osxarm/makefile
+++ b/bld/cc/axp/osxarm/makefile
@@ -1,4 +1,4 @@
-#pmake: build os_osx cpu_arm
+#pmake: nobuild target_axp osx os_osx cpu_arm

and:

--- /dev/null
+++ b/bld/as/axp/osxarm/makefile
@@ -0,0 +1,6 @@
+#pmake: build target_axp os_osx cpu_arm
+
+host_os  = osx
+host_cpu = arm
+
+!include ../target.mif

both consistent with the osxx64 versions of the same files, which let the build get further, but it now fails here:

============= 22:02:02 /Users/delilah/open-watcom-v2/bld/commonui =============
========== 22:02:02 /Users/delilah/open-watcom-v2/bld/commonui/nt386 ==========
wcpp -D_BLDVER=1300 -D_CYEAR=2022 -zk0 -D__NT__ -D_WIN32 -D_M_IX86 -I"../h" -ouistrmsg.gh ../res/commonui.msg
awk -f /Users/delilah/open-watcom-v2/build/msgtoh.awk -v rcbase=10000 -v basename=MSG_CUI_BASE -v OUTFILE=uistr.grh uistrmsg.gh
awk: towc: multibyte conversion failure on: '?G???n???X?h???[?h" )'

 input record number 8, file uistrmsg.gh
 source line number 12
Error(E42): Last command making (uistr.grh) returned a bad status
Error(E02): Make execution terminated
<pmake -d build          -h> => non-zero return: 512
Build failed

I guess it should be using watcom's own awk, but that does not seem to have been built - which awk gives /usr/bin/awk

[jmalak]

Yes, there can be another problems due to quick duplicate of template make file, but some of them require so special setup for #pmake line. I will try to review such know project which require it. To awk, it builds awk only for non-NIX* platforms otherwise it use system awk. But it is problem with Japanese double-byte text processing. It looks like OSX version of awk process Japanese texts with incorrect encoding or you have git source file with incorrect encoding. I mostly transform Japanese texts to UTF-8 encoding, but some are still in CP932 (mainly Windows resources because 16-bit Windows requires double-byte CP932 encoding). in this case bld/commonui/res/commonui.msg is in CP932 and OSX awk has problem with it's processing.

[DelilahHoare]

I'm surprised then, seeing as the CI build on osx x64 is working. I wonder why it's not having the same problem with awk. At any rate the build completes using gawk instead of the system awk! But I just changed bhost.mif to explicitly and unconditionally use gawk, obviously that can't be upstreamed. Maybe the user could set an awk command with an OWAWK environment variable similar to OWDOSBOX?

[jmalak]

I will update commonui project to use UTF-8 source files for resources, but it is shared project by many GUI sub-projects that it take me some time to reconfigure resource compiler commands and other resource files to be UTF-8. I think it should fix this problem. Anyway I think it is undefined behavior for awk. Also difference can be in Nation Language Support setup for your system and CI system.

[jmalak]

I changed commonui/gui projects resource source files to use UTF-8 encoding and changed appropriate command line options. I hope awk should be able process UTF-8 source file without problem.

[jmalak]

I think I fixed all issues related to Japanese texts in CP932. Now all these files are converted to UTF-8 encoding and appropriate command line options for processing UTF-8 source files where setup. Please could you setup on your system user locale with UTF-8 encoding (I suppose you have it, but for sure) and check build with standard sources.

[DelilahHoare]

Yes, the build is now finishing from upstream. Thanks for all your work on this!

There's still a small problem with build.sh rel: the stub files don't end up in rel/armo, although many copies of them are placed into the bin* directories for other hosts. Trying to build hello world for DOS with wcl386 gives me Warning! W1093: cannot open wstub.exe until I get that file onto the PATH. Once I do that, wcl386 can create working executables.

[jmalak]

You need read readme in the OW run-time root where is described how you should configure OW2 manualy. At absolute minimum you must define WATCOM environment variable to point to OW run-time root and PATH to point your host version binary. For OSX you should define on the PATH $WATCOM/armo where is OSX version of OW binaries. Anyway wstub.exe path is configured in wlink.lnk but if you don't have setup WATCOM environment variable you are in trouble. As soon as you define WATCOM variable all should be working as expected. Sorry there is no path for wstub.exe only file name. you need to have it on the path. The commit 6c897bf2 add wstub.exe to binary subfolder for ARM.

[DelilahHoare]

yes, I set WATCOM to the absolute path of the rel directory and added $WATCOM/armo to the PATH. build.sh rel doesn't put the stub files in armo, so they don't end up on the PATH until I copy them there myself

% find $WATCOM -name 'wstub.exe'
/Users/delilah/open-watcom-v2/rel/binnt64/wstub.exe
/Users/delilah/open-watcom-v2/rel/binp/wstub.exe
/Users/delilah/open-watcom-v2/rel/binw/wstub.exe
/Users/delilah/open-watcom-v2/rel/binl/wstub.exe
/Users/delilah/open-watcom-v2/rel/binnt/wstub.exe
/Users/delilah/open-watcom-v2/rel/binl64/wstub.exe

[DelilahHoare]

excellent, that's fixed it. Thanks again.

[jmalak]

There can be more files missing, it is not tested anyway, if you report what is mising then I can fix it.

[DelilahHoare]

Well, if you want to know, there's a lot more files in binl than in armo, though I don't know how many of these extra files really belong in armo.

armo at the moment contains:

1250.uni    dmpobj      wasaxp      wccaxp      wcl     wlink       wppaxp01.int
1251.uni    edbind      wasm        wccaxp01.int    wcl386      wlink.lnk   wppi8601.int
1252.uni    kanji.uni   wasmps      wcci8601.int    wclaxp      wmake       wrc
936.uni     owcc        wasppc      wccmps      wclppc      wpp     wstrip
949.uni     specs.owc   wcc     wccmps01.int    wdis        wpp386      wstub.exe
950.uni     to932.uni   wcc386      wccppc      wdump       wpp38601.int    wstubq.exe
ctags       vi      wcc38601.int    wccppc01.int    wlib        wppaxp      wtouch

and binl from this same build contains:

1250.uni    cwc     ideos2.cfg  owcc        to932.uni   wcc     wd.sym      wmake.sym
1251.uni    cwdstub.exe ideos232.cfg    owcc.sym    unicode.437 wcc.sym     wdis        wpp
1252.uni    cwstub.exe  iderdos.cfg owvkeys.dbg unicode.850 wcc386      wdis.sym    wpp.sym
936.uni     default.dbg idew32.cfg  par.sym     unicode.852 wcc386.sym  wdkeys.dbg  wpp386
949.uni     dmpobj      idew386.cfg par.trp     unicode1.250    wcc38601.int    wdump       wpp386.sym
950.uni     dwarf.dip   idewin.cfg  parserv     unicode1.252    wccaxp      wfc     wpp38601.int
addsrc.dbg  dwarf.sym   idex.cfg    parserv.sym vc.dbg      wccaxp.sym  wfc.sym     wppaxp
ads.dbg     edbind      java.prs    resize.dbg  vi      wccaxp01.int    wfc386      wppaxp.sym
animate.dbg exe2bin     kanji.uni   restart.dbg vi.sym      wcci8601.int    wfc386.sym  wppaxp01.int
bcnt.dbg    export.dip  madaxp.mad  return.dbg  vikeys.dbg  wccppc      wfl     wppi8601.int
bif.dbg     export.sym  madaxp.sym  save.dbg    vr.dbg      wccppc.sym  wfl.sym     wprof
bp.dbg      fcenable    madmps.mad  setup.dbg   wasaxp      wccppc01.int    wfl386      wprof.sym
c.prs       fill.dbg    madmps.sym  slow.dbg    wasaxp.sym  wcl     wfl386.sym  wrc
cntbrks.dbg fortran.prs madppc.mad  specs.owc   wasm        wcl.sym     whc     wrc.sym
codeview.dip    fox.dbg     madppc.sym  std.sym     wasm.sym    wcl386      whelp       wsample
codeview.sym    ide.cfg     madx86.mad  std.trp     wasmps      wcl386.sym  wipfc       wsample.sym
colour.dbg  ide2make    madx86.sym  tcp.sym     wasmps.sym  wclaxp      wlib        wstrip
cpp.prs     ideaxp.cfg  mapsym.dip  tcp.trp     wasppc      wclaxp.sym  wlib.sym    wstub.exe
ctags       idedos.cfg  mapsym.sym  tcpserv     wasppc.sym  wclmps      wlink       wstubq.exe
cvkeys.dbg  idedos32.cfg    mono.dbg    tcpserv.sym watcom.dip  wclmps.sym  wlink.lnk   wtouch
cvpack      idelnx.cfg  ms2wlink    tdemu.dbg   watcom.sym  wd      wlink.sym
cw32.exe    idenlm.cfg  nwvkeys.dbg tdkeys.dbg  wbind       wd.dbg      wmake

I'll report if I encounter any more problems using it but it seems to be working.

[jmalak]

It is clear many tools were not ported to OSX, by example all GUI tools and not all architecture is build on ARM. Debugger was not ported etc. Only compilers (for Intel) and basic tools are built for command line usage.

[pchapin]

I'm going to ask what might be a silly question about this thread. The build process uses the bootstrap compiler in phase 1, but then uses the tools generated by that step to do the bulk of the build, yes? So, since Open Watcom can't generate ARM code (yes?), the tools created by the bootstrap are ARM executables, made by clang, that generate x86 code. That means the final generated programs are x86 executables that generate x86 code, and they work because of Apple's Rosetta dynamic machine code translation system.

Is that correct, or am I misunderstanding something?

[DelilahHoare]

The host Clang is used to build the final tools in the ‘armo’ directory, which are arm64 executables. Rosetta can't execute 32-bit x86 code osx applications and I don't think Watcom can target osx on any architecture.

[jmalak]

Yes, you can not build OSX executable by OW, you can only cross-compile from OSX to OW supported platforms. OW is not ported to support OSX, it can be used only as host not target.

[pchapin]

Okay, thanks, I understand. Let me see if I have this correct: On the macOS host, the clang-built tools are ARM executables, but they can only target other platforms (there being no macOS target of any kind regardless of processor type). The tools built for other hosts are, naturally, executables for those hosts and won't run on macOS even if the processor type is compatible (i.e., x86) because, of course, they are trying to interact with a non-macOS operating system.

Let me know if that isn't correct. Thanks again!

[jmalak]

Yes exactly, to create executable for OSX we need C run-time library and linker which is able to create OSX executable. Other thing is code generator for OSX architecture, we have only 32-bit Intel. We could be able to create 32-bit OSX executable for Intel, but I am not sure if it has sense. Anyway C run-time library should be ported to OSX for such purpose.

Our OW2 build system is POSIX compatible that it can run on any such system with system native compiler. Therefore we are able create system native OW tools running on this system but we can not create executables for such system.

It can be used only for cross-compilation for standard OW supported targets (16/32-bit and limited set of OS-es). It is sometimes used by user for DOS development on such systems, by example OW2 on Rapsbery PI etc.

[pchapin]

The ARM based Macs have a tool called Rosetta that dynamically translates x86_64 opcodes to ARM opcodes, allowing the system to run x86_64 OSX programs compiled for earlier Macs. I don't know if Rosetta can cope with 32-bit Intel opcodes. In any case, I understand that a proper runtime library is needed as well.

What is the status of the 64-bit Intel code generator and the x86_64 Linux target? Since OSX is a flavor of Unix, it might not be too difficult tweaking a 64-bit, Intel Linux target to work as a 64-bit, Intel OSX target. The resulting executables would run on the ARM based Macs via Rosetta, even without an ARM code generator in Open Watcom.

[XVilka]

It would be nice to add Open-Watcom in Homebrew and/or MacPorts.