Wiimm / wiimms-iso-tools

»Wiimms ISO Tools« is a set of command line tools to extract, modify and create Wii and GameCube ISO images and WBFS containers.
https://wit.wiimm.de
115 stars 15 forks source link

v3.04a universal binaries not working under silicon Macos #18

Open darth-hp opened 2 years ago

darth-hp commented 2 years ago

Hi,

I downloaded wit-v3.04a-r8427-mac.tar.gz and run the installer. I do see all files are universal binaries but unfortunately the system complaints that the files are broken and can't be run.

I then downloaded wit-v3.03a-r8245-mac.tar.gz as x86_64 and this works.

file shows arm64 as architecture for the v3.04a binaries where other universal binaries I have show arm64e.

Wiimm commented 2 years ago

This will change with the next release.

impierced commented 1 year ago

Looks like wit-v3.05a still doesn't work for me on Apple Silicon. The issue appears to be with the arm64 binary as the x86_64 binary works fine.

me@system bin % ./wit zsh: killed ./wit

me@system bin % lipo -archs wit x86_64 arm64 me@system bin % lipo -remove arm64 wit -output wit_x86_64 me@system bin % ./wit_x86_64

    *****************************************
    *    __            __ _ ___________     *
    *    \ \          / /| |____   ____|    *
    *     \ \        / / | |    | |         *
    *      \ \  /\  / /  | |    | |         *
    *       \ \/  \/ /   | |    | |         *
    *        \  /\  /    | |    | |         *
    *         \/  \/     |_|    |_|         *
    *                                       *
    *           Wiimms ISO Tools            *
    *         https://wit.wiimm.de/         *
    *                                       *
    *****************************************

wit: Wiimms ISO Tool v3.05a r8638 mac/x64 - Dirk Clemens - 2022-08-27 Visit https://wit.wiimm.de/wit for more info.

-> Type 'wit_x86 -h' or 'wit_x86 help' (pipe it to a pager like 'less') for more help.

impierced commented 1 year ago

I simply modified the Makefile (quick and very dirty) from the source:

ifeq ($(SYSTEM),mac) CFLAGS += -I/usr/local/include CFLAGS += -arch arm64 -arch x86_64 else LDFLAGS += -static-libgcc endif

Unsure why, but I had to "export SYSTEM=mac", but I was able to then use "make" to create a universal binary of v3.05a that worked.

Wiimm commented 1 year ago

My problem is, that I haven't any access to a Mac and github doesn't support arm64. But I need github to test a compiler run for Mac. A friend of mine creates the universal binary at home. Because of this situation, the makefile doesn't support arm64.

Your told "not working": Do you mean the binaries created by my friend, or self compiled binaries?

impierced commented 1 year ago

The binaries created by your friend are not working. When I modified the Makefile and compiled it myself, those work just fine.

Happy to upload/send you the universal binaries that I compile.

Wiimm commented 1 year ago

Can you please do a test for me?

  1. Enter the wit project directory (place, where system.c exists: ./project) and execute gcc -E -DPRINT_SYSTEM_SETTINGS system.c. It won't compile anything, but do some preprocessor stuff.
  2. Load http://download.wiimm.de/misc/2022-10/mac-test.zip and unzip. Enter directory mac-test and execute same command.
  3. Finally please post both outputs.
darth-hp commented 1 year ago
➜  project git:(master) gcc -E -DPRINT_SYSTEM_SETTINGS system.c
# 1 "system.c"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 400 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "system.c" 2
# 37 "system.c"
# 1 "./dclib/dclib-system.h" 1
# 136 "./dclib/dclib-system.h"
result_SYSTEM="mac"
result_SYSTEM2="mac/arm"
result_SYSTEM_LINUX=0
# 38 "system.c" 2
➜  mac-test git:(master) ✗ gcc -E -DPRINT_SYSTEM_SETTINGS system.c
# 1 "system.c"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 400 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "system.c" 2
# 37 "system.c"
# 1 "./dclib/dclib-system.h" 1
# 141 "./dclib/dclib-system.h"
result_SYSTEM="mac"
result_SYSTEM2="mac/arm"
result_SYSTEM_LINUX=0
# 38 "system.c" 2
Wiimm commented 1 year ago

Then I don't understand why you have to export SYSTEM=mac. The old self-check already defines it by result_SYSTEM="mac" (result_ is removed by script). So my update won't help. :(

Anyway, thanks for testing.

Leseratte10 commented 1 year ago

I have compiled most of the previous WIT and WSZST tools Mac releases using a nonstandard Mac cross-compiler on Linux, because neither Wiimm nor me have proper access to a Mac machine.

The only difference between arm64 and arm64e is that the latter uses / requires pointer authentication, so that's most likely not going to be the issue.

Is there nothing more useful in the system logs or some other apple error logs? Just "Killed" in the terminal? Maybe this is the typical Mac error for unsigned binaries? Does this help?

If the Intel version is working just fine, maybe we should just drop the ARM version then and rely on Rosetta 2 on the ARM macs to run Wiimms tools; unless someone with more Mac knowledge figures out what exactly is wrong with the binaries.

impierced commented 1 year ago

The issue seems to be that the SYSTEM variable isn't getting written to the Makefile.setup file.

The awk function gensub doesn't appear to be available in macOS, "awk: calling undefined function gensub".

From setup.sh: gcc $xflags -E -DPRINT_SYSTEMSETTINGS system.c \ | awk -F= '/^result/ {printf("%s := %s\n",substr($1,8),gensub(/"/,"","g",$2))}' \

Makefile.setup

impierced commented 1 year ago

Leseratte10, I can compile an ARM version (or universal binary) and it works just fine.

Edit: I see you posted more information about the compiler being used. I'm compiling natively on macOS with Xcode. This explains why the function gensub is working for you and not for me.

M1 binaries shouldn't require a recompile for M2 chips, I was throwing whatever I could think of out there.

Leseratte10 commented 1 year ago

The compiler we're using is apple-darwin20.1-clang with MACOSX_DEPLOYMENT_TARGET=10.10 and the SDK version 11.0; using osxcross from June.

Do we need a newer SDK for M2 Macs? I don't know that much about MacOS but I assumed that M2 devices should be able to run binaries from an M1 Mac?

impierced commented 1 year ago

I just downloaded the 3.05a universal binary. I'm getting a different error than I did with the 3.04a universal binary (that appears to have been removed).

% ./wit macOS pops up a dialog box with: "'wit' is damaged and can’t be opened. You should move it to the Trash."

Using "spctl --add wit" didn't have any effect on the binary.

% lipo -archs wit x86_64 arm64

Extracting out the x86_64 binary worked, so it's still an issue with the arm64 binary.

Leseratte10 commented 1 year ago

I don't know much about MacOS so maybe that's what you already tried, but what happens when you open the folder in Finder, then Ctrl+click on the binary and then select "Open" from the context menu? Does that give you an option to override that, and if so, does that open the ARM binary?

We have one compiler call with CFLAGS="-arch x86_64 -arch arm64 so it should be using the same settings for both architectures ...

Wiimm commented 1 year ago

The SYSTEM=mac bug is solved in my working copy. I replaced line 73 of file setup.sh from | awk -F= '/^result_/ {printf("%s := %s\n",substr($1,8),gensub(/"/,"","g",$2))}' \ to | awk -F= '/^result_/ { gsub(/"/,"",$2); printf("%s := %s\n",substr($1,8),$2) }' \

impierced commented 1 year ago

"...what happens when you open the folder in Finder, then Ctrl+click on the binary and then select "Open" from the context menu? Does that give you an option to override that, and if so, does that open the ARM binary?"

I'm prompted to allow the app to run ... I authenticate as an admin ... and it fails like the 3.04a universal binary did.

impierced commented 1 year ago

The SYSTEM=mac bug is solved in my working copy. I replaced line 73 of file setup.sh from | awk -F= '/^result_/ {printf("%s := %s\n",substr($1,8),gensub(/"/,"","g",$2))}' \ to | awk -F= '/^result_/ { gsub(/"/,"",$2); printf("%s := %s\n",substr($1,8),$2) }' \

That worked for me as well.

impierced commented 1 year ago

Leseratte10, can you try compiling just an arm64 version for me to test?

Leseratte10 commented 1 year ago

Here you go: wit_1_normal is another universal binary compiled just like 3.05a (just in case something went wrong the last time I compiled it), wit_2_arm_only is a standalone arm64 binary.

Same compiler settings / versions / SDK as before. wit_mac_test.zip

EDIT: Also, would you mind testing if the latest release of Wiimms SZS tools works on your Mac? https://szs.wiimm.de/download.html That's compiled with the same cross-compiler but a bunch of different settings, so knowing if this works or if it fails as well could help track this down.

impierced commented 1 year ago

Unfortunately, they all failed.

wit_1_normal: unsigned developer, was able to authenticate to allow it to run, failed to launch. wit_2_arm_only: corrupt, couldn't even authenticate to allow it to run.

Wimms SZS tools: same as wit_1_normal (latest x86_64, arm64 build).

breakerbar commented 1 year ago

I have an Apple silicon machine and am experiencing this issue (it manifests as "Killed: 9") and would like to help if I can.

Output (M1 MacBook Pro):

$ ./wit_1_normal
Killed: 9
$ ./wit_2_arm_only
Killed: 9
$ file *
wit_1_normal:   Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]
wit_1_normal (for architecture x86_64): Mach-O 64-bit executable x86_64
wit_1_normal (for architecture arm64):  Mach-O 64-bit executable arm64
wit_2_arm_only: Mach-O 64-bit executable arm64
$ xattr -l *
wit_1_normal: com.apple.quarantine: 0281;6452ef7d;Firefox;FFAB83C2-B76C-4B1E-BB3B-1859E84F4989
wit_2_arm_only: com.apple.quarantine: 0281;6452ef7d;Firefox;FFAB83C2-B76C-4B1E-BB3B-1859E84F4989
$ xattr -d com.apple.quarantine *
$ xattr -l *
wit_1_normal: com.apple.provenance:
wit_2_arm_only: com.apple.provenance:
$ ./wit_1_normal
Killed: 9
$ ./wit_2_arm_only
Killed: 9
breakerbar commented 1 year ago

And for the szs tools (ver szs-v2.33a-r8773-mac64)

$ file bin/wbmgt
bin/wbmgt: Mach-O universal binary with 1 architecture: [x86_64:Mach-O 64-bit executable x86_64]
bin/wbmgt (for architecture x86_64):    Mach-O 64-bit executable x86_64
$ ./bin/wbmgt --help
Killed: 9
$ xattr -d com.apple.quarantine bin/wbmgt
$ ./bin/wbmgt --help
wbmgt: Wiimms BMG Tool v2.33a r8773 mac/x64 - Dirk Clemens - 2023-01-14

wbmgt : Wiimms BMG Tool : Convert and patch raw and text BMG (message) files.

Syntax: wbmgt [option]... command [option|parameter|file]...

Commands:

  VERSION    | V   : Print program name and version and exit.
  HELP       | H   : Print help for commands and options.
  CONFIG           : Show all information about the search for the
<snip>
clunkclunk commented 1 year ago

The SYSTEM=mac bug is solved in my working copy. I replaced line 73 of file setup.sh from | awk -F= '/^result_/ {printf("%s := %s\n",substr($1,8),gensub(/"/,"","g",$2))}' \ to | awk -F= '/^result_/ { gsub(/"/,"",$2); printf("%s := %s\n",substr($1,8),$2) }' \

That worked for me as well.

For me as well.

flanter21 commented 6 months ago

I can compile the binaries however, i cannot link them. Anyone got any ideas? EDIT: Doesn't link when I use gcc either.

ld: warning: alignment (1) of atom '_sizeof_info_dol' from '/Users/me/Downloads/wiimms-iso-tools.source-3.05a/lib-dol.o' is too small and may result in unaligned pointers 
ld: building fixups: pointer not aligned at _SRT_UsageCount+0x1C from /Users/me/Downloads/wiimms-iso-tools.source-3.05a/dclib-numeric.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [gen-ui] Error 1
evillogic commented 4 months ago

I had a similar issue where I was getting Killed: 9 on my m3 macbook.

***    load ./share/titles.txt from http://gametdb.com/titles.txt
./load-titles.sh: line 80:  9324 Exit 3                  webget "$URI"
      9325 Killed: 9               | wit titles / - > "$DEST.tmp"
... more of the above ...
***    load ./share/titles-zhtw.txt from http://gametdb.com/titles.txt?LANG=ZHTW
./load-titles.sh: line 80:  9412 Exit 3                  webget "$URI"
      9413 Killed: 9               | wit titles / - > "$DEST.tmp"
*** install titles to /usr/local/share/wit

I solved this by running the following line after install.sh:

sudo codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime /usr/local/bin/wit
sudo ./load-titles.sh

It seems the latest release (v3.05a) isn't signed correctly, then load_titles fails to import titles because of this.