search

libretro / mupen64plus-libretro-nx

Improved mupen64plus libretro core reimplementation
GNU General Public License v2.0
225 stars 111 forks source link

gen_asm_defines.awk not compatible with gcc 13 #543

Closed aktau closed 1 month ago

aktau commented 1 month ago

I encountered the following build error:

strings "mupen64plus-core/src/asm_defines/asm_defines.o" | tr -d '\r' | awk -v dest_dir="./mupen64plus-core/src/asm_defines" -f ./mupen64plus-core/tools/gen_asm_defines.awk
nasm -i./mupen64plus-core/src/asm_defines/ -f elf64 -d ELF_TYPE mupen64plus-core/src/device/r4300/new_dynarec/x64/linkage_x64.asm -o mupen64plus-core/src/device/r4300/new_dynarec/x64/linkage_x64.o
mupen64plus-core/src/device/r4300/new_dynarec/x64/linkage_x64.asm:19: error: unable to open include file `asm_defines_nasm.h': No such file or directory
gmake: *** [Makefile:674: mupen64plus-core/src/device/r4300/new_dynarec/x64/linkage_x64.o] Error 1

I manually ran the above command-line and saw that there was no output.

Input (first stage of the pipeline):

$ strings "mupen64plus-core/src/asm_defines/asm_defines.o" | tr -d '\r' | head -20
@ASM_DEFINE offsetof_struct_device_r4300 0x
031017e8.
@ASM_DEFINE offsetof_struct_r4300_core_cp0 0x(
00000278-
@ASM_DEFINE offsetof_struct_cp0_last_addr 0x(
0000027c0
@ASM_DEFINE offsetof_struct_cp0_count_per_op 0x(
00000284'
@ASM_DEFINE offsetof_struct_cp0_tlb 0x(
@ASM_DEFINE offsetof_struct_tlb_entries 0x
00000680)
@ASM_DEFINE offsetof_struct_tlb_LUT_r 0x(
00400680)
@ASM_DEFINE offsetof_struct_tlb_LUT_w 0x(
000000f08
@ASM_DEFINE offsetof_struct_r4300_core_cached_interp 0x(
@ASM_DEFINE offsetof_struct_cached_interp_invalid_code 0x
02901000@
@ASM_DEFINE offsetof_struct_r4300_core_new_dynarec_hot_state 0x(
009010007

Script:

BEGIN {
  nasm_file = dest_dir"/asm_defines_nasm.h";
  gas_file =  dest_dir"/asm_defines_gas.h";
}

/@ASM_DEFINE offsetof_struct_[a-zA-Z_0-9]+ 0x[0-9a-fA-F]+/ {
    print "%define "$2" ("$3")" > nasm_file;
    print "#define "$2" ("$3")" > gas_file;
}

As one can see, the new format is either 0x( or 0x without any following characters. Neither is matched by the awk regex. I'm not sure what the old format looked like, so I'm not sure how to alter the script.

The gcc version this object file was compiled with:

$ gcc --version
gcc (Debian 13.2.0-25) 13.2.0
Jj0YzL5nvJ commented 1 month ago

What language is your OS or build environment using?

aktau commented 1 month ago

What language is your OS or build environment using?

I'm running Debian Testing (x86-64), and:

$ echo $LANG
en_US.UTF-8

The first solution sadly doesn't work:

$ ( cd libretro-mupen64plus_next && git clean -d -x -f -q )
$ (
  COMPFLAGS="-march=native -mtune=native -ftree-vectorize -funsafe-math-optimizations -fno-strict-aliasing -flto"
  COMLDFLAGS="-flto"
  export LDFLAGS="$COMLDFLAGS"
  export CFLAGS="$COMPFLAGS"
  export CPPFLAGS="$COMPFLAGS"
  export CXXFLAGS="$COMPFLAGS"
  export LC_ALL=C
  ./libretro-build.sh mupen64plus_next
)
...
strings "mupen64plus-core/src/asm_defines/asm_defines.o" | tr -d '\r' | awk -v dest_dir="./mupen64plus-core/src/asm_defines" -f ./mupen64plus-core/tools/gen_asm_defines.awk
nasm -i./mupen64plus-core/src/asm_defines/ -f elf64 -d ELF_TYPE mupen64plus-core/src/device/r4300/new_dynarec/x64/linkage_x64.asm -o mupen64plus-core/src/device/r4300/new_dynarec/x64/linkage_x64.o
mupen64plus-core/src/device/r4300/new_dynarec/x64/linkage_x64.asm:19: error: unable to open include file `asm_defines_nasm.h': No such file or directory
gmake: *** [Makefile:674: mupen64plus-core/src/device/r4300/new_dynarec/x64/linkage_x64.o] Error 1
gmake: *** Waiting for unfinished jobs....
cp "mupen64plus_next_libretro.so" "/home/aktau/github/libretro-super/dist/unix/mupen64plus_next_libretro.so"
cp: cannot stat 'mupen64plus_next_libretro.so': No such file or directory

The second solution appears to give an error too:

$ bash ./mupen64plus-core/tools/gen_asm_script.sh "./mupen64plus-core/src/asm_defines" "mupen64plus-core/src/asm_defines/asm_defines.o"
./mupen64plus-core/tools/gen_asm_script.sh: line 22: warning: command substitution: ignored null byte in input

I checked the command substitution, and indeed it contains NUL bytes:

$ LC_ALL=C grep -a "@ASM_DEFINE" "mupen64plus-core/src/asm_defines/asm_defines.o" | sort | cut -d ' ' -f2,3 | xxd | head -10
00000000: 6f66 6673 6574 6f66 5f73 7472 7563 745f  offsetof_struct_
00000010: 6361 6368 6564 5f69 6e74 6572 705f 696e  cached_interp_in
00000020: 7661 6c69 645f 636f 6465 2030 7802 0007  valid_code 0x...
00000030: b90c 000e 28b5 2ffd 0a6f 6666 7365 746f  ....(./..offseto
00000040: 665f 7374 7275 6374 5f63 7030 5f63 6f75  f_struct_cp0_cou
00000050: 6e74 5f70 6572 5f6f 7020 3078 28b5 2ffd  nt_per_op 0x(./.
00000060: 0a6f 6666 7365 746f 665f 7374 7275 6374  .offsetof_struct
00000070: 5f63 7030 5f6c 6173 745f 6164 6472 2030  _cp0_last_addr 0
00000080: 7828 b52f fd0a 6f66 6673 6574 6f66 5f73  x(./..offsetof_s
00000090: 7472 7563 745f 6370 305f 746c 6220 3078  truct_cp0_tlb 0x
...
aktau commented 1 month ago

On my system the output of the first part of the pipeline is like this:

$ strings "mupen64plus-core/src/asm_defines/asm_defines.o" | tr -d '\r' | head -10
@ASM_DEFINE offsetof_struct_device_r4300 0x
031017e8.
@ASM_DEFINE offsetof_struct_r4300_core_cp0 0x(
00000278-
@ASM_DEFINE offsetof_struct_cp0_last_addr 0x(
0000027c0
@ASM_DEFINE offsetof_struct_cp0_count_per_op 0x(
00000284'
@ASM_DEFINE offsetof_struct_cp0_tlb 0x(
@ASM_DEFINE offsetof_struct_tlb_entries 0x

I suspect that when the AWK (and other) scripts were developed, the offset was still on the same line. Here's the hex (without tr), confirming that it's a newline (0x0A):

$ strings "mupen64plus-core/src/asm_defines/asm_defines.o" | xxd | head -10
00000000: 4041 534d 5f44 4546 494e 4520 6f66 6673  @ASM_DEFINE offs
00000010: 6574 6f66 5f73 7472 7563 745f 6465 7669  etof_struct_devi
00000020: 6365 5f72 3433 3030 2030 780a 3033 3130  ce_r4300 0x.0310
00000030: 3137 6538 2e0a 4041 534d 5f44 4546 494e  17e8..@ASM_DEFIN
00000040: 4520 6f66 6673 6574 6f66 5f73 7472 7563  E offsetof_struc
00000050: 745f 7234 3330 305f 636f 7265 5f63 7030  t_r4300_core_cp0
00000060: 2030 7828 0a30 3030 3030 3237 382d 0a40   0x(.00000278-.@
00000070: 4153 4d5f 4445 4649 4e45 206f 6666 7365  ASM_DEFINE offse
00000080: 746f 665f 7374 7275 6374 5f63 7030 5f6c  tof_struct_cp0_l
00000090: 6173 745f 6164 6472 2030 7828 0a30 3030  ast_addr 0x(.000
000000a0: 3030 3237 6330 0a40 4153 4d5f 4445 4649  0027c0.@ASM_DEFI
000000b0: 4e45 206f 6666 7365 746f 665f 7374 7275  NE offsetof_stru
000000c0: 6374 5f63 7030 5f63 6f75 6e74 5f70 6572  ct_cp0_count_per
000000d0: 5f6f 7020 3078 280a 3030 3030 3032 3834  _op 0x(.00000284
000000e0: 270a 4041 534d 5f44 4546 494e 4520 6f66  '.@ASM_DEFINE of
000000f0: 6673 6574 6f66 5f73 7472 7563 745f 6370  fsetof_struct_cp
00000100: 305f 746c 6220 3078 280a 4041 534d 5f44  0_tlb 0x(.@ASM_D
00000110: 4546 494e 4520 6f66 6673 6574 6f66 5f73  EFINE offsetof_s
00000120: 7472 7563 745f 746c 625f 656e 7472 6965  truct_tlb_entrie
00000130: 7320 3078 0a30 3030 3030 3638 3029 0a40  s 0x.00000680).@
aktau commented 1 month ago

Looking at what asm_defines.c does:

https://github.com/libretro/mupen64plus-libretro-nx/blob/80e48a0f6e3e6b4a295d1e96ba010b3223b96c78/mupen64plus-core/src/asm_defines/asm_defines.c#L48-L68

And comparing that with the output in hex (see above), there's some off things going on. We'd expect:

before = "\n@ASM_DEFINE offsetof_struct_r4300_core_cp0 0x"
hexval = "DEADBEEF"
after = '\n'
ensure_32bit = '\0'

What we see:

before_hex = 0a 4041 534d 5f44 4546 494e 4520 6f66 6673 6574 6f66 5f73 7472 7563 745f 7234 3330 305f 636f 7265 5f63 7030 2030 78
char[.]   before = "\n@ASM_DEFINE offsetof_struct_r4300_core_cp0  0x"
char[8]   hexval = "(.00000278-" // corrupt (11 bytes?!)
char      after = '\n'
char[1]   ensure_32bit = "@'" // corrupt

The problem is I can't reproduce this with a reduced example:

// defines.c
#include <stdio.h>
#include <stddef.h>

#define HEX(n) ((n) >= 10 ? ('a' + ((n) - 10)) : ('0' + (n)))

/* Creates a structure whose bytes form a string like
 * "\n@ASM_DEFINE offsetof_blah_blah 0xdeadbeef\n"
 *
 * This should appear somewhere in the object file, and is distinctive enough
 * that it shouldn't appear by chance.  Thus we can pipe the object file
 * directly to awk, and extract the values without having to use
 * platform-specific tools (e.g. objdump, dumpbin, nm).
 */
#define _DEFINE(str, sym, val) \
    const struct { \
        char before[sizeof(str)-1]; \
        char hexval[8]; \
        char after; \
        char ensure_32bit[(val) > 0xffffffffUL ? -1 : 1]; \
    } sym = { \
        str, \
        { \
            HEX(((val) >> 28) & 0xf), \
            HEX(((val) >> 24) & 0xf), \
            HEX(((val) >> 20) & 0xf), \
            HEX(((val) >> 16) & 0xf), \
            HEX(((val) >> 12) & 0xf), \
            HEX(((val) >>  8) & 0xf), \
            HEX(((val) >>  4) & 0xf), \
            HEX(((val) >>  0) & 0xf) \
        }, \
        '\n', \
        {0} \
    }

/* Export member m of structure s.
 * Suitable parsing of corresponding object file (with strings) can be used to
 * generate header suitable for inclusion in assembly files.
 */
#define DEFINE(s, m) \
    _DEFINE("\n@ASM_DEFINE offsetof_struct_" #s "_" #m " 0x", \
            __offsetof_struct_##s##_##m, \
            offsetof(struct s, m))

struct r4300_core {
  int i;
  int j;
  int k;
  int l;
  int cp0;
};

/* Structure members definitions */
DEFINE(r4300_core, cp0);

Which gives:

$ gcc defines.c -c -o defines.o && strings defines.o | xxd
00000000: 4041 534d 5f44 4546 494e 4520 6f66 6673  @ASM_DEFINE offs
00000010: 6574 6f66 5f73 7472 7563 745f 7234 3330  etof_struct_r430
00000020: 305f 636f 7265 5f63 7030 2030 7830 3030  0_core_cp0 0x000
00000030: 3030 3031 300a 4743 433a 2028 4465 6269  00010.GCC: (Debi
00000040: 616e 2031 332e 332e 302d 3129 2031 332e  an 13.3.0-1) 13.
00000050: 332e 300a 6465 6669 6e65 732e 630a 5f5f  3.0.defines.c.__
00000060: 6f66 6673 6574 6f66 5f73 7472 7563 745f  offsetof_struct_
00000070: 7234 3330 305f 636f 7265 5f63 7030 0a2e  r4300_core_cp0..
00000080: 7379 6d74 6162 0a2e 7374 7274 6162 0a2e  symtab..strtab..
00000090: 7368 7374 7274 6162 0a2e 7465 7874 0a2e  shstrtab..text..
000000a0: 6461 7461 0a2e 6273 730a 2e72 6f64 6174  data..bss..rodat
000000b0: 610a 2e63 6f6d 6d65 6e74 0a2e 6e6f 7465  a..comment..note
000000c0: 2e47 4e55 2d73 7461 636b 0a              .GNU-stack.

Which doesn't look corrupted. I tried to apply all of the flag from the original file, but it makes no difference:

$ gcc -march=native -mtune=native -ftree-vectorize -funsafe-math-optimizations -fno-strict-aliasing -flto -march=native -mtune=native -ftree-vectorize -funsafe-math-optimizations -fno-strict-aliasing -flto -std=gnu11 -D_CRT_SECURE_NO_WARNINGS -Wno-discarded-qualifiers -O3 -DNDEBUG -fsigned-char -ffast-math -fno-strict-aliasing -fomit-frame-pointer -fvisibility=hidden -fcommon -DGIT_VERSION=\"" 147dc7e"\" -DOS_LINUX -I./mupen64plus-core/subprojects/md5 -DARCH_MIN_SSE2 -msse -msse2 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -D__LIBRETRO__ -DUSE_FILE32API -DM64P_PLUGIN_API -DM64P_CORE_PROTOTYPES -D_ENDUSER_RELEASE -DSINC_LOWER_QUALITY -DTXFILTER_LIB -D__VEC4_OPT -DMUPENPLUSAPI -I./custom -I./custom/mupen64plus-core -I./custom/android/include -I./custom/GLideN64 -I./GLideN64/src -I./GLideN64/src/osal -I./mupen64plus-core/src -I./mupen64plus-core/src/api -I./custom/mupen64plus-core/plugin/audio_libretro -I./libretro-common/include -I./libretro -I./GLideN64/src/inc -I./custom/dependencies/libpng -I./mupen64plus-core/subprojects/minizip -I./xxHash -I./custom/dependencies/libzlib  -fPIC  -DCORE -DHAVE_OPENGL -DNEW_DYNAREC=2 -DDYNAREC -I./mupen64plus-core/src/asm_defines/ -c mupen64plus-core/src/asm_defines/asm_defines.c -o mupen64plus-core/src/asm_defines/asm_defines.o

I keep thinking that some optimization flag is causing the struct fields to be rejiggered (padding?).

EDIT: I tried adding

-    } sym = { \
+    } __attribute__((packed)) sym = { \

But it doesn't have any effect.

Jj0YzL5nvJ commented 1 month ago

Try adding:

# It is important to disable LTO for this object file
# otherwise we can't extract usefull information from it.
$(ASM_DEFINES_OBJ): $(AWK_DEST_DIR)/asm_defines.c
    $(CC) $(CPPFLAGS) $(CFLAGS) -fno-lto -c $< -o $@

Before: https://github.com/libretro/mupen64plus-libretro-nx/blob/develop/Makefile#L668

aktau commented 1 month ago

It was definitely LTO:

$ gcc defines.c -c -o defines.o && strings defines.o | xxd
00000000: 4041 534d 5f44 4546 494e 4520 6f66 6673  @ASM_DEFINE offs
00000010: 6574 6f66 5f73 7472 7563 745f 7234 3330  etof_struct_r430
00000020: 305f 636f 7265 5f63 7030 2030 7830 3030  0_core_cp0 0x000
00000030: 3030 3031 300a 4743 433a 2028 4465 6269  00010.GCC: (Debi
00000040: 616e 2031 332e 332e 302d 3129 2031 332e  an 13.3.0-1) 13.
00000050: 332e 300a 6465 6669 6e65 732e 630a 5f5f  3.0.defines.c.__
00000060: 6f66 6673 6574 6f66 5f73 7472 7563 745f  offsetof_struct_
00000070: 7234 3330 305f 636f 7265 5f63 7030 0a2e  r4300_core_cp0..
00000080: 7379 6d74 6162 0a2e 7374 7274 6162 0a2e  symtab..strtab..
00000090: 7368 7374 7274 6162 0a2e 7465 7874 0a2e  shstrtab..text..
000000a0: 6461 7461 0a2e 6273 730a 2e72 6f64 6174  data..bss..rodat
000000b0: 610a 2e63 6f6d 6d65 6e74 0a2e 6e6f 7465  a..comment..note
000000c0: 2e47 4e55 2d73 7461 636b 0a              .GNU-stack.

$ gcc defines.c -flto -c -o defines.o && strings defines.o | xxd
00000000: 3030 3030 3030 3130 2e0a 4041 534d 5f44  00000010..@ASM_D
00000010: 4546 494e 4520 6f66 6673 6574 6f66 5f73  EFINE offsetof_s
00000020: 7472 7563 745f 7234 3330 305f 636f 7265  truct_r4300_core
00000030: 5f63 7030 2030 7828 0a48 4144 3750 0a2d  _cp0 0x(.HAD7P.-
00000040: 784f 490a 6a48 3d5c 0a5c 452d 3d0a 2755  xOI.jH=\.\E-=.'U
00000050: 7e74 0a54 243c 2976 0a3c 7b57 510a 6434  ~t.T$<)v.<{WQ.d4
00000060: 322b 2c60 0a79 224b 3228 0a40 5e58 4c0a  2+,`.y"K2(.@^XL.
00000070: 5f5f 6f66 6673 6574 6f66 5f73 7472 7563  __offsetof_struc
00000080: 745f 7234 3330 305f 636f 7265 5f63 7030  t_r4300_core_cp0
00000090: 0a27 2d66 6e6f 2d6f 7065 6e6d 7027 2027  .'-fno-openmp' '
000000a0: 2d66 6e6f 2d6f 7065 6e61 6363 2720 272d  -fno-openacc' '-
000000b0: 6650 4943 2720 272d 6663 662d 7072 6f74  fPIC' '-fcf-prot
000000c0: 6563 7469 6f6e 3d6e 6f6e 6527 2027 2d6d  ection=none' '-m
000000d0: 7475 6e65 3d67 656e 6572 6963 2720 272d  tune=generic' '-
000000e0: 6d61 7263 683d 7838 362d 3634 2720 272d  march=x86-64' '-

Your edit works. Thanks a lot. Could this be added to the main repo or should I just remove -flto from CFLAGS for mupen?

aktau commented 1 month ago

...I cried victory too soon. The process goes all the way to the end (final link), but fails near the finish line:

lto-wrapper: warning: using serial compilation of 46 LTRANS jobs
lto-wrapper: note: see the '-flto' option documentation for more information
/usr/bin/ld: /tmp/ccCFMXAN.ltrans0.ltrans.o: warning: relocation against `co_active_handle' in read-only section `.text'
/usr/bin/ld: /tmp/ccCFMXAN.ltrans0.ltrans.o: in function `co_switch':
<artificial>:(.text+0x3): undefined reference to `co_active_handle'
/usr/bin/ld: /tmp/ccCFMXAN.ltrans0.ltrans.o: relocation R_X86_64_PC32 against undefined symbol `co_active_handle' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
gmake: *** [Makefile:665: mupen64plus_next_libretro.so] Error 1
cp "mupen64plus_next_libretro.so" "/home/aktau/github/libretro-super/dist/unix/mupen64plus_next_libretro.so"
cp: cannot stat 'mupen64plus_next_libretro.so': No such file or directory
1 core(s) failed:
   mupen64plus_next
Jj0YzL5nvJ commented 1 month ago

At this point, I have no idea how this core has even worked... (I don't use RA)

Try adding:

ifeq ($(PIC), 1)
   fpic = -fPIC
   ASFLAGS += -d PIC
else
   fpic = -fno-PIC
endif

Here: https://github.com/libretro/mupen64plus-libretro-nx/blob/develop/Makefile#L638

aktau commented 1 month ago

At this point, I have no idea how this core has even worked... (I don't use RA)

Out of curiosity? What frontend are you using then? Or rather, how are you using the core?

I will try out your proposed fix once I get home again (currently away), but I do think it's strange that the compilation works fine without -flto, but gives the PIC error with -flto. I still need to verify whether the core (without -flto) actually even works. (I tried about a year ago, so it's a bit out of date.)

Jj0YzL5nvJ commented 1 month ago

Out of curiosity? What frontend are you using then? Or rather, how are you using the core?

I don't use libretro, I use mupen64plus-ui-console with the upstream core and my own scripts:

https://imgur.com/a/WAjdL1B https://imgur.com/a/zli3e1Z https://imgur.com/a/M9kOJSA

I do think it's strange that the compilation works fine without -flto, but gives the PIC error with -flto.

Hmm... then try removing asm_defines.c from Makefile.common... https://github.com/libretro/mupen64plus-libretro-nx/blob/develop/Makefile.common#L38

aktau commented 1 month ago

I can confirm that the .so produced without -flto works.

I can also confirm that libco.o, which contains the co_active_handle it complains about:

<artificial>:(.text+0x3): undefined reference to `co_active_handle'
/usr/bin/ld: /tmp/ccxb1mXW.ltrans0.ltrans.o: relocation R_X86_64_PC32 against undefined symbol `co_active_handle' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad valu

...was actually compiled with -fPIC (without changing the makefile):

gcc -march=native -mtune=native -ftree-vectorize -funsafe-math-optimizations -fno-strict-aliasing -flto -march=native -mtune=native -ftree-vectorize -funsafe-math-optimizations -fno-strict-aliasing -flto -std=gnu11 -D_CRT_SECURE_NO_WARNINGS -Wno-discarded-qualifiers -O3 -DNDEBUG -fsigned-char -ffast-math -fno-strict-aliasing -fomit-frame-pointer -fvisibility=hidden -fcommon -DGIT_VERSION=" 80e48a0" -DOS_LINUX -I./mupen64plus-core/subprojects/md5 -DARCH_MIN_SSE2 -msse -msse2 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -D__LIBRETRO__ -DUSE_FILE32API -DM64P_PLUGIN_API -DM64P_CORE_PROTOTYPES -D_ENDUSER_RELEASE -DSINC_LOWER_QUALITY -DTXFILTER_LIB -D__VEC4_OPT -DMUPENPLUSAPI -I./custom -I./custom/mupen64plus-core -I./custom/android/include -I./custom/GLideN64 -I./GLideN64/src -I./GLideN64/src/osal -I./mupen64plus-core/src -I./mupen64plus-core/src/api -I./custom/mupen64plus-core/plugin/audio_libretro -I./libretro-common/include -I./libretro -I./GLideN64/src/inc -I./custom/dependencies/libpng -I./mupen64plus-core/subprojects/minizip -I./xxHash -I./custom/dependencies/libzlib -fPIC -DCORE -DHAVE_OPENGL -DNEW_DYNAREC=2 -DDYNAREC -I./mupen64plus-core/src/asm_defines/ -c libretro-common/libco/libco.c -o libretro-common/libco/libco.o

I tried your proposal regardless:

Hmm... then try removing asm_defines.c from Makefile.common... https://github.com/libretro/mupen64plus-libretro-nx/blob/develop/Makefile.common#L38

And the error is the same. I'll reflect upon this some more. Though I think removing -flto from asm_defines.o is already an improvement.

m4xw commented 1 month ago

Well you cant use flto because it doesnt actually compile, just lifts it to a IR and we need the compiled object to pull the required infos out of it. LTO generally gives little to none improvement on this codebase and usually only hides bugs so i recommend not using it.

Also you look like u need a make clean

m4xw commented 1 month ago

At this point, I have no idea how this core has even worked... (I don't use RA)

Try adding:

ifeq ($(PIC), 1)
   fpic = -fPIC
   ASFLAGS += -d PIC
else
   fpic = -fno-PIC
endif

Here: https://github.com/libretro/mupen64plus-libretro-nx/blob/develop/Makefile#L638

not required here, just phony builds on his end

m4xw commented 1 month ago

@aktau btw can u add the output of uname -m

aktau commented 1 month ago

Sure:

$ uname -a
Linux n2807 6.9.9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.9-1 (2024-07-13) x86_64 GNU/Linux
$ uname -m
x86_64
aktau commented 1 month ago

Well you cant use flto because it doesnt actually compile, just lifts it to a IR and we need the compiled object to pull the required infos out of it. LTO generally gives little to none improvement on this codebase and usually only hides bugs so i recommend not using it.

Yes, LTO was indeed problematic for the AWK script mentioned above, but that was resolved using https://github.com/libretro/mupen64plus-libretro-nx/issues/543#issuecomment-2226799708. After that's fixed, the PIC error occurs while doing the final link.

aktau commented 1 month ago

Of note, when I use clang (same flags, with -flto), I get different errors:

clang -march=native -mtune=native -ftree-vectorize -funsafe-math-optimizations -fno-strict-aliasing -flto -march=native -mtune=native -ftree-vectorize -funsafe-math-optimizations -fno-strict-aliasing -flto -std=gnu11 -D_CRT_SECURE_NO_WARNINGS -Wno-discarded-qualifiers -O3 -DNDEBUG -fsigned-char -ffast-math -fno-strict-aliasing -fomit-frame-pointer -fvisibility=hidden -fcommon -DGIT_VERSION=\"" 80e48a0"\" -DOS_LINUX -I./mupen64plus-core/subprojects/md5 -DARCH_MIN_SSE2 -msse -msse2 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -D__LIBRETRO__ -DUSE_FILE32API -DM64P_PLUGIN_API -DM64P_CORE_PROTOTYPES -D_ENDUSER_RELEASE -DSINC_LOWER_QUALITY -DTXFILTER_LIB -D__VEC4_OPT -DMUPENPLUSAPI -I./custom -I./custom/mupen64plus-core -I./custom/android/include -I./custom/GLideN64 -I./GLideN64/src -I./GLideN64/src/osal -I./mupen64plus-core/src -I./mupen64plus-core/src/api -I./custom/mupen64plus-core/plugin/audio_libretro -I./libretro-common/include -I./libretro -I./GLideN64/src/inc -I./custom/dependencies/libpng -I./mupen64plus-core/subprojects/minizip -I./xxHash -I./custom/dependencies/libzlib  -fPIC  -DCORE -DHAVE_OPENGL -DNEW_DYNAREC=2 -DDYNAREC -I./mupen64plus-core/src/asm_defines/ -c custom/dependencies/libzlib/gzlib.c -o custom/dependencies/libzlib/gzlib.o
warning: unknown warning option '-Wno-discarded-qualifiers'; did you mean '-Wno-ignored-qualifiers'? [-Wunknown-warning-option]
custom/dependencies/libzlib/gzlib.c:75:12: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
local void gz_reset(state)
           ^
custom/dependencies/libzlib/gzlib.c:91:14: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
local gzFile gz_open(path, fd, mode)
             ^
custom/dependencies/libzlib/gzlib.c:256:24: error: call to undeclared function 'lseek'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        state->start = LSEEK(state->fd, 0, SEEK_CUR);
                       ^
custom/dependencies/libzlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
custom/dependencies/libzlib/gzlib.c:256:24: note: did you mean 'fseek'?
custom/dependencies/libzlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
/usr/include/stdio.h:779:12: note: 'fseek' declared here
extern int fseek (FILE *__stream, long int __off, int __whence)
           ^
custom/dependencies/libzlib/gzlib.c:268:16: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
gzFile ZEXPORT gzopen(path, mode)
               ^
custom/dependencies/libzlib/gzlib.c:276:16: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
gzFile ZEXPORT gzopen64(path, mode)
               ^
custom/dependencies/libzlib/gzlib.c:284:16: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
gzFile ZEXPORT gzdopen(fd, mode)
               ^
custom/dependencies/libzlib/gzlib.c:314:13: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
int ZEXPORT gzbuffer(file, size)
            ^
custom/dependencies/libzlib/gzlib.c:339:13: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
int ZEXPORT gzrewind(file)
            ^
custom/dependencies/libzlib/gzlib.c:355:9: error: call to undeclared function 'lseek'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    if (LSEEK(state->fd, state->start, SEEK_SET) == -1)
        ^
custom/dependencies/libzlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
custom/dependencies/libzlib/gzlib.c:362:19: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
z_off64_t ZEXPORT gzseek64(file, offset, whence)
                  ^
custom/dependencies/libzlib/gzlib.c:396:15: error: call to undeclared function 'lseek'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        ret = LSEEK(state->fd, offset - state->x.have, SEEK_CUR);
              ^
custom/dependencies/libzlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
custom/dependencies/libzlib/gzlib.c:439:17: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
z_off_t ZEXPORT gzseek(file, offset, whence)
                ^
custom/dependencies/libzlib/gzlib.c:451:19: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
z_off64_t ZEXPORT gztell64(file)
                  ^
custom/dependencies/libzlib/gzlib.c:468:17: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
z_off_t ZEXPORT gztell(file)
                ^
custom/dependencies/libzlib/gzlib.c:478:19: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
z_off64_t ZEXPORT gzoffset64(file)
                  ^
custom/dependencies/libzlib/gzlib.c:492:14: error: call to undeclared function 'lseek'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    offset = LSEEK(state->fd, 0, SEEK_CUR);
             ^
custom/dependencies/libzlib/gzlib.c:14:17: note: expanded from macro 'LSEEK'
#  define LSEEK lseek
                ^
custom/dependencies/libzlib/gzlib.c:501:17: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
z_off_t ZEXPORT gzoffset(file)
                ^
custom/dependencies/libzlib/gzlib.c:511:13: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
int ZEXPORT gzeof(file)
            ^
custom/dependencies/libzlib/gzlib.c:528:22: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
const char * ZEXPORT gzerror(file, errnum)
                     ^
custom/dependencies/libzlib/gzlib.c:549:14: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
void ZEXPORT gzclearerr(file)
             ^
custom/dependencies/libzlib/gzlib.c:575:20: warning: a function definition without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
void ZLIB_INTERNAL gz_error(state, err, msg)
                   ^
18 warnings and 4 errors generated.
gmake: *** [Makefile:685: custom/dependencies/libzlib/gzlib.o] Error 1
gmake: *** Waiting for unfinished jobs....
26 warnings generated.
cp "mupen64plus_next_libretro.so" "/home/aktau/github/libretro-super/dist/unix/mupen64plus_next_libretro.so"
cp: cannot stat 'mupen64plus_next_libretro.so': No such file or directory
1 core(s) failed:
   mupen64plus_next
m4xw commented 1 month ago

Use gcc

m4xw commented 1 month ago

I can confirm that the .so produced without -flto works.

I can also confirm that libco.o, which contains the co_active_handle it complains about:

<artificial>:(.text+0x3): undefined reference to `co_active_handle'
/usr/bin/ld: /tmp/ccxb1mXW.ltrans0.ltrans.o: relocation R_X86_64_PC32 against undefined symbol `co_active_handle' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad valu

...was actually compiled with -fPIC (without changing the makefile):

gcc -march=native -mtune=native -ftree-vectorize -funsafe-math-optimizations -fno-strict-aliasing -flto -march=native -mtune=native -ftree-vectorize -funsafe-math-optimizations -fno-strict-aliasing -flto -std=gnu11 -D_CRT_SECURE_NO_WARNINGS -Wno-discarded-qualifiers -O3 -DNDEBUG -fsigned-char -ffast-math -fno-strict-aliasing -fomit-frame-pointer -fvisibility=hidden -fcommon -DGIT_VERSION=" 80e48a0" -DOS_LINUX -I./mupen64plus-core/subprojects/md5 -DARCH_MIN_SSE2 -msse -msse2 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -D__LIBRETRO__ -DUSE_FILE32API -DM64P_PLUGIN_API -DM64P_CORE_PROTOTYPES -D_ENDUSER_RELEASE -DSINC_LOWER_QUALITY -DTXFILTER_LIB -D__VEC4_OPT -DMUPENPLUSAPI -I./custom -I./custom/mupen64plus-core -I./custom/android/include -I./custom/GLideN64 -I./GLideN64/src -I./GLideN64/src/osal -I./mupen64plus-core/src -I./mupen64plus-core/src/api -I./custom/mupen64plus-core/plugin/audio_libretro -I./libretro-common/include -I./libretro -I./GLideN64/src/inc -I./custom/dependencies/libpng -I./mupen64plus-core/subprojects/minizip -I./xxHash -I./custom/dependencies/libzlib -fPIC -DCORE -DHAVE_OPENGL -DNEW_DYNAREC=2 -DDYNAREC -I./mupen64plus-core/src/asm_defines/ -c libretro-common/libco/libco.c -o libretro-common/libco/libco.o

I tried your proposal regardless:

Hmm... then try removing asm_defines.c from Makefile.common... https://github.com/libretro/mupen64plus-libretro-nx/blob/develop/Makefile.common#L38

And the error is the same. I'll reflect upon this some more. Though I think removing -flto from asm_defines.o is already an improvement.

just do a make clean and compile without lto