Mips support

[ericriff]

I'm wondering if mold supports mips. I've been trying to link a hello world app with it but it fails with an odd message.

$ mipsel-poky-linux-musl-g++ -mel -mabi=32 -msoft-float -march=mips32r2 -mtune=24kec -mdsp -pipe -funwind-tables --sysroot=path-to/mips32r2-24kec-nf-poky-linux-musl /tmp//main.cpp -B/path-to/libexec/mold

mold: fatal: unknown -m argument: ips32r2
collect2: error: ld returned 1 exit statu

From this thread I get that mold did not support mips back in October. I'm wondering if that's still the case.

This is on mold 1.11.0.

Thanks.

[rui314]

MIPS isn't supported due to the following two reasons. First, it looks like many Linux distros are dropping MIPS support implying that the need is declining. Second, the MIPS ABIs have been diverted too much from other ELF ABIs and thus hard to support.

[ericriff]

I see. Thank you very much for te quick response. I suggest adding this disclaimer on the readme for future reference (assuming it is not already there and I missed it). Feel free to close this one.

Eric.

[rui314]

We have a list of supported targets in the readme instead of naming all non-supported targets, so I think it's already covered. Btw if you have a commercial interest in getting a MIPS support, we can negotiate. MIPS is too complicated to support as a volunteer, but for paying customers it could be a different story.

[ericriff]

Nice to hear that. The company where I work is still evaluating mold. We work with a variety of platforms and a single one is based on MIPS but, AFAIK, is very close the EOL. I'll email you if we decide to move forward. Thanks.

[alphaonex86]

PS: all the GPON fiber device of realtek is today released with mips32r2 without FPU

[antermin]

Many routers are using MIPS, and OpenWrt has introduced support for the mold linker (https://github.com/openwrt/openwrt/pull/11895). If mold adds MIPS support, OpenWrt will be able to use mold for MIPS devices.

[dhewg]

Not just many, mips is the most used arch for OpenWrt supported devices, the current count is 1615 out of 2172: https://openwrt.org/toh/views/toh_extended_all?dataflt%5BPackage+architecture%7E%5D=mips Compared to just 316 arm* devices.

[rui314]

I've been trying to add mips support this week. I cannot promise it'll make it, but at least I'm trying to understand what's exactly needed to support mips.

[dhewg]

Nice! If you require testers I have mipsel_24kc hardware myself, which is compiled via -mips32r2 -mtune=24kc

[mcprat]

Btw if you have a commercial interest in getting a MIPS support, we can negotiate. MIPS is too complicated to support as a volunteer, but for paying customers it could be a different story.

@rui314 consider crowdfunding. I wouldn't be able to individually pay for your time on achieving MIPS support, but I'm sure there are several parties interested for various reasons, including the poster of this issue, myself, and the Openwrt community.

Also, don't think you would be fully alone in this adventure. Basic/initial MIPS support would lead to some projects like Openwrt starting to use mold more, at least for testing, and then someone in that project may come up with solutions for a random issue.

[rui314]

I've been attempting to support MIPS, but unfortunately, it's more challenging than expected due to its ABI. The MIPS instruction set is okay, but the ABI diverges significantly from that of the rest of the world. A lot of small details are different for no obvious reason. This is likely due to the lack of ownership of the ABI for almost 25 years since the collapse of Silicon Graphics. If someone had tried to narrow the gap between the MIPS ABI and other ABIs, it could have been doable. However, it seems that no one wanted to invest that kind of effort into this ABI. That's unfortunate, but I also lack the incentive to address this issue.

Given the result of the experiment, it is unfortunately more likely that we will drop the current incomplete MIPS support than complete it.

[mcprat]

I took a look into this and I somewhat agree, it is quite a nightmare at a glance

There are apparently 12 variants of the MIPS ABI, however, I think most of them are uncommon and not used, and in the context of building Linux, it may be that only 2 or 4 are used, depending on the hardware...

GCC Binutils (bfd) names them as follows:

elf32-littlemips
elf32-bigmips
elf32-tradlittlemips
elf32-tradbigmips
elf32-nlittlemips
elf32-nbigmips
elf32-ntradlittlemips
elf32-ntradbigmips
elf64-littlemips
elf64-bigmips
elf64-tradlittlemips
elf64-tradbigmips

the 32 vs 64 is obviously the bit width of the architecture

the big vs little is obviously endianness

and the trad stands for "traditional" vs "non-traditional" which apparently is the difference between what SGI's compilers did compared to the ELF standard

and then finally the "n" which stands for New ABI (N32 or N64 vs O32 in SGI's terminology) (where "O" stands for old)

Some info and literature I found... https://binutils.sourceware.narkive.com/ANMqKmua/really-confused-by-mips-target-formats https://www.linux-mips.org/wiki/P32_Linux_ABI (p32 is not really for MIPS but it has info on the other ABIs) https://www.linux-mips.org/pub/linux/mips/doc/ABI/MIPS-N32-ABI-Handbook.pdf

Hopefully I'm helping and not just linking things here that you've already found...

I would suggest you at least keep whatever you have added to the source so far rather than delete whatever progress you made. Someone may see what has started and want to finish it. I don't know much about this level of software but I would recommend picking the most popular of these variants and just focus on that just to see if one of them is doable (or maybe whatever one is most commonly used for hardware that you already have).

GCC Binutils source should be a valuable resource right? Or maybe I'm mistaken and mold takes a drastically different approach? https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=bfd/elfn32-mips.c

[mcprat]

I see you were specifically working on mips64, so here's more literature

https://math-atlas.sourceforge.net/devel/assembly/mipsabi64.pdf

[rui314]

The lack of documentation and having too many ABIs for the single ISA are problems for sure, but the real problem I'm raising is the ABI themselves. All of the MIPS ABIs are very strange, diverged too much from ABIs for other ISAs, and are in my opinion badly designed. These factors increase the cost of porting the linker to MIPS and also increase the cost of maintainership.

[mcprat]

so is it really outside of the current scope of mold due to how you designed it? or just much more difficult?

maybe asking the linux-mips mailing list if anyone wants to help?

[rui314]

Why don't you ask yourself? If the MIPS community wants this, you could ask them for help. I personally don't need a MIPS support and don't consider myself a member of the MIPS community, so it might seem strange if I were to request help. Sorry.

[mcprat]

my concern though, is that it sounds like mold would have to be slightly "redesigned" in order to accommodate MIPS since it is so different from other modern ABIs. Or is that an overstatement and it's just difficult?

[rui314]

Some parts need to be redesigned. For example, we need to remove the assumption that the .got section is a singleton section.

[rui314]

I cannot believe this, but somehow I managed to improve the MIPS support to the point that all of our tests pass on MIPS64LE/MIPS64BE. If you have a 64-bit MIPS machine, please try it out and let me know if it works for you.

[dhewg]

Woot, nice! Throwing this at a random OpenWrt malta build (-mips64r2 -mtune=mips64r2 -mabi=64), some things seem to build, but others don't:

mold: error: /tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/lib/gcc/mips64-openwrt-linux-musl/13.2.0/crtbeginS.o:(.text): relocation R_MIPS_GOT_DISP against __cxa_finalize out of range: 47480 is not in [-32768, 32768); recompile with -mxgot
mold: error: /tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/lib/gcc/mips64-openwrt-linux-musl/13.2.0/crtbeginS.o:(.text): relocation R_MIPS_GOT_DISP against __dso_handle out of range: 39104 is not in [-32768, 32768); recompile with -mxgot
mold: error: policydb_convert.lo:(.text): relocation R_MIPS_GOT_DISP against __stack_chk_guard out of range: 47864 is not in [-32768, 32768); recompile with -mxgot
mold: error: policydb_convert.lo:(.text): relocation R_MIPS_CALL16 against policy_file_init out of range: 41176 is not in [-32768, 32768); recompile with -mxgot
mold: error: ibendport_record.lo:(.text): relocation R_MIPS_CALL16 against calloc out of range: 47848 is not in [-32768, 32768); recompile with -mxgot
mold: error: policydb_convert.lo:(.text): relocation R_MIPS_CALL16 against policydb_read out of range: 41192 is not in [-32768, 32768); recompile with -mxgot
mold: error: policydb_convert.lo:(.text): relocation R_MIPS_CALL16 against policydb_destroy out of range: 41112 is not in [-32768, 32768); recompile with -mxgot

(and a few hundred more of those)

And poking at the things that did actually link:

readelf -a zlib-1.2.13/libz.so.1.2.13 1>/dev/null
readelf: Error: Bad attribute length (1090519040 > 288)
readelf: Error: Bad subsection length (1090519040 > 257)
readelf: Error: Corrupt MIPS ABI Flags section.

[dhewg]

I noticed 9893bb331fd33465b922ad2ff5aa81762d03dd95, but even when recompiling everything with -mxgot I do get those relocation errors (or I'm doing it wrong)

[rui314]

Did you get the same error message as before after recompiling your source files with -mxgot?

[dhewg]

At least in part, there're so many I didn't check that closely. Here's an example: out.txt

While there're some libc symbols mentioned in the errors, there're definitely errors about symbols which were just built with -mxgot, like:

mold: error: policydb_convert.lo:(.text): relocation R_MIPS_CALL_LO16 against policy_file_init out of range: 41048 is not in [-32768, 32768); recompile with -mxgot
$ grep -r policy_file_init *
libsepol-3.5/src/policydb.c:void policy_file_init(policy_file_t *pf)
mips64-openwrt-linux-musl-gcc -Os -pipe -mno-branch-likely -mxgot -mips64r2 -mtune=mips64r2 -mabi=64 -fno-caller-saves -fno-plt -fhonour-copts -msoft-float -fmacro-prefix-map=/home/andre/src/openwrt/build_dir/target-mips64_mips64r2_64_musl/libsepol-3.5=libsepol-3.5 -DPIC -fpic -Wformat -Werror=format-security -DPIC -fpic -fstack-protector-strong -D_FORTIFY_SOURCE=2 -Wl,-z,now -Wl,-z,relro  -I/home/andre/src/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/usr/include -I/home/andre/src/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/include/fortify -I/home/andre/src/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/include  -I. -I../include -D_GNU_SOURCE -I../cil/include -DHAVE_REALLOCARRAY -fPIC -c -o policydb.o policydb.c

[dhewg]

policydb.o does have R_MIPS_CALL_LO16 relocations, but in this case it's:

000000006418  00b700051807 R_MIPS_GPREL16    0000000000006410 policy_file_init + 0
                    Type2: R_MIPS_SUB       
                    Type3: R_MIPS_HI16      
000000006420  00b700061807 R_MIPS_GPREL16    0000000000006410 policy_file_init + 0
                    Type2: R_MIPS_SUB       
                    Type3: R_MIPS_LO16      

[rui314]

Please try again with the git head.

[dhewg]

mold segfaults on a release build:

        Signal: 11 (SEGV)
  Command Line: /home/andre/src/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/mips64-openwrt-linux-musl-ld.mold -plugin /tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../libexec/gcc/mips64-openwrt-linux-musl/13.2.0/liblto_plugin.so -plugin-opt=/tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../libexec/gcc/mips64-openwrt-linux-musl/13.2.0/lto-wrapper -plugin-opt=-fresolution=/home/andre/src/openwrt/tmp/ccZlcmId.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s --eh-frame-hdr -EB -mips64r2 -export-dynamic -dynamic-linker /lib/ld-musl-mips64-sf.so.1 -melf64btsmip -pie -o cmTC_6df28 -z now -z relro /tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../lib/gcc/mips64-openwrt-linux-musl/13.2.0/../../../../mips64-openwrt-linux-musl/lib/../lib64/Scrt1.o /tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../lib/gcc/mips64-openwrt-linux-musl/13.2.0/../../../../mips64-openwrt-linux-musl/lib/../lib64/crti.o /tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../lib/gcc/mips64-openwrt-linux-musl/13.2.0/crtbeginS.o -L/home/andre/src/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/usr/lib -L/home/andre/src/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/lib -L /home/andre/src/openwrt/staging_dir/target-mips64_mips64r2_64_musl/usr/lib -rpath-link /home/andre/src/openwrt/staging_dir/target-mips64_mips64r2_64_musl/usr/lib -L/tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../lib/gcc/mips64-openwrt-linux-musl/13.2.0 -L/tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../lib/gcc -L/tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../lib/gcc/mips64-openwrt-linux-musl/13.2.0/../../../../mips64-openwrt-linux-musl/lib/../lib64 -L/tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../lib/gcc/mips64-openwrt-linux-musl/13.2.0/../../../../mips64-openwrt-linux-musl/lib -z now -z relro CMakeFiles/cmTC_6df28.dir/testCCompiler.c.o -lssp_nonshared -lgcc_s -lc -lgcc_s /tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../lib/gcc/mips64-openwrt-linux-musl/13.2.0/crtendS.o /tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/../lib/gcc/mips64-openwrt-linux-musl/13.2.0/../../../../mips64-openwrt-linux-musl/lib/../lib64/crtn.o
    Executable: /tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/mips64-openwrt-linux-musl-ld.mold
                Stack trace of thread 479488:
                #0  0x00007f6a7baa80fc __pthread_kill_implementation (libc.so.6 + 0x8a0fc)
                #1  0x00007f6a7ba5a472 __GI_raise (libc.so.6 + 0x3c472)
                #2  0x0000556cab2eeb9d n/a (/tmp/openwrt/staging_dir/toolchain-mips64_mips64r2_64_gcc-13.2.0_musl/bin/mips64-openwrt-linux-musl-ld.mold + 0x1057b9d)
                #3  0x0000000000000000 n/a (n/a + 0x0)
                ELF object binary architecture: AMD x86-64

I tried to get a more useful backrace with a debug build, but that seems to work?

[rui314]

Can you try again?

If it still fails, please rebuild your mold with -DCMAKE_BUILD_TYPE=Debug and paste a stack trace here.

[dhewg]

Same thing, a release build segfaults with a useless backtrace while a debug build doesn't.

No idea if it's related, but there's still:

readelf -a libz.so.1.2.13 >/dev/null
readelf: Error: Bad attribute length (1090519040 > 288)
readelf: Error: Bad subsection length (1090519040 > 257)
readelf: Error: Corrupt MIPS ABI Flags section.

That's from this BE -mxgot binary: libz.so.1.2.13.gz

[dhewg]

The segfault is due to either ea75dae8ec95d7b95be38055e7838387d9c2562c or 6d290babe29f3a34e473ccf94339bdc17e12df73 (it doesn't build in between)

[dhewg]

I can build a full OpenWrt image using 428faa75b269daf99d89698769bed95d708a0aae with 24078a4395be62cad26bec2647baf70b57f0d768 cherry-picked on top and without -mxgot. But the binaries do not work

[dhewg]

710dfd01d695a47807d98e32d6f41495c1c70478 fixed the crash, but binaries are still non functional.

In case it helps, attached are two libz build, bfd and mold linked: libz.so.1.2.13.tar.gz

[mcprat]

@dhewg just a friendly reminder to make sure you're not using ccache

[rui314]

I removed our incomplete MIPS support code from mold. Please read the commit message (quote below) for the reason.

--

I made a fair amount of effort to try to support MIPS, but it turned out that it is much harder than expected. I punted it instead of making further efforts.

The problem is the MIPS ABI is hostile to the linker in the modern environment. MIPS object files are still compiled for the small code model in which GOT entries and data in the small data area are expected to be accessible with a single machine instruction with a 16-bit displacement. In other words, .got/.sdata/.sbss are expected to be smaller than 64 KiB. This might have been a reasonable assumption in the 90s, but it's not suitable for modern applications that can be 1000x larger than the binaries in the 90s.

MIPS requires the linker to implement tons of workarounds for its legacy ABI assumptions. Our incomplete MIPS support can build binaries that pass all our unit tests. However, it needed more effort to support real-world programs that are larger than our test cases.

At this point, I don't think it is productive to implement workarounds for the old ABI that is stuck in the 90s. It is honestly annoying to think about workarounds for the code that is intentionally compiled to be hostile to the linker. The situation is unfortunate, but if the Open Source community is still serious about MIPS, they should improve the ABI and the compiler instead of asking us to implement the legacy ABI.

[dhewg]

Aww, thanks alot for trying though!

[rui314]

It's an unfortunate conclusion for me. But honestly, unless someone takes the responsibility to improve the compiler and modernize the processor-specific ABI, it'd be hard to stop the trend of deterioration of the MIPS toolchain ecosystem. I'm not sure if there's still enough people who are interested in working on something new for the MIPS ecosystem, given that the majority seems to have been migrated to ARM or RISC-V, though.