Need a way to disable GOT accesses for rtld startup code

[arichardson]

rtld-cheri-elf started crashing after I added two new cases to the initial switch to handle DT_RELA and DT_RELASZ cases for the RELA relocations generated by LLD.

0x0000000160027050:  cld        t4,zero,0(c3)
    Memory Read [0000000160013750] = 0000000000000010
    Write t4 = 0000000000000010
0x0000000160027054:  slti       at,t4,17
    Write at = 0000000000000001
0x0000000160027058:  bnez       at,0x1600270a0
0x000000016002705c:  nop
0x00000001600270a0:  sltiu      at,t4,9
    Write at = 0000000000000000
0x00000001600270a4:  beqz       at,0x16002704c
0x00000001600270a8:  nop
0x000000016002704c:  cincoffset c3,c3,a3
    Write C03|v:1 s:0 p:7fff807d b:0000000000000000 l:0000010000000000
             |o:0000000160013760 t:0
0x0000000160027050:  cld        t4,zero,0(c3)
    Memory Read [0000000160013760] = 0000000000000004
    Write t4 = 0000000000000004
0x0000000160027054:  slti       at,t4,17
    Write at = 0000000000000001
0x0000000160027058:  bnez       at,0x1600270a0
0x000000016002705c:  nop
0x00000001600270a0:  sltiu      at,t4,9
0x00000001600270a4:  beqz       at,0x16002704c
0x00000001600270a8:  nop
0x00000001600270ac:  dmult      t4,a0
0x00000001600270b0:  mflo       at
    Write at = 0000000000000020
0x00000001600270b4:  daddu      at,at,a2
    Write at = 0000000000084e60
0x00000001600270b8:  ld at,0(at)
mips_cpu_do_interrupt enter: PC 00000001600270b8 EPC 0000000160027050 TLB load exception

The last few instructions up to the crash make me think that clang is generating a jump table for the switch now. I guess before there weren't enough consecutive cases to make that a useful optimization. Unfortunately the GOT has not been relocated yet so the attempted load fails and causes RTLD to crash. Furthermore, the jump tables use plain MIPS ld and jr, we should be using the capability equivalent.

I tried to work around this issue by using -O0 which makes the code run a bit further but then I get a crash due to a strange value loaded from the GOT when it tries to call the function to perform the relocations. I might be able to work around this by copy-pasting the relocation loop instead of having a separate function because it seems that clang is ignoring the __always_inline attribute.

I saw there is a LLVM patch to add -fno-jumptables which might help: https://reviews.llvm.org/rL265425. However, this is only a workaround, because e.g. calling other functions (I think even inlined ones) will use the GOT again.

The real solution would be a way to prevent any kind of GOT access for a given function/file because writing it all in assembly would not be much fun.

Here is the function that crashes:

void
_rtld_relocate_nonplt_self(Elf_Dyn *dynp, caddr_t relocbase)
{
    const Elf_Rel *rel = 0, *rellim;
    const Elf_Rela *rela = 0, *relalim;
    Elf_Addr relsz = 0, relasz = 0;
    const Elf_Sym *symtab = NULL, *sym;
    Elf_Addr *got = NULL;
    Elf_Word local_gotno = 0, symtabno = 0, gotsym = 0;
    size_t i;
    for (; dynp->d_tag != DT_NULL; dynp++) {
        switch (dynp->d_tag) {
        case DT_REL:
            rel = (const Elf_Rel *)(relocbase + dynp->d_un.d_ptr);
            break;
        case DT_RELSZ:
            relsz = dynp->d_un.d_val;
            break;
        case DT_RELA:
            rela = (const Elf_Rela *)(relocbase + dynp->d_un.d_ptr);
            break;
        case DT_RELASZ:
            relasz = dynp->d_un.d_val;
            break;
        case DT_SYMTAB:
            symtab = (const Elf_Sym *)(relocbase + dynp->d_un.d_ptr);
            break;
        case DT_PLTGOT:
            got = (Elf_Addr *)(relocbase + dynp->d_un.d_ptr);
            break;
        case DT_MIPS_LOCAL_GOTNO:
            local_gotno = dynp->d_un.d_val;
            break;
        case DT_MIPS_SYMTABNO:
            symtabno = dynp->d_un.d_val;
            break;
        case DT_MIPS_GOTSYM:
            gotsym = dynp->d_un.d_val;
            break;
        }
    }
    i = (got[1] & GOT1_MASK) ? 2 : 1;
    /* Relocate the local GOT entries */
    got += i;
    for (; i < local_gotno; i++) {
        *got++ += (uintptr_t)relocbase;
    }

    sym = symtab + gotsym;
    /* Now do the global GOT entries */
    for (i = gotsym; i < symtabno; i++) {
        *got = sym->st_value + (uintptr_t)relocbase;
        ++sym;
        ++got;
    }
    /* LLD generates RELA instead of REL so we need to process both. */
    relalim = rela ? (const Elf_Rela *)((char *)rela + relasz) : NULL;
    for (; rela < relalim; rela++) {
        _rtld_process_rela_self(rela, relocbase, symtab, gotsym, true);
    }
    rellim = rel ? (const Elf_Rel *)((char *)rel + relsz) : NULL;
    for (; rel < rellim; rel++) {
        Elf_Rela converted = {
            .r_info = rel->r_info,
            .r_offset = rel->r_offset,
            .r_addend = 0
        };
        _rtld_process_rela_self(&converted, relocbase, symtab, gotsym,
            false);
    }
}

[davidchisnall]

Clang has no knowledge of the GOT or anything to do with the binary format, so this is almost certainly an LLVM issue not a clang one.

[arichardson]

We now have -fno-jump-tables so this should be okay