search

ucb-bar / chipyard

An Agile RISC-V SoC Design Framework with in-order cores, out-of-order cores, accelerators, and more
https://chipyard.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
1.63k stars 646 forks source link

RV32 cross compiler #1188

Open nachoge98 opened 2 years ago

nachoge98 commented 2 years ago

Background Work

Chipyard Version and Hash

Release: 1.5.0 Hash: a6a6a6

OS Setup

Linux TUD211596 5.14.0-1042-oem #47-Ubuntu SMP Fri Jun 3 18:17:11 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.4 LTS Release: 20.04 Codename: focal

Other Setup

First I enabled multilib support at ~/chipyard/toolchains/libgloss following the instructions at https://github.com/ucb-bar/libgloss-htif.

Then I added the following flags to the CFLAGS section of the Makefile at ~/chipyard/tests -march=rv32im -mabi=ilp32

Current Behavior

Not being able to compile the RV32 simple test I wrote for my TinyRocketConfig:

libgloss-htif: Using global install
riscv64-unknown-elf-gcc -std=gnu99 -O2 -fno-common -fno-builtin-printf -Wall -march=rv32im -mabi=ilp32 -specs=htif_nano.specs -c 32b_test.c -o 32b_test.o
riscv64-unknown-elf-gcc -static  -specs=htif_nano.specs 32b_test.o -o 32b_test.riscv
/home/ignacio/Desktop/thesis/chipyard/riscv-tools-install/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: 32b_test.o: ABI is incompatible with that of the selected emulation:
  target emulation `elf32-littleriscv' does not match `elf64-littleriscv'
/home/ignacio/Desktop/thesis/chipyard/riscv-tools-install/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: failed to merge target specific data of file 32b_test.o
collect2: error: ld returned 1 exit status

Expected Behavior

The generated RV32 riscv of my C code Maybe adding a script to set the compiler in the toolchain automatically for RV32 would be interesting

Other Information

Edited Makefile for compiling RV32:

GCC=riscv64-unknown-elf-gcc
OBJDUMP=riscv64-unknown-elf-objdump
CFLAGS= -std=gnu99 -O2 -fno-common -fno-builtin-printf -Wall -march=rv32im -mabi=ilp32
LDFLAGS= -static

include libgloss.mk

PROGRAMS = 32b_test

spiflash.img: spiflash.py
    python3 $<

.DEFAULT_GOAL := default

.PHONY: default
default: $(addsuffix .riscv,$(PROGRAMS)) spiflash.img

.PHONY: dumps
dumps: $(addsuffix .dump,$(PROGRAMS))

%.o: %.S
    $(GCC) $(CFLAGS) -D__ASSEMBLY__=1 -c $< -o $@

%.o: %.c mmio.h spiflash.h
    $(GCC) $(CFLAGS) -c $< -o $@

%.riscv: %.o $(libgloss)
    $(GCC) $(LDFLAGS) $< -o $@

%.dump: %.riscv
    $(OBJDUMP) -D $< > $@

.PHONY: clean
clean:
    rm -f *.riscv *.o *.dump
    $(if $(libgloss),rm -rf $(libgloss_builddir)/)
leduchuybk commented 2 years ago

remove the spec option?

nachoge98 commented 2 years ago

remove the spec option?

there is no section in the makefile default to modify that option, how can it be done modifying the makefile?

leduchuybk commented 2 years ago

try make 32b_test.dump in tests

leduchuybk commented 2 years ago 
riscv64-unknown-elf-gcc -std=gnu99 -O2 -fno-common -fno-builtin-printf -Wall -march=rv32im -mabi=ilp32 -c 32b_test.c -o 32b_test.o
riscv64-unknown-elf-gcc -static 32b_test.o -o 32b_test.riscv

or you can tried this first

nachoge98 commented 2 years ago

try make 32b_test.dump in tests

I get the same error:

libgloss-htif: Using global install
riscv64-unknown-elf-gcc -static  -specs=htif_nano.specs 32b_test.o -o 32b_test.riscv
/home/ignacio/Desktop/thesis/chipyard/riscv-tools-install/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: 32b_test.o: ABI is incompatible with that of the selected emulation:
  target emulation `elf32-littleriscv' does not match `elf64-littleriscv'
/home/ignacio/Desktop/thesis/chipyard/riscv-tools-install/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: failed to merge target specific data of file 32b_test.o
collect2: error: ld returned 1 exit status
make: *** [Makefile:30: 32b_test.riscv] Error 1
nachoge98 commented 2 years ago 
riscv64-unknown-elf-gcc -std=gnu99 -O2 -fno-common -fno-builtin-printf -Wall -march=rv32im -mabi=ilp32 -c 32b_test.c -o 32b_test.o
riscv64-unknown-elf-gcc -static 32b_test.o -o 32b_test.riscv

or you can tried this first

I tried this after you told me to try removing the spec option, the first command works fine, the second one gives me the same error: 

riscv64-unknown-elf-gcc -static 32b_test.o -o 32b_test.riscv
/home/ignacio/Desktop/thesis/chipyard/riscv-tools-install/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: 32b_test.o: ABI is incompatible with that of the selected emulation:
  target emulation `elf32-littleriscv' does not match `elf64-littleriscv'
/home/ignacio/Desktop/thesis/chipyard/riscv-tools-install/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: failed to merge target specific data of file 32b_test.o
collect2: error: ld returned 1 exit status

Edit:

I'm adding my makefile code in the initial post.

leduchuybk commented 2 years ago

I suspected that your libgloss-htif is not built with muti lib option. Please check this link: https://github.com/ucb-bar/libgloss-htif Or maybe your toolchain also is not built with muti lib option Check: riscv64-unknown-elf-gcc --print-multi-lib

nachoge98 commented 2 years ago

I suspected that your libgloss-htif is not built with muti lib option. Please check this link: https://github.com/ucb-bar/libgloss-htif Or maybe your toolchain also is not built with muti lib option Check: riscv64-unknown-elf-gcc --print-multi-lib

As I explained in the initial post, I already followed the instructions in the libgloss page to enable multilib before opening the issue. when I do the check:

ignacio@TUD211596:~/Desktop/thesis/chipyard/toolchains/libgloss$ riscv64-unknown-elf-gcc --print-multi-lib
.;
michael-etzkorn commented 2 years ago

I think OpenTitan's riscv32 binaries should work for you. https://github.com/lowRISC/opentitan

Their flow installs a riscv32-unknown-elf-gcc and is pretty simple to get installed. You can point your $RISCV environment variable to where it installs under /tools/riscv. You'll need to make sure the bin is on your path, and might have to move the binaries.

Their python script fetches a pre-built tool chain with 32-bit compilers. Step3 in https://docs.opentitan.org/doc/getting_started/

nachoge98 commented 2 years ago

I think OpenTitan's riscv32 binaries should work for you. https://github.com/lowRISC/opentitan

Their flow installs a riscv32-unknown-elf-gcc and is pretty simple to get installed. You can point your $RISCV environment variable to where it installs under /tools/riscv. You'll need to make sure the bin is on your path, and might have to move the binaries.

Their python script fetches a pre-built tool chain with 32-bit compilers. Step3 in https://docs.opentitan.org/doc/getting_started/

Your reply was very helpful as I finally got to generate the .riscv file.

I changed the $RISCV and $PATH variables to point to the new toolchain, and compiled the C code using:

riscv32-unknown-elf-gcc -std=gnu99 -O2 -fno-common -fno-builtin-printf -Wall -march=rv32im -mabi=ilp32 -c 32b_test.c -o 32b_test.o
riscv32-unknown-elf-gcc -static 32b_test.o -o 32b_test.riscv

But now I encounter different problems, as now I am trying to run the generated test on a TinyRocketCore system:

ignacio@TUD211596:~/Desktop/thesis/chipyard/sims/verilator$ ./simulator-chipyard-myTinyRocketConfig-debug ../../tests/32b_test.riscv run-binary
This emulator compiled with JTAG Remote Bitbang client. To enable, use +jtag_rbb_enable=1.
Listening on port 35023
[UART] UART0 is here (stdin/stdout).
warning: tohost and fromhost symbols not in ELF; can't communicate with target
michael-etzkorn commented 2 years ago

Cool! Glad that toolchain compiles. You'll need to have tohost and fromhost in a linker script so that the FESVR can be set up. You can probably just use chipyard's htif.ld which is hopefully here toolchains/libgloss/util/htif.ld. Or you can just try re-adding the specs option -specs=htif_nano.specs https://chipyard.readthedocs.io/en/latest/Software/Baremetal.html used here.

Last time this happened, I suggested the person try merging their linker script with htif.ld and that worked out https://github.com/ucb-bar/chipyard/issues/1055. You can probably just directly use htif.ld as your linker script.

nachoge98 commented 2 years ago

Cool! Glad that toolchain compiles. You'll need to have tohost and fromhost in a linker script so that the FESVR can be set up. You can probably just use chipyard's htif.ld which is hopefully here toolchains/libgloss/util/htif.ld. Or you can just try re-adding the specs option -specs=htif_nano.specs https://chipyard.readthedocs.io/en/latest/Software/Baremetal.html used here.

Last time this happened, I suggested the person try merging their linker script with htif.ld and that worked out #1055. You can probably just directly use htif.ld as your linker script.

Given that I am still a bit of a newbie with all of this, I have tried just adding the -specs=htif_nano.specs.

ignacio@TUD211596:~/Desktop/thesis/chipyard/tests$ riscv32-unknown-elf-gcc -std=gnu99 -O2 -fno-common -fno-builtin-printf -Wall -march=rv32im -mabi=ilp32 -specs=/home/ignacio/Desktop/thesis/chipyard/toolchains/libgloss/util/htif_nano.specs -c 32b_test.c -o 32b_test.o
ignacio@TUD211596:~/Desktop/thesis/chipyard/tests$ riscv32-unknown-elf-gcc -specs=/home/ignacio/Desktop/thesis/chipyard/toolchains/libgloss/util/htif_nano.specs -static 32b_test.o -o 32b_test.riscv
/home/ignacio/Desktop/thesis/opentitan/tools/riscv/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lc_nano
/home/ignacio/Desktop/thesis/opentitan/tools/riscv/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lc_nano
/home/ignacio/Desktop/thesis/opentitan/tools/riscv/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lgloss_htif
collect2: error: ld returned 1 exit status

Looks like its missing some files which I do not know how to fix.

If I wanted to add htif.ld as my linker script, how should I do it? I don't see any option in the compiler to use a different linker script, neither I know where the one currently being used by the new toolchain is.

michael-etzkorn commented 2 years ago

Yeah, I'm not surprised that didn't work. I don't know the toolchains very well, so we're kinda just throwing things at the binary and seeing what sticks.

These linker flags might help. I believe -T is used to hook the linker script.

-static -T %path%to%htif.ld

nachoge98 commented 2 years ago

Yeah, I'm not surprised that didn't work. I don't know the toolchains very well, so we're kinda just throwing things at the binary and seeing what sticks.

These linker flags might help. I believe -T is used to hook the linker script.

-static -T %path%to%htif.ld

Sadly this didn't work either, if you have any other idea it would be helpful, thanks for your time.

ignacio@TUD211596:~/Desktop/thesis/chipyard/tests$ riscv32-unknown-elf-gcc -std=gnu99 -O2 -fno-common -fno-builtin-printf -Wall -march=rv32im -mabi=ilp32 -specs=/home/ignacio/Desktop/thesis/chipyard/toolchains/libgloss/util/htif_nano.specs -c 32b_test.c -o 32b_test.o
ignacio@TUD211596:~/Desktop/thesis/chipyard/tests$ riscv32-unknown-elf-gcc -static -T /home/ignacio/Desktop/thesis/chipyard/toolchains/libgloss/util/htif.ld -specs=/home/ignacio/Desktop/thesis/chipyard/toolchains/libgloss/util/htif_nano.specs -static 32b_test.o -o 32b_test.riscv
/home/ignacio/Desktop/thesis/opentitan/tools/riscv/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lc_nano
/home/ignacio/Desktop/thesis/opentitan/tools/riscv/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lc_nano
/home/ignacio/Desktop/thesis/opentitan/tools/riscv/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lgloss_htif
collect2: error: ld returned 1 exit status

I have also tried: -Adding the linker script as you mentioned in the first comand for the generation of 32b_test.o but didn't work either. -Adding the htif.ld linker script withouth the -specs=htif_nano.specs but I get the same problem as before: warning: tohost and fromhost symbols not in ELF; can't communicate with target. I have a question regarding this, if I can not communicate with the target, does this mean that I can not run the test on an FPGA implementation either? or just that I can not simulate using verilator?

michael-etzkorn commented 2 years ago

I'm not sure about Arty, but the VCU118 FPGA flow supported by chipyard uses a bootrom that reads the binary off an SD card, loads that to DDR memory, and jumps to it. That doesn't use fesvr/HTIF at all. I'll look into the Arty path this weekend, but certainly, an FPGA flow can go without the whole front end server approach.

Sorry to hear about all the compiling hiccups you're experiencing trying to target 32-bit. Looking a bit closer at your error message, I'm guessing your $LD_LIBRARY_PATH environment variable has riscv-tools-install/lib but the gloss_htif and c_nano libraries are 64-bit.

I'm sadly still a bit out of my depth here. My next crack at this would involve installing the riscv-32 toolchain from rocket-tools.

Though I'm not sure if these have been maintained recently (https://github.com/chipsalliance/rocket-chip/issues/2951).

I'll ping @a0u on the off chance he has some time to help.

Even if we solve your particular issue, it's clear chipyard could benefit from a 32 bit toolchain install path or better documentation on how to use it, especially since Chipyard now supports Ibex!

michael-etzkorn commented 2 years ago

-mcmodel=medany -Wl,--start-group -lc_nano -lgloss_htif -Wl,--end-group -lgcc -static -nostartfiles -dT htif.ld Is equivalent to passing -specs=htif_nano.specs

You can try these flags -mcmodel=medany -lgcc -static -nostartfiles -dT htif.ld but I'm guessing the libraries are needed for HTIF/FESVR to work properly.

Perhaps better than installing a third RISCV toolchain, you can try building libgloss directly per the instructions in its README https://github.com/ucb-bar/libgloss-htif with the --enable-multilib='rv32i/ilp32 rv32iac/ilp32 rv32im/ilp32 rv32imac/ilp32 rv32imafc/ilp32f rv64imac/lp64 rv64imafdc/lp64d' flag to configure

nachoge98 commented 2 years ago

I'm not sure about Arty, but the VCU118 FPGA flow supported by chipyard uses a bootrom that reads the binary off an SD card, loads that to DDR memory, and jumps to it. That doesn't use fesvr/HTIF at all. I'll look into the Arty path this weekend, but certainly, an FPGA flow can go without the whole front end server approach.

Do not worry, I don't want to make you spend time on that, it was just a question that I had as I am also working on testing it on the Arty system. But my knowledge on that is quite reduced so far.

I'm sadly still a bit out of my depth here. My next crack at this would involve installing the riscv-32 toolchain from rocket-tools.

I have just tried this, and again I have the problem of lacking the fromhost and tohost. I have also tried adding flags as suggested before to hook the linker script, but I ran again into the same problems as explained before using the other riscv32-unknown-elf-gcc.

Perhaps better than installing a third RISCV toolchain, you can try building libgloss directly per the instructions in its README https://github.com/ucb-bar/libgloss-htif with the --enable-multilib='rv32i/ilp32 rv32iac/ilp32 rv32im/ilp32 rv32imac/ilp32 rv32imafc/ilp32f rv64imac/lp64 rv64imafdc/lp64d' flag to configure

I have just tried this, using just the flag --enable-multilib (without the specific combinations, as some of them were giving errors, and in theory the default enable includes most of them) when building them from the beginning and I get the same error as before:

ABI is incompatible with that of the selected emulation: target emulation `elf32-littleriscv' does not match `elf64-littleriscv'

You can try these flags -mcmodel=medany -lgcc -static -nostartfiles -dT htif.ld but I'm guessing the libraries are needed for HTIF/FESVR to work properly.

Well, I also tried that, but although I can get the riscv executable it seems to lack again the fromhost and tohost symbols.

ignacio@TUD211596:~/Desktop/thesis/chipyard/tests$ riscv32-unknown-elf-gcc -std=gnu99 -O2 -fno-common -fno-builtin-printf -Wall -march=rv32im -mabi=ilp32 -mcmodel=medany -lgcc -static -nostartfiles -T /home/ignacio/Desktop/thesis/chipyard/toolchains/libgloss/util/htif.ld  -c 32b_test.c -o 32b_test.o
ignacio@TUD211596:~/Desktop/thesis/chipyard/tests$ riscv32-unknown-elf-gcc -static -mcmodel=medany -lgcc -static -nostartfiles -T /home/ignacio/Desktop/thesis/chipyard/toolchains/libgloss/util/htif.ld -static 32b_test.o -o 32b_test.riscv
/home/ignacio/Desktop/thesis/opentitan/tools/riscv/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: warning: cannot find entry symbol _start; defaulting to 0000000080000000

ignacio@TUD211596:~/Desktop/thesis/chipyard/sims/verilator$ ./simulator-chipyard-TinyRocketConfig-debug ../../tests/32b_test.riscv run-binary
This emulator compiled with JTAG Remote Bitbang client. To enable, use +jtag_rbb_enable=1.
Listening on port 38479
[UART] UART0 is here (stdin/stdout).
warning: tohost and fromhost symbols not in ELF; can't communicate with target

At this point I am wondering if it might be my fault, can you reproduce the error trying to compile a RV32 test? Maybe I do something wrong when setting up the --enable-multilib at libgloss

michael-etzkorn commented 2 years ago

Presumably you're running $ ../configure --prefix=${RISCV}/riscv32-unknown-elf --host=riscv32-unknown-elf

Or something to that effect. Can you try only using the 32-bit ISA/ABIs?

../configure --prefix=${RISCV}/riscv32-unknown-elf --host=riscv32-unknown-elf --enable-multilib='rv32i/ilp32 rv32iac/ilp32 rv32im/ilp32 rv32imac/ilp32 rv32imafc/ilp32f

I'd still try compiling with -specs=htif_nano.specs since I'm guessing you need that htif-libgloss library to correctly link the fromhost and tohost memory regions. Also, don't forget to run make clean. When you have a result from libgloss, add that output library file to your LD_LIBRARY_PATH

I'll try a build myself with enable-multilib sometime soon.

nachoge98 commented 2 years ago

Presumably you're running $ ../configure --prefix=${RISCV}/riscv32-unknown-elf --host=riscv32-unknown-elf

Or something to that effect. Can you try only using the 32-bit ISA/ABIs?

../configure --prefix=${RISCV}/riscv32-unknown-elf --host=riscv32-unknown-elf --enable-multilib='rv32i/ilp32 rv32iac/ilp32 rv32im/ilp32 rv32imac/ilp32 rv32imafc/ilp32f

When I try to run make the libgloss with a riscv32 prefix/host it fails:

$ ./configure --prefix=${RISCV}/riscv32-unknown-elf --host=riscv32-unknown-elf --enable-multilib
$ make
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany   -I ../include -D__ASSEMBLY__ -c -o misc/crt0.o ../misc/crt0.S
../misc/crt0.S: Assembler messages:
../misc/crt0.S:180: Error: Instruction csrr requires absolute expression
make: *** [Makefile:137: misc/crt0.o] Error 1

I have only copied the last message before the error.

Also, I can not configure with the ISA/ABIs that you posted on the message:

$ ../configure --prefix=${RISCV}/riscv32-unknown-elf --host=riscv32-unknown-elf --enable-multilib='rv32i/ilp32 rv32iac/ilp32 rv32im/ilp32 rv32imac/ilp32 rv32imafc/ilp32f'
checking for riscv32-unknown-elf-gcc... riscv32-unknown-elf-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether riscv32-unknown-elf-gcc accepts -g... yes
checking for riscv32-unknown-elf-gcc option to accept ISO C89... none needed
checking for riscv32-unknown-elf-ar... riscv32-unknown-elf-ar
checking for riscv32-unknown-elf-size... riscv32-unknown-elf-size
checking whether multilib is enabled... unknown
configure: error: illegal march-mabi value: rv32i/ilp32

It doesn't matter, I get the same error with all the combinations. So I have to do the default one --enable-multilib

michael-etzkorn commented 2 years ago

I can reproduce the ../configure failure. Not sure how best to fix it at the moment, but while poking around the opentitan tool, I did notice a libgloss.a file.

Optimastically, I think that one will work. Try adding /tools/riscv/riscv32-unknown-elf/lib to your LD_LIBRARY_PATH and recompiling with -specs=htif_nano.specs

nachoge98 commented 2 years ago

I can reproduce the ../configure failure. Not sure how best to fix it at the moment, but while poking around the opentitan tool, I did notice a libgloss.a file.

Optimastically, I think that one will work. Try adding /tools/riscv/riscv32-unknown-elf/lib to your LD_LIBRARY_PATH and recompiling with -specs=htif_nano.specs

Thank you for your time, is there any way to report this so they have a look for fixing it in the future?

Sadly it didn't work:

$ $RISCV
bash: /tools/riscv: Is a directory
$ $LD_LIBRARY_PATH
bash: /tools/riscv/riscv32-unknown-elf/lib:/tools/riscv/lib:/tools/riscv/lib: No such file or directory
$ riscv32-unknown-elf-gcc -std=gnu99 -O2 -fno-common -fno-builtin-printf -Wall -march=rv32im -mabi=ilp32 -specs=/home/ignacio/Desktop/thesis/chipyard/toolchains/libgloss/util/htif_nano.specs -c 32b_test.c -o 32b_test.o
$ riscv32-unknown-elf-gcc -specs=/home/ignacio/Desktop/thesis/chipyard/toolchains/libgloss/util/htif_nano.specs -static 32b_test.o -o 32b_test.riscv
/tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lc_nano
/tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lc_nano
/tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lgloss_htif
collect2: error: ld returned 1 exit status
michael-etzkorn commented 2 years ago

Something occurred to me while digging into how to debug this and finding this SO post.

Can you try running sudo chmod -R 777 /tools/riscv? It may be that ld is finding the library but cannot open it because of permissions (since /tools/riscv is installed off the root directory).

If that doesn't work, I'd be curious to see the verbose output of ld here. Try passing -Xlinker --verbose to gcc on the linking step.

nachoge98 commented 2 years ago

Something occurred to me while digging into how to debug this and finding this SO post.

Can you try running sudo chmod -R 777 /tools/riscv? It may be that ld is finding the library but cannot open it because of permissions (since /tools/riscv is installed off the root directory).

If that doesn't work, I'd be curious to see the verbose output of ld here. Try passing -Xlinker --verbose to gcc on the linking step.

The first idea didn't fix it, so there you have the part of the verbose with the access attempts:

==================================================
/tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: mode elf32lriscv
attempt to open 32b_test.o succeeded
32b_test.o
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/libgcc.a succeeded
/tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/libgcc.a
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/libc_nano.a failed
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/lib/libc_nano.a failed
/tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lc_nano
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/c_nano.a failed
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/lib/c_nano.a failed
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/libgcc.a succeeded
/tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/libgcc.a
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/libgcc.a succeeded
/tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/libgcc.a
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/libc_nano.a failed
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/lib/libc_nano.a failed
/tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lc_nano
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/c_nano.a failed
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/lib/c_nano.a failed
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/libgloss_htif.a failed
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/lib/libgloss_htif.a failed
/tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot find -lgloss_htif
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/gloss_htif.a failed
attempt to open /tools/riscv/lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/lib/gloss_htif.a failed
collect2: error: ld returned 1 exit status
michael-etzkorn commented 2 years ago

Ok. Yeah so turns out we need gloss_htif.a and not libgloss.a because of the htif extension.

Looking at the configure script, it matches rv32?*-?*|rv64?*-?* so my guess is that's a bug.

I managed to get a gloss_htif.a file by changing the - to / in that case statement in configure and passing in the non-64 bit arguments

    for multilib in $enableval ; do
            case "$multilib" in #(
-  rv32?*-?*|rv64?*-?*) :
+  rv32?*/?*|rv64?*/?*) :
../configure --prefix=${RISCV}/riscv32-unknown-elf --host=riscv32-unknown-elf --enable-multilib='rv32i/ilp32 rv32iac/ilp32 rv32im/ilp32 rv32imac/ilp32 rv32imafc/ilp32f'

https://github.com/ucb-bar/libgloss-htif/pull/9

metzkorn@DESKTOP:~/libgloss-htif/build$ ../configure --prefix=/tools/riscv/riscv32-unknown-elf --host=riscv32-unknown-elf --enable-multilib="rv32i/ilp32 rv32iac/ilp32 rv32im/ilp32 rv32imac/ilp32 rv32imafc/ilp32f"
checking for riscv32-unknown-elf-gcc... riscv32-unknown-elf-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether riscv32-unknown-elf-gcc accepts -g... yes
checking for riscv32-unknown-elf-gcc option to accept ISO C89... none needed
checking for riscv32-unknown-elf-ar... riscv32-unknown-elf-ar
checking for riscv32-unknown-elf-size... riscv32-unknown-elf-size
checking whether multilib is enabled... yes
configure: creating ./config.status
config.status: creating Makefile
metzkorn@DESKTOP:~/libgloss-htif/build$ make
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/misc/traps.o ../misc/traps.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/misc/tls.o ../misc/tls.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/misc/main.o ../misc/main.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/misc/htif.o ../misc/htif.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/nanosleep.o ../sys/nanosleep.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/access.o ../sys/access.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/chdir.o ../sys/chdir.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/chmod.o ../sys/chmod.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/chown.o ../sys/chown.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/close.o ../sys/close.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/execve.o ../sys/execve.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/exit.o ../sys/exit.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/faccessat.o ../sys/faccessat.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/fork.o ../sys/fork.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/fstat.o ../sys/fstat.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/fstatat.o ../sys/fstatat.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/ftime.o ../sys/ftime.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/getcwd.o ../sys/getcwd.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/getpid.o ../sys/getpid.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/gettimeofday.o ../sys/gettimeofday.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/isatty.o ../sys/isatty.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/kill.o ../sys/kill.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/link.o ../sys/link.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/lseek.o ../sys/lseek.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/lstat.o ../sys/lstat.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/open.o ../sys/open.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/openat.o ../sys/openat.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/read.o ../sys/read.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/sbrk.o ../sys/sbrk.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/stat.o ../sys/stat.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/sysconf.o ../sys/sysconf.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/times.o ../sys/times.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/unlink.o ../sys/unlink.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/utime.o ../sys/utime.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/wait.o ../sys/wait.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/sys/write.o ../sys/write.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/libc/stdio/vxprintf.o ../libc/stdio/vxprintf.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/libc/stdio/printf.o ../libc/stdio/printf.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/libc/stdio/sprintf.o ../libc/stdio/sprintf.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/libc/stdio/snprintf.o ../libc/stdio/snprintf.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/libc/stdio/puts.o ../libc/stdio/puts.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -c -o rv32imafc/ilp32f/libc/stdio/printbuf.o ../libc/stdio/printbuf.c
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -D__ASSEMBLY__ -c -o rv32imafc/ilp32f/misc/crt0.o ../misc/crt0.S
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -D__ASSEMBLY__ -c -o rv32imafc/ilp32f/misc/crtmain.o ../misc/crtmain.S
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -D__ASSEMBLY__ -c -o rv32imafc/ilp32f/misc/crtmain-argv.o ../misc/crtmain-argv.S
riscv32-unknown-elf-gcc -g -O2 -std=c11 -pedantic -Wall -fno-common -fno-builtin -mcmodel=medany -march=rv32imafc -mabi=ilp32f  -I ../include -D__ASSEMBLY__ -c -o rv32imafc/ilp32f/misc/init.o ../misc/init.S
riscv32-unknown-elf-ar rcs rv32imafc/ilp32f/libgloss_htif.a rv32imafc/ilp32f/misc/traps.o rv32imafc/ilp32f/misc/tls.o rv32imafc/ilp32f/misc/main.o rv32imafc/ilp32f/misc/htif.o rv32imafc/ilp32f/sys/nanosleep.o rv32imafc/ilp32f/sys/access.o rv32imafc/ilp32f/sys/chdir.o rv32imafc/ilp32f/sys/chmod.o rv32imafc/ilp32f/sys/chown.o rv32imafc/ilp32f/sys/close.o rv32imafc/ilp32f/sys/execve.o rv32imafc/ilp32f/sys/exit.o rv32imafc/ilp32f/sys/faccessat.o rv32imafc/ilp32f/sys/fork.o rv32imafc/ilp32f/sys/fstat.o rv32imafc/ilp32f/sys/fstatat.o rv32imafc/ilp32f/sys/ftime.o rv32imafc/ilp32f/sys/getcwd.o rv32imafc/ilp32f/sys/getpid.o rv32imafc/ilp32f/sys/gettimeofday.o rv32imafc/ilp32f/sys/isatty.o rv32imafc/ilp32f/sys/kill.o rv32imafc/ilp32f/sys/link.o rv32imafc/ilp32f/sys/lseek.o rv32imafc/ilp32f/sys/lstat.o rv32imafc/ilp32f/sys/open.o rv32imafc/ilp32f/sys/openat.o rv32imafc/ilp32f/sys/read.o rv32imafc/ilp32f/sys/sbrk.o rv32imafc/ilp32f/sys/stat.o rv32imafc/ilp32f/sys/sysconf.o rv32imafc/ilp32f/sys/times.o rv32imafc/ilp32f/sys/unlink.o rv32imafc/ilp32f/sys/utime.o rv32imafc/ilp32f/sys/wait.o rv32imafc/ilp32f/sys/write.o rv32imafc/ilp32f/libc/stdio/vxprintf.o rv32imafc/ilp32f/libc/stdio/printf.o rv32imafc/ilp32f/libc/stdio/sprintf.o rv32imafc/ilp32f/libc/stdio/snprintf.o rv32imafc/ilp32f/libc/stdio/puts.o rv32imafc/ilp32f/libc/stdio/printbuf.o rv32imafc/ilp32f/misc/crt0.o rv32imafc/ilp32f/misc/crtmain.o rv32imafc/ilp32f/misc/crtmain-argv.o rv32imafc/ilp32f/misc/init.o
metzkorn@DESKTOP:~/libgloss-htif/build$ ls
Makefile  config.log  config.status  libc  libgloss_htif.a  misc  rv32imac-ilp32  rv32imafc  rv32imafc-ilp32f  sys

I'm guessing libc_nano.a isn't necessary for this. If that change to configure works for you, try compiling with ./toolchains/libgloss/util/htif.specs instead of htif_nano.specs

I ended up getting the riscv binary to compile by linking the libgloss result under riscv32-unknown-elf as it seems the opentitan toolchain ignores LD_LIBRARY_PATH

/tools/riscv/riscv32-unknown-elf/lib$ sudo ln -s /home/metzkorn/libgloss-htif/build/libgloss_htif.a

michael-etzkorn commented 2 years ago

My solution only compiles march/mabi for the last argument to --enable-multilib. The $multilib variable needed to be set outside of the case statement. https://github.com/ucb-bar/libgloss-htif/pull/10 has the full fix.

michael-etzkorn commented 2 years ago

This appears to fix the tohost fromhost problem, but you might still have issues getting your printf to appear. Unfortunately, even after adding this change, I couldn't get a "Hello World" to appear in the verilator sim with TinyRocketConfig. I see the proper output with spike --isa=RV32 32b_test.riscv

nachoge98 commented 2 years ago

Thanks Michael.

This solves the previous tohost and fromhost errors, but I am getting a new one now, don't know if it is relevant or not and I am curious whether also get it:

ignacio@TUD211596:~/Desktop/thesis/chipyard/sims/verilator$ ./simulator-chipyard-TinyRocketConfig-debug ../../tests/32b_test.riscv run-binary
This emulator compiled with JTAG Remote Bitbang client. To enable, use +jtag_rbb_enable=1.
Listening on port 46273
[UART] UART0 is here (stdin/stdout).
*** FAILED *** (tohost = 7)
*** FAILED *** (code = 7, seed 1658204768) after 54561 cycles