search

xpack-dev-tools / riscv-none-embed-gcc-xpack

DEPRECATED: Please update to risc-none-elf-gcc-xpack
MIT License
113 stars 30 forks source link

Issue with multilib in Windows v10.1.0-1.1 #10

Closed manukuma-intel closed 2 years ago

manukuma-intel commented 3 years ago

Description

On the Windows build of v10.1.0-1.1, there appear to be some missing multilib arch/abi to directory mappings. The Linux build of v10.1.0-1.1 works fine however. As well, the previous v8.3.0-2.3 works fine on both platforms.

This becomes an issue when manually specifying link options like -lstdc++ when using -nostdlib as the the appropriate library search path for the arch/abi is not included. For example when using -march=rv32ia and mabi=ilp32 the ...\xpack-riscv-none-embed-gcc-8.3.0-2.3\riscv-none-embed\lib\rv32ia\ilp32 is not in the linker invocation library search path (-L).

Steps to Reproduce

  1. Extract binary zip on Windows
  2. riscv-none-embed-gcc.exe --print-multi-lib

(Good/Expected) Linux behavior with v10.1.0-1.1*

-> riscv-none-embed-gcc --print-multi-lib
.;
rv32e/ilp32e;@march=rv32e@mabi=ilp32e
rv32ea/ilp32e;@march=rv32ea@mabi=ilp32e
rv32em/ilp32e;@march=rv32em@mabi=ilp32e
rv32eac/ilp32e;@march=rv32eac@mabi=ilp32e
rv32emac/ilp32e;@march=rv32emac@mabi=ilp32e
rv32i/ilp32;@march=rv32i@mabi=ilp32
rv32if/ilp32f;@march=rv32if@mabi=ilp32f
rv32ifd/ilp32d;@march=rv32ifd@mabi=ilp32d
rv32ia/ilp32;@march=rv32ia@mabi=ilp32
rv32iaf/ilp32f;@march=rv32iaf@mabi=ilp32f
rv32iafd/ilp32d;@march=rv32iafd@mabi=ilp32d
rv32imaf/ilp32f;@march=rv32imaf@mabi=ilp32f
rv32im/ilp32;@march=rv32im@mabi=ilp32
rv32imf/ilp32f;@march=rv32imf@mabi=ilp32f
rv32imfc/ilp32f;@march=rv32imfc@mabi=ilp32f
rv32imfd/ilp32d;@march=rv32imfd@mabi=ilp32d
rv32iac/ilp32;@march=rv32iac@mabi=ilp32
rv32imac/ilp32;@march=rv32imac@mabi=ilp32
rv32imafc/ilp32f;@march=rv32imafc@mabi=ilp32f
rv32imafdc/ilp32d;@march=rv32imafdc@mabi=ilp32d
rv64i/lp64;@march=rv64i@mabi=lp64
rv64if/lp64f;@march=rv64if@mabi=lp64f
rv64ifd/lp64d;@march=rv64ifd@mabi=lp64d
rv64ia/lp64;@march=rv64ia@mabi=lp64
rv64iaf/lp64f;@march=rv64iaf@mabi=lp64f
rv64iafd/lp64d;@march=rv64iafd@mabi=lp64d
rv64imaf/lp64f;@march=rv64imaf@mabi=lp64f
rv64im/lp64;@march=rv64im@mabi=lp64
rv64imf/lp64f;@march=rv64imf@mabi=lp64f
rv64imfc/lp64f;@march=rv64imfc@mabi=lp64f
rv64imfd/lp64d;@march=rv64imfd@mabi=lp64d
rv64iac/lp64;@march=rv64iac@mabi=lp64
rv64imac/lp64;@march=rv64imac@mabi=lp64
rv64imafc/lp64f;@march=rv64imafc@mabi=lp64f
rv64imafdc/lp64d;@march=rv64imafdc@mabi=lp64d

(Bad/Unexpected) Windows behavior with v10.1.0-1.1

PS C:\Users\manukuma\Downloads> .\xpack-riscv-none-embed-gcc-10.1.0-1.1-win32-x64\xpack-riscv-none-embed-gcc-10.1.0-1.1\bin\riscv-none-embed-gcc.exe --print-multi-lib
.;
rv32i/ilp32;@march=rv32i@mabi=ilp32
rv32im/ilp32;@march=rv32im@mabi=ilp32
rv32iac/ilp32;@march=rv32iac@mabi=ilp32
rv32imac/ilp32;@march=rv32imac@mabi=ilp32
rv32imafc/ilp32f;@march=rv32imafc@mabi=ilp32f
rv64imac/lp64;@march=rv64imac@mabi=lp64
rv64imafdc/lp64d;@march=rv64imafdc@mabi=lp64d

There's many multilib configurations missing when querying the Windows GCC v10.1.0-1.1 version!

Versions

ilg-ul commented 3 years ago

Thank you for your report, this is definitely a build error.

I'll make a new release to fix it, hopefully in a few days.

manukuma-intel commented 3 years ago

Thank you for looking into it. I can also help test it out before hand or when the new release comes out.

ilg-ul commented 3 years ago

Thank you, I'll let you know when I have a test pre-release.

It'll take a while, right now I'm re-working all build scripts to add some automation. When ready, I'll make releases for all xPack binary packages, including this one.

ilg-ul commented 3 years ago

The default toolchain builds only a limited number of libraries. To build more, it is required to run a script to update a file in the source tree.

Due to a problem in the build scripts, this multi-lib script was not executed for Windows.

Fixed in 10.1.0-1.2.

Please check if everything is ok and possibly close the ticket.

manukuma-intel commented 2 years ago

Thanks for building the new version Liviu. It looks like the new release contains more multilib mappings than before, but I think there's a regression. All of the mappings (from riscv-none-embed-gcc --print-multi-lib) are all compressed instruction variants (i.e. has the c in the arch string). So the libraries would no longer work with CPUs that do not support compressed instructions.

Note this seems to be the case both with Windows and Linux builds.

ilg-ul commented 2 years ago

The xPack binaries fully replicate the SiFive binaries, so if they use only compressed instructions, so does the the xPack binaries.

For me the RISC-V multi-lib configurations is just a huge mess and I simply have no criteria which are useful and which are not; I hoped that SiFive knows better, but apparently not. :-(

Just curious, why would CPUs not support compressed instructions?

TommyMurphyTM1234 commented 2 years ago

Just curious, why would CPUs not support compressed instructions?

One reason might be omitting the C (compressed) extension or making it optional for a gatecount/area conscious soft IP implementation targeting FPGA? I doubt that there are (m)any hard ASIC RISC-V implementations that don't support the C extension, but I could be wrong.

FWIW some (most?) available RISC-V implementations are listed here and some do not support the C extension or have it as an option:

https://riscv.org/exchange/cores-socs/

Note that the default riscv-gnu-toolchain (https://github.com/riscv-collab/riscv-gnu-toolchain) multilib list includes two non C variants for the bare metal toolchain, namely rv32i/ilp32 and rv32im/ilp32:

https://github.com/riscv-collab/riscv-gcc/blob/riscv-gcc-10.2.0/gcc/config/riscv/t-elf-multilib

ilg-ul commented 2 years ago

omitting the C extension or making it optional for a gatecount/area conscious soft IP implementation targeting FPGA?

Well, if someone decides to step outside the bitten path with a custom implementation, should also be prepared to consider a custom toolchain.

the default riscv-gnu-toolchain (https://github.com/riscv-collab/riscv-gnu-toolchain) multilib list includes two non C variants, namely rv32i/ilp32 and rv32im/ilp32:

I don't know of that configuration, the SiFive toolchain uses a different list of multilibs:

TommyMurphyTM1234 commented 2 years ago

Well, if someone decides to step outside the bitten path with a custom implementation, should also be prepared to consider a custom toolchain.

Perhaps. Or somebody (not necessarily you, mind) might build a toolchain with a wider set of multilibs than the SiFive or riscv-gnu-toolchain defaults (which I, I guess mistakenly?, assumed were identical). Obviously the more multilibs the bigger the toolchain bloat of course.

A pro or con (depending on your viewpoint) of RISC-V and all its optional extensions is that there is significant flexibility possible in the hardware - which has implications for the tools needed to support the wide variety of possible targets.

I don't know of that configuration, the SiFive toolchain uses a different list of multilibs:

Fair enough. I tend to focus on the riscv-gnu-toolchain repo myself these days and not what SiFive might be doing to support their own targets.

manukuma-intel commented 2 years ago

Yup! As Tommy says, for FPGA implementations omitting the C extension can help.

What's weird is that 10.1.0 on Linux contains the rv32ia variants, but taking a look at SiFive's GCC branch for 10.1.0 (and even for 8.x), it doesn't contain rv32ia! So it looks like they only targeted variants with Compressed extension support all along.

Sounds like custom toolchain might be the way to go...

TommyMurphyTM1234 commented 2 years ago

Sounds like custom toolchain might be the way to go...

If you're going to roll/build your own then I'd suggest that you either use @ilg-ul's build scripts (https://xpack.github.io/riscv-none-embed-gcc/) with a modified t-elf-multilib or have a look at crosstool-NG (https://crosstool-ng.github.io/). I've had success in building custom tools/multilibs with both approaches. Of course you could also just build straight from the riscv-gnu-toolchain or SiFive repos too having tweaked the multilib list appropriately... :-)

ilg-ul commented 2 years ago

Or somebody ... might build a toolchain with a wider set of multilibs than the SiFive or riscv-gnu-toolchain defaults

If necessary I can do this, but what would be the extra combinations?

The SiFive 'basic' configuration includes some non-c libs, but they are no longer present in the large one:

    rv32e-ilp32e-- \
    rv32i-ilp32-- \
    rv32if-ilp32f-- \
    rv32ifd-ilp32d-- \

    rv64i-lp64-- \
    rv64if-lp64f-- \
    rv64ifd-lp64d-- \

I have no idea if these are useful, if they are enough, and if not, which one would be needed?

TommyMurphyTM1234 commented 2 years ago

If necessary I can do this, but what would be the extra combinations?

Difficult to answer that. When I worked for Microchip/Microsemi we built something like 40 different multilibs as part of SoftConsole in an attempt to cover our own soft IP/FPGA and ASIC/SoC FPGA RISC-V targets as well as other variants that third parties might use. But short of building absolutely every possible multilib (which would result in an absolutely huge toolchain) there's no obvious answer. The ones that SoftConsole bundles are listed in the SoftConsole release notes.

https://www.microsemi.com/product-directory/design-tools/4879-softconsole#downloads

As you can see from the release notes the aim was to support all relevant architectures and their "natural" ABIs. RVE (embedded) was all explicit omission because it was still in transition at the time (not sure if that's changed since?). And other extensions (e.g. RVV, bit manipulation etc.) are not catered for either for similar reasons.

TommyMurphyTM1234 commented 2 years ago

What's weird is that 10.1.0 on Linux contains the rv32ia variants

I presume you mean a/the RISC-V toolchain installed on a Linux distro using the platform's package manager (e.g. apt or similar)? What multilibs are supported really depends on how the maintainer of the binary distribution builds the tools which may differ from other approaches (e.g. such as xPack/@ilg-ul, SiFive, riscv-gnu-toolchain etc.).

manukuma-intel commented 2 years ago

@TommyMurphyTM1234 Sorry should be "rv32ia variant" (singular) - that variant was present from this repo's 8.x (Windows/Linux) and 10.1.0 (Linux) toolchain releases. The Windows version of 10.1.0 did not have the rv32ia variant, which was why I filed this issue. But now... we seemed to have uncovered some more differences in what multilibs are present after @ilg-ul re-did the build scripts :) .

TommyMurphyTM1234 commented 2 years ago

@TommyMurphyTM1234 Sorry should be "rv32ia variant" (singular) - that variant was present from this repo's 8.x (Windows/Linux) and 10.1.0 (Linux) toolchain releases. The Windows version of 10.1.0 did not have the rv32ia variant, which was why I filed this issue. But now... we seemed to have uncovered some more differences in what multilibs are present after @ilg-ul re-did the build scripts :) .

Just curious, have you tried installing a/the RISC-V bare metal toolchain (if any!) using your Linux distro's package manager to see what multilibs it supports? E.g. for Ubuntu...

https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=gcc-riscv64-unknown-elf&searchon=names

So something like

sudo apt install -y gcc-riscv64-unknown-elf

should do it.

Maybe it has what you need?

ilg-ul commented 2 years ago

I think that @manukuma-intel refers only to the xPack binaries, which in the past used different SiFive configurations, and, at a certain point, included the rv32ia variant.

I also remember that in some early builds I added to the SiFive list one extra configuration, suggested by Microsemi.

manukuma-intel commented 2 years ago

I've only used the xPack binaries that @ilg-ul creates so far. The Ubuntu package may have the rv32ia variant (I unfortunately don't have an Ubuntu box/container around to try this on), but in terms of having customers use a toolchain, I think this xPack repo provides great value - since it's not distribution or even operating system specific (i.e. usable on Windows and any recent Linux distribution).

I was trying to trace as to why the rv32ia lib was built in the past, probably something related to https://github.com/xpack-dev-tools/riscv-none-embed-gcc-xpack/blob/00f92d73aa7d753622b4b66a28c6c96c31383286/scripts/common-apps-functions-source.sh#L246 Was GCC_MULTILIB set previously?

ilg-ul commented 2 years ago

The bug fixed in 10.1.0-1.2 did exactly that, invoked the multi-lib generator for the Windows build.

The rv32ia lib was built in the past because it was defined in those releases, see how the GCC_MULTILIB definitions evolved:

In 10.1.0 I noticed that the SiFive list is ok, it required no further patches.

In 10.2.0 I assumed the same, but I did not notice that the non-c libs are missing entirely.

I did a dumb/combinatorial exercise and the following libs resulted:

rm32e-ilp32e--

rm32i-ilp32--
rv32im-ilp32--
rv32ia-ilp32--
rv32ima-ilp32--

rm32if-ilp32f--
rm32ifd-ilp32d--
rm32imf-ilp32f--
rm32imfd-ilp32d--
rm32imaf-ilp32f--
rm32imafd-ilp32d--

rm64i-lp64--
rv64im-lp64--
rv64ia-lp64--
rv64ima-lp64--

rm64if-lp64f--
rm64ifd-lp64d--
rm64imf-lp64f--
rm64imfd-lp64d--
rm64imaf-lp64f--
rm64imafd-lp64d--

I'm sure some are simply useless, but I can't tell for sure which.

ilg-ul commented 2 years ago

we seemed to have uncovered some more differences in what multilibs are present after @ilg-ul re-did the build scripts

The build scripts suffered major internal changes, but they are not relevant for this issue. Relevant is the bug fix that brought Windows inline with the other platforms.

And also relevant is the content of the GCC_MULTILIB, which was copied as-is from the SiFive official distribution.

I can do a new release and add the relevant missing multi-libs, but I would try to avoid adding useless combinations, since they increase the archive size and build time.

TommyMurphyTM1234 commented 2 years ago

I did a dumb/combinatorial exercise and the following libs resulted:

rm32e-ilp32e--

rm32i-ilp32--
rv32im-ilp32--
rv32ia-ilp32--
rv32ima-ilp32--

rm32if-ilp32f--
rm32ifd-ilp32d--
rm32imf-ilp32f--
rm32imfd-ilp32d--
rm32imaf-ilp32f--
rm32imafd-ilp32d--

rm64i-lp64--
rv64im-lp64--
rv64ia-lp64--
rv64ima-lp64--

rm64if-lp64f--
rm64ifd-lp64d--
rm64imf-lp64f--
rm64imfd-lp64d--
rm64imaf-lp64f--
rm64imafd-lp64d--

I'm sure some are simply useless, but I can't tell for sure which.

FWIW, if you omit rv32e from that list and add C variants of all others then the resulting list of 40 multilibs matches what Microchip/Microsemi SoftConsole bundles. I don't think that any of these are intrinsically useless. In the past, I seem to recall rv32 with D (double precision floating point) not being feasible for some reason (possibly some specific limitation in the Rocket Chip Chisel/Scala implementation?) but it's not "illegal" as far as I know. It was never clear to me why one would implement both D and F, but then I'm no floating point expert. In short, all of the above (plus the C variants) seem like plausible options to me.

ilg-ul commented 2 years ago

if you omit rv32e from that list and add C variants of all others then the resulting list of 40 multilibs matches what Microchip/Microsemi SoftConsole bundles.

Yeah, this only tells me that my dumb algorithm worked, but otherwise knowing this it is not particularly useful :-(

manukuma-intel commented 2 years ago

d = double-precision floating point = 64-bit floating point repr... which is present on rv32! I'm sure there's a use case, but that one seems more niche considering usage of 64-bit FP on a 32-bit CPU.

TommyMurphyTM1234 commented 2 years ago

d = double-precision floating point = 64-bit floating point repr... which is present on rv32! I'm sure there's a use case, but that one seems more niche considering usage of 64-bit FP on a 32-bit CPU.

Yes, I mention that above in my (now edited!) comment. It's an odd one but I think it's theoretically possible. No idea if any actual implementation exists though. :-)

ilg-ul commented 2 years ago

I did a (hopefully) smarter attempt, and used the SiFive list, but without c:

        rv32e-ilp32e-- \
        rv32ea-ilp32e-- \
        rv32em-ilp32e-- \
        rv32ema-ilp32e-- \

        rv32i-ilp32-- \
        rv32ia-ilp32-- \
        rv32im-ilp32-- \
        rv32ima-ilp32-- \
        rv32imf-ilp32f-- \
        rv32imaf-ilp32f-- \
        rv32imfd-ilp32d-- \
        rv32imafd-ilp32d-- \

        rv64i-lp64-- \
        rv64ia-lp64-- \
        rv64im-lp64-- \
        rv64ima-lp64-- \
        rv64imf-lp64f-- \
        rv64imaf-lp64f-- \
        rv64imfd-lp64d-- \
        rv64imafd-lp64d-- \

The list is still long, but hopefully makes more sense.

TommyMurphyTM1234 commented 2 years ago

I don't understand what you mean by "smarter"? That list is less comprehensive than the previous one.

ilg-ul commented 2 years ago

@manukuma-intel, the only combination you need is rv32ia-ilp32--?

ilg-ul commented 2 years ago

That list is less comprehensive than the previous one.

Yes that's the point, to shorten the list.

manukuma-intel commented 2 years ago

I like these rules, no need to add FP support if multiply is not available.

On the topic of size, it seems like these libs take up a majority of the space in a zip/binary release. (Windows .zip sizes)

So will enabling (C)ompressed and non-(C)ompressed double the size of the binary release? Another thing to note is that the 10.1.0 release was ~450 MB. Not sure if you care about keeping size down or if you have any tricks up your sleeve for this @ilg-ul .

TommyMurphyTM1234 commented 2 years ago

Not sure if you care about keeping size down or if you have any tricks up your sleeve for this @ilg-ul .

I think he does. :-)

https://github.com/xpack-dev-tools/riscv-none-embed-gcc-xpack/issues/11

manukuma-intel commented 2 years ago

image These were the only cores I could find on the core exchange which were 32-bit but (optionally) supported (D)ouble precision floating point.

Since the picture didn't turn out great, this is Nuclei's processors, https://www.nucleisys.com/product.php?site=n900 .

ilg-ul commented 2 years ago

these libs take up a majority of the space in a zip/binary release

That's correct.

enabling (C)ompressed and non-(C)ompressed double the size of the binary release?

The increase is roughly linear.

Adding the non-c entries will add 50% more libraries, since there are already those weird *zba_zbb* entries.

Not sure if you care about keeping size down or if you have any tricks up your sleeve for this

I do care about the size. I don't have a trick in the sleeve (yet), but I noticed that my libraries are larger than those in the SiFive archives, so there might be something to do to further reduce the size. (https://github.com/xpack-dev-tools/riscv-none-embed-gcc-xpack/issues/11).

TommyMurphyTM1234 commented 2 years ago

image These were the only cores I could find on the core exchange which were 32-bit but (optionally) supported (D)ouble precision floating point.

Don't forget simulators/emulators such as Spike, QEMU, Antmicro Renode, Imperas OVP etc.

ilg-ul commented 2 years ago

I'll see if I can find the trick to reduce the size, and, if so, add the new list, which will probably bring us back to 600-700 MB, which is a lot.

TommyMurphyTM1234 commented 2 years ago

I like these rules, no need to add FP support if multiply is not available.

Why? Aren't the integer multiply/divide M and floating point F/D[/Q] extensions independent?

manukuma-intel commented 2 years ago

Why? Aren't the integer multiply/divide M and floating point F/D[/Q] extensions independent?

Right, they are independent. But to me, it doesn't make sense to have a processor with floating point support but not integer multiply support. Maybe there is a specialized core somewhere which uses mostly floating point ops but not integer mul/div ops - but in the interest of reducing variants built, can it be left out?

ilg-ul commented 2 years ago

The first attempt to further strip the libraries was disappointing, the macOS archive came down from 636493201 to 618808300 bytes.

This means that adding all those multi-libs would probably increase the size to more than 900 MB, which is a lot.

TommyMurphyTM1234 commented 2 years ago

Why? Aren't the integer multiply/divide M and floating point F/D[/Q] extensions independent?

Right, they are independent. But to me, it doesn't make sense to have a processor with floating point support but not integer multiply support. Maybe there is a specialized core somewhere which uses mostly floating point ops but not integer mul/div ops - but in the interest of reducing variants built, can it be left out?

Yes, but you happen need rv32ia but who's to say that the next person doesn't need rv32iaf for some reason. ;-)

If it was me I'd do what I/we did with SoftConsole and support all 40 or so multilibs for maximum compatibility at the expense of a larger tools bundle.

But, as I mentioned earlier, when other extensions are finalised/ratified and added to the mix (e.g. B, J, K, V, RVWMO etc.) the number of possible multilibs will increase even further. :-(

Maybe some approach that allows for all multilibs to be built but only selected ones downloaded/installed would make more sense?

manukuma-intel commented 2 years ago

Fair point @TommyMurphyTM1234. Up to @ilg-ul in the end - but I'd say don't support the more niche variants, and if someone needs it, they can open an issue like this one :)?

Maybe some approach that allows for all multilibs to be built but only selected ones downloaded/installed would make more sense?

Not sure if GCC allows this kind of "dynamic" behavior. From just a quick try - deleting libs out of the riscv-none-embed\lib does not change what's printed by riscv-none-embed --print-multi-lib - that information seems to be compile time defined?

TommyMurphyTM1234 commented 2 years ago

Not sure if GCC allows this kind of "dynamic" behavior. From just a quick try - deleting libs out of the riscv-none-embed\lib does not change what's printed by riscv-none-embed --print-multi-lib - that information seems to be compile time defined?

You're correct. I was focusing on just the toolchain "payload" and forgot that knowledge of the bundled multilibs seems to be embedded into the toolchain at compile time.

ilg-ul commented 2 years ago

The current extra non-c libraries are:

          rv32e-ilp32e-- \
          rv32ea-ilp32e-- \
          rv32em-ilp32e-- \
          rv32ema-ilp32e-- \
          rv32i-ilp32-- \
          rv32ia-ilp32-- \
          rv32im-ilp32-- \
          rv32ima-ilp32-- \
          rv32if-ilp32f-- \
          rv32ifd-ilp32d-- \
          rv32imf-ilp32f-- \
          rv32imaf-ilp32f-- \
          rv32imfd-ilp32d-- \
          rv32imafd-ilp32d-- \
          rv64i-lp64-- \
          rv64ia-lp64-- \
          rv64im-lp64-- \
          rv64ima-lp64-- \
          rv64if-lp64f-- \
          rv64ifd-lp64d-- \
          rv64imf-lp64f-- \
          rv64imaf-lp64f-- \
          rv64imfd-lp64d-- \
          rv64imafd-lp64d-- \

I did a quick check of the list at https://riscv.org/exchange/cores-socs/ and I hope that most, if not all, are now supported.

We'll know in a few hours time how eficient the solution to reduce the size is. There are more than 60 multi-libs, and the build takes quite a while.

But even for the same multi-libs, the archive is anyway larger than SiFive's, since they do not rebuild the compiler specific libraries (like the libstdc++.a) with the nano configuration.

ilg-ul commented 2 years ago

The new macOS archive size is 681299366, which is large but not much larger than before.

For completeness (and avoid future iterations, as Tommy correctly noticed), I'll add the *iaf[d] combinations.

FYI, the main trick to reduce the size was to completely avoid -g when compiling the libraries; it is true that strip removes the debugging sections, but -g also adds lots of local symbols, which are not removed by strip.

I'll restart the full build for all platforms, and, if everything goes as planned, by tomorrow night East European time we'll have v10.2.0-1.2.

ilg-ul commented 2 years ago

For further discussions on the missing multi-libs, please use the separate [#12].