search

jcmvbkbc / crosstool-NG

crosstool-NG with support for Xtensa
Other
108 stars 183 forks source link

Newlib overlay incorrectly packaged in lx106 overlay #42

Open stilor opened 7 years ago

stilor commented 7 years ago

An issue has been submitted in upstream crosstool-NG (https://github.com/crosstool-ng/crosstool-ng/issues/698) and while looking at it, I tried to build a configuration using an overlay, xtensa_lx106.tar. It failed.

The reason for the failure is that the newlib portion of the overlay is extracted into a wrong directory. The overlay tarball is extracted using the tar --strip-components=1 -xv -f /home/avn/work/ctng/xtensa/crosstool-ng/overlays/xtensa_lx106.tar newlib command. The contents of the archive is as follows:

binutils/
binutils/ld/
binutils/bfd/
binutils/bfd/xtensa-modules.c
binutils/include/
binutils/include/xtensa-config.h
gcc/
gcc/include/
gcc/include/xtensa-config.h
gdb/
gdb/gdb/
gdb/gdb/gdbserver/
gdb/gdb/gdbserver/xtensa-xtregs.c
gdb/gdb/regformats/
gdb/gdb/regformats/reg-xtensa.dat
gdb/gdb/xtensa-config.c
gdb/bfd/
gdb/bfd/xtensa-modules.c
gdb/include/
gdb/include/xtensa-config.h
newlib/
newlib/libc/
newlib/libc/sys/
newlib/libc/sys/xtensa/
newlib/libc/sys/xtensa/include/
newlib/libc/sys/xtensa/include/xtensa/
newlib/libc/sys/xtensa/include/xtensa/config/
newlib/libc/sys/xtensa/include/xtensa/config/core-isa.h

The newlib build fails because the newlib-2.0.0/newlib/libc/sys/xtensa/include/xtensa/config/core-isa.h file is missing. Accounting for the first stripped path component, this file should have been packaged as newlib/newlib/libc/sys/xtensa/include/xtensa/config/core-isa.h (i.e. one level deeper in the directory tree).

The build does not fail in this fork because one of the local patches (local-patches/newlib/2.0.0/0002-xtensa-cleanups.patch) creates this missing file.

The other two overlays in the overlays/ directory have the newlib packaged correctly. E.g. in esp32 overlay:

newlib/newlib/libc/sys/xtensa/include/xtensa/config/core-isa.h

With that fixed, newlib still fails though ("Newlib does not support CPU xtensa") - it looks like your local patches need to be ported upstream, to crosstool-NG and/or newlib.

davydnorris commented 7 years ago

Since I submitted the original crosstool issue, and I'm starting a general cleanup and refactor of esp-open-sdk, I'll take a look at this and make the changes. newlib is currently building fine for me so I'll change the overlay structure, remove the specific patch above and make sure it builds

davydnorris commented 7 years ago

OK just finished building the toolchain with no errors. Had to rebuild the tar overlay with the correct structure and edit the patch mentioned above to remove the part that created the missing file. I checked the patch content against core-isa.h and they were identical

Will do some more sanity checks but here are the files to support gcc 6.3.0 as well crosstool-NG.zip

jcmvbkbc commented 7 years ago

The newlib build fails because the newlib-2.0.0/newlib/libc/sys/xtensa/include/xtensa/config/core-isa.h file is missing. Accounting for the first stripped path component, this file should have been packaged as newlib/newlib/libc/sys/xtensa/include/xtensa/config/core-isa.h (i.e. one level deeper in the directory tree).

The build does not fail in this fork because one of the local patches (local-patches/newlib/2.0.0/0002-xtensa-cleanups.patch) creates this missing file.

Good catch, thanks for the report.

With that fixed, newlib still fails though ("Newlib does not support CPU xtensa")

In the newer crosstool-NG, right? That's because it no longer supports newlib-2.0.0 and tries to build a different newlib version.

stilor commented 7 years ago

In the newer crosstool-NG, right? That's because it no longer supports newlib-2.0.0 and tries to build a different newlib version.

Why, we do. Although 2.5.0 is the default, 2.0.0 is still available as a valid choice (and not even marked obsolete).

Rather, it is failing because we need the other parts of the local-patches that you have for newlib.

davydnorris commented 7 years ago

Rather, it is failing because we need the other parts of the local-patches that you have for newlib.

I think it failed simply because one of the patches expected to create the file, and when you adjusted it in the overlay it was already there.

I removed that specific patch and left the rest in the local patch file and it built fine.

I think the bigger question is whether we can change newlib support to remove the extensive local patching and integrate the xtensa changes directly into the newlib tree like other architectures. Has this been tried previously? If it were possible, it would have to make the job of supporting new versions much simpler.

stilor commented 7 years ago

As a side note - is there any kind of an official repository for these overlays? I.e., does it make sense for crosstool-NG to offer fetching them from some known location by default?

jcmvbkbc commented 7 years ago

Should be fixed with commit a43bb699167064de

jcmvbkbc commented 7 years ago

is there any kind of an official repository for these overlays

No, not yet/not that I know of. But I hope we'll get to it soon. Part of the problem with that is that only company that develops solution that has xtensa core gets an overlay for it, and it's their decision to share it or not.

does it make sense for crosstool-NG to offer fetching them from some known location by default

Maybe not from one location, but yes, fetching overlay from URL is very convenient.

davydnorris commented 7 years ago

I have fixed the archive and have tested it with the latest crosstool and it's fine.

Do you want my file? EDIT: Just realised I already attached it higher up. This included the patches as well as the overlays file

davydnorris commented 7 years ago

I was also thinking - I have been building esp-open-sdk with the very latest crosstool-ng for a while now, and selecting the latest gcc - and it is a simple matter of renaming the clone of this repo, and then copying the samples, overlays, and local patches into the crosstool tree.

Do you think it is still worth maintaining this clone of the whole crosstool tree, or should we perhaps look at bringing everything into the main crosstool repo? This repo/effort could then become the place where we could work on supporting xtensa ports of latest versions of newlib and the other tools, and in maintaining the download URLs for the overlays etc? Right now the newlib build relies on extensive patching which makes it very fragile in supporting later releases - would be awesome to have a way to more easily move to the latest lib

esp-open-sdk can then pull directly from crosstool-ng for its build as well.

jcmvbkbc commented 7 years ago

Do you think it is still worth maintaining this clone of the whole crosstool tree, or should we perhaps look at bringing everything into the main crosstool repo?

I think there's not much to bring there. newlib patch needs to go upstream, ok, I'm on it. Overlays should really be fetched from external URL instead of being a part of the crosstool-NG. Samples are easy.

esp-open-sdk can then pull directly from crosstool-ng for its build as well.

I'm not sure it's ready to abandon the default gcc-4.8.

davydnorris commented 7 years ago

I've been building and using esp-open-sdk with gcc 5.2.0 and now 6.3.0 for almost a year. The later versions do a lot more with LTO that is quite useful.

BTW, the force-l32 patch you sent me through the forums seems to be working fine in gcc 6.3.0 :-) and I am about to pull your latest patches and see if they apply to 6.3.0 and merge if necessary. I also see gcc 7.1.0 has just been added too, so 4.8.x is getting quite old.

davydnorris commented 7 years ago

And you are right - merging this work completely into upstream would be really quite simple, especially if pulling overlays from a URL is added.

I am happy to help out in any work to newlib if you would like.

someburner commented 6 years ago

@davydnorris

I'm interested in compiling 6.3.0/7.1.0 for esp-open-sdk - any tips on getting that to work? Seems like master is not compatible with esp-open-sdk, is there a specific commit you're using?

davydnorris commented 6 years ago

Been a while since I've recompiled so I'm a bit rusty but I did make rough notes.

I had to:

  1. Pull down the latest esp_open_sdk
  2. Pull down the latest crosstool-NG
  3. Check with jcmvbkbc/crosstool-NG to see if there are any recent patches (I don't think there are since my last recompile - it's not been touched since Jan'17)
  4. Rename esp_open_sdk/crosstool-NG to crosstool-NG.orig
  5. Copy the generic crosstool-NG into place
  6. move local-patches, overlays and samples extracted from the old one into the right path
  7. use crosstool-config-overrides to select the right versions
  8. Build and then strip the ROM functions from the standard libs

My current gcc is 6.4.0 - you'd have to make some adjustments to the patches for 7.1.0 but it's not too hard. Email me if you want a zip of the project I made containing the deltas