search

espressif / crosstool-NG

crosstool-NG with support for Xtensa
Other
121 stars 63 forks source link

expat-2.2.9.tar.gz Not Found #13

Closed iot49 closed 3 years ago

iot49 commented 3 years ago

Issue submission guidelines

See detailed guidelines. In short:

[EXTRA]    Verifying SHA512 checksum for 'mpc-1.1.0.tar.gz'
[EXTRA]    Retrieving 'expat-2.2.9'
[ERROR]    expat: download failed
[ERROR]   
[ERROR]  >>[0m
[ERROR]  >>  Build failed in step 'Retrieving needed toolchain components' tarballs'
[ERROR]  >>        called in step '(top-level)'
[ERROR]  >>[0m
[ERROR]  >>  Error happened in: CT_Abort[scripts/functions@487]
[ERROR]  >>        called from: CT_DoFetch[scripts/functions@2125]
[ERROR]  >>        called from: CT_PackageRun[scripts/functions@2085]
git clone https://github.com/espressif/crosstool-NG.git
cd crosstool-NG
git checkout esp-2021r1
git submodule update --init
./bootstrap && ./configure --enable-local && make
cd ~/esp-idf/crosstool-NG
./ct-ng xtensa-esp32-elf
./ct-ng build

I'm trying this since the official esp-idf does not (yet?) support aarch64.

chris-bishop commented 3 years ago

I had the same issue trying to build the ESP Toolchain from scratch on my RPi 4B+ Ubuntu 20.10 aarch64 dev box...

On initial attempt to configure and build with:

./ct-ng xtensa-esp32-elf  #// Configures AOK
./ct-ng build  #// Build Fails

I received the following error:

cbishop@cbishop-ubuntu:~/esp/crosstool-NG$ ./ct-ng build
[INFO ]  Performing some trivial sanity checks
[INFO ]  Build started 20210702.233915
[INFO ]  Building environment variables
[INFO ]  =================================================================
[INFO ]  Retrieving needed toolchain components' tarballs
[ERROR]   
[ERROR]  >>
[ERROR]  >>  Build failed in step 'Retrieving needed toolchain components' tarballs'
[ERROR]  >>        called in step '(top-level)'
[ERROR]  >>
[ERROR]  >>  Error happened in: do_expat_get[scripts/build/companion_libs/210-expat.sh@12]
[ERROR]  >>        called from: do_companion_libs_get[scripts/build/companion_libs.sh@15]
[ERROR]  >>        called from: main[scripts/crosstool-NG.sh@591]
[ERROR]  >>
[ERROR]  >>  For more info on this error, look at the file: 'build.log'
[ERROR]  >>  There is a list of known issues, some with workarounds, in:
[ERROR]  >>      'docs/B - Known issues.txt'
[ERROR]   
[ERROR]  (elapsed: 0:21.56)
[00:22] / gmake: *** [ct-ng:152: build] Error 1
cbishop@cbishop-ubuntu:~/esp/crosstool-NG$

After reviewing my resulting build.log file - I determined that the issue seems to root from a hard coded URL in the following build script's do_expat_get() method:

https://github.com/espressif/crosstool-NG/blob/xtensa-1.22.x/scripts/build/companion_libs/210-expat.sh#L11-L14

Note the hard coded sourceforge URL which when I paste in to my browser, results in a URL rewrite to new sourceforge URL structure:

do_expat_get() {
    CT_GetFile "expat-${CT_EXPAT_VERSION}" .tar.gz    \
               http://downloads.sourceforge.net/project/expat/expat/${CT_EXPAT_VERSION}
}

Note that there are two issues here - first, when attempt to nav to what would be the resulting URL as-is in browser:

http://downloads.sourceforge.net/project/expat/expat/2.1.0

https://sourceforge.net/projects/expat/files/expat/2.1.0/

second issue, note that version 2.1.0 has been deprecated due to vulnerabilities, and 2.3.0 is the minimum recommended version now...

For giggles and grins, I decided to manually update the offending build script hard coded URL to the current URL structure, and latest expat version (2.4.1 at time of comment) - so my resulting:

https://github.com/espressif/crosstool-NG/blob/xtensa-1.22.x/scripts/build/companion_libs/210-expat.sh#L13

now is:

do_expat_get() {
    CT_GetFile "expat-${CT_EXPAT_VERSION}" .tar.gz    \
               https://sourceforge.net/projects/expat/files/expat/${CT_EXPAT_VERSION}
}

my next attempt at building does NOT immediately fail like it did the first time...

I am actually trying this as I am commenting, so clearly my build will take some time to complete (if successful this time). I will report back later with final results, and submit a pull request with my suggested fix, should it work.

For reference, here are the particulars regarding my local development environment:

cbishop@cbishop-ubuntu:~$ uname -a
Linux cbishop-ubuntu 5.8.0-1029-raspi #32-Ubuntu SMP PREEMPT Thu Jun 17 09:56:18 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
cbishop@cbishop-ubuntu:~$ 
cbishop@cbishop-ubuntu:~$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.10 (Groovy Gorilla)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.10"
VERSION_ID="20.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=groovy
UBUNTU_CODENAME=groovy
cbishop@cbishop-ubuntu:~$ 
cbishop@cbishop-ubuntu:~$ cat /proc/cpuinfo 
processor   : 0
BogoMIPS    : 108.00
Features    : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part    : 0xd08
CPU revision    : 3

processor   : 1
BogoMIPS    : 108.00
Features    : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part    : 0xd08
CPU revision    : 3

processor   : 2
BogoMIPS    : 108.00
Features    : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part    : 0xd08
CPU revision    : 3

processor   : 3
BogoMIPS    : 108.00
Features    : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part    : 0xd08
CPU revision    : 3

Hardware    : BCM2835
Revision    : d03114
Serial      : 1000000019dc0f0c
Model       : Raspberry Pi 4 Model B Rev 1.4
cbishop@cbishop-ubuntu:~$
chris-bishop commented 3 years ago

Further, strangely it looks like the very issue I note above was already noted and fix suggested (and slightly better suggestion to use the expat official Git Repo vs fixing sourceforge URLs I might add) some time ago here:

https://github.com/jcmvbkbc/crosstool-NG/pull/53/commits/9ca18a216d317694c32e657cbade35661032dc51

However despite comments to the contrary - seems that fix must not have been merged into the xtensa-1.2.2.x branch...

That change IS however merged in to later release branches - seems the REAL root of my issue was that I was working with an old version of the Espressif esp32 docs...

https://docs.espressif.com/projects/esp-idf/en/release-v3.0/get-started/linux-setup-scratch.html

when I should have been working with this one:

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/linux-setup-scratch.html

So - for anyone else who bangs their heads against the wall with any kind of similar toolchain source download failures during build in your local environment - make sure you are using steps to build from the latest version of the docs before diving in to the code head first like is my default setting :)

Cheers!

iot49 commented 3 years ago

@chris-bishop Thanks for these suggestions!

Unfortunately I keep getting the same error:

[INFO ]  Retrieving needed toolchain components' tarballs
[EXTRA]    Retrieving 'expat-2.2.9'
[ERROR]    expat: download failed

The full log is below and has some additional information, not sure if due to a difference between setups or if your report omits a few lines.

I've also noticed that the espressif version of crosstool-NG for 210-expat.sh (https://github.com/espressif/crosstool-NG/blob/esp-1.23.x/scripts/build/companion_libs/210-expat.sh) is just

do_expat_get() {
    CT_Fetch EXPAT
}

Is EXPAT a macro that gets expanded somehow? Perhaps the problem is in that expansion? I wouldn't know why, as I have deleted ~/esp and redownloaded everything.

Many thanks for any further pointers you may be able to provide!

[INFO ]  Performing some trivial sanity checks
[INFO ]  Build started 20210705.123448
[INFO ]  Building environment variables
[EXTRA]  Preparing working directories
[EXTRA]  Installing user-supplied crosstool-NG configuration
[EXTRA]  =================================================================
[EXTRA]  Dumping internal crosstool-NG configuration
[EXTRA]    Building a toolchain for:
[EXTRA]      build  = aarch64-unknown-linux-gnu
[EXTRA]      host   = aarch64-unknown-linux-gnu
[EXTRA]      target = xtensa-esp32-elf
[EXTRA]  Dumping internal crosstool-NG configuration: done in 0.42s (at 00:06)
[INFO ]  =================================================================
[INFO ]  Retrieving needed toolchain components' tarballs
[EXTRA]    Retrieving 'expat-2.2.9'
[ERROR]    expat: download failed
[ERROR]   
[ERROR]  >>  Build failed in step 'Retrieving needed toolchain components' tarballs'
[ERROR]  >>        called in step '(top-level)'
[ERROR]  >>  Error happened in: CT_Abort[scripts/functions@487]
[ERROR]  >>        called from: CT_DoFetch[scripts/functions@2125]
[ERROR]  >>        called from: CT_PackageRun[scripts/functions@2085]
[ERROR]  >>        called from: CT_Fetch[scripts/functions@2196]
[ERROR]  >>        called from: do_expat_get[scripts/build/companion_libs/210-expat.sh@12]
[ERROR]  >>        called from: do_companion_libs_get[scripts/build/companion_libs.sh@15]
[ERROR]  >>        called from: main[scripts/crosstool-NG.sh@647]
[ERROR]  >>  For more info on this error, look at the file: 'build.log'
[ERROR]  >>  There is a list of known issues, some with workarounds, in:
[ERROR]  >>      https://crosstool-ng.github.io/docs/known-issues/
[ERROR]  >>
[ERROR]  >> NOTE: Your configuration includes features marked EXPERIMENTAL.
[ERROR]  >> Before submitting a bug report, try to reproduce it without enabling
[ERROR]  >> any experimental features. Otherwise, you'll need to debug it
[ERROR]  >> and present an explanation why it is a bug in crosstool-NG - or
[ERROR]  >> preferably, a fix.
[ERROR]  >>  If you feel this is a bug in crosstool-NG, report it at:
[ERROR]  >>      https://github.com/crosstool-ng/crosstool-ng/issues/
[ERROR]  >>  Make sure your report includes all the information pertinent to this issue.
[ERROR]  >>  Read the bug reporting guidelines here:
[ERROR]  >>      http://crosstool-ng.github.io/support/
[ERROR]   
[ERROR]  (elapsed: 0:05.64)
wieck commented 3 years ago

I think retrieving expat-2.2.9 fails primarily because sourceforge has renamed the tarball due to vulnerabilities and recommends using 2.4.1 instead.

iot49 commented 3 years ago

I managed to build the tools by first downloading expat-2.2.9 tar to ~/src (don't ask me where this path comes from), giving it the correct name, and then run the install. Later ran into other issues and ultimately just decided to wait for the official espressif build. Hopefully it's coming soon (they promised spring, and for osx/aarch64 a version is already available it seems).

On Tue, Jul 13, 2021 at 2:16 PM Jan Wieck @.***> wrote:

I think retrieving expat-2.2.9 fails primarily because sourceforge has renamed the tarbal due to vulnerabilities and recommends using 2.4.1 instead.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/espressif/crosstool-NG/issues/13#issuecomment-879408760, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATHYSZIPHLGJET4IF7CJZLTTXSUKXANCNFSM47LOXLPQ .

wieck commented 3 years ago

@iot49 my build using expat-2.4.1 is still under way. I'm building this on CentOS 7.9.2009 (AltArch). Let's see how far I can get.

There were 4 packages that the process is pulling out of git (binutils, gcc, newlib and gdb). All of them report in the log that they switched to a different commit hash. This also caused problems when I had to restart the build. It resulted in a symlink in .build/tarballs pointing to itself, then later on unpacking it fails (of course).

wieck commented 3 years ago

@iot49 update: the build with expat-2.4.1 succeeded. However, this is only one step further. Later on installing the esp-idf fails because there are no versions for linux-arm64 of the ulp and openocd components available either and linux-armel ones don't work because they are dynamically linked (statically linked armel 32-bit code seems to run on a PI4).

chris-bishop commented 3 years ago

@wieck

There were some changes between build versions to the

do_expat_get() {
    CT_Fetch EXPAT
}

Functions and others like it, where in the older version branch I initially cloned locally (blindly following along with Espressif Docs...) the dependency mirror URLs were all hardcoded, vs the current implementation which does expansion and interpolation as you note.

If it rains again all weekend and I am stuck in my bat cave I plan to take another crack at building a proper cross compiler for the esp toolchain on my x86-x64 laptop targeting my aarch64 RPi4B+ Ubuntu 21.04 box, because as you have also noted unfortunately some of the dependency in toolchain still will not properly install on aarch64.

I'll gladly share my experience here, as well as working toolchain cross compiler code and config should I be successful.

Cheers!

chris-bishop commented 3 years ago

@wieck

I also forgot to mention these lines in the stdout log snippet you shared:

...
[ERROR]  >>  For more info on this error, look at the file: 'build.log'
[ERROR]  >>  There is a list of known issues, some with workarounds, in:
[ERROR]  >>      https://crosstool-ng.github.io/docs/known-issues/
[ERROR]  >>
...

Please forgive me if this is a "blinding flash of the obvious", but whenever you run in to issues I suggest taking a look in the ./build.log file as it generally contains more verbosity and deeper stack traces.

wieck commented 3 years ago

@chris-bishop yes, that build.log is helpful to get to the actual build problems.

As said the build for the xtensa-esp32-elf cross compiler succeeded with using expat 2.4.1. But the other components (like the ultra low power compiler) are also needed for a complete esp-idf, and I am stuck on those now.

Checked out https://github.com/espressif/binutils-esp32ulp.git tag v2.28.51-esp-20191205. But that fails with some really strange errors like libbfd.h:532:4: error: cast between incompatible function types from ‘long int (*)(bfd *)’ {aka ‘long int (*)(struct bfd *)’} to ‘long int (*)(bfd *, long int, asymbol **, long int, asymbol **, asymbol **)’ {aka ‘long int (*)(struct bfd *, long int, struct bfd_symbol **, long int, struct bfd_symbol **, struct bfd_symbol **)’} [-Werror=cast-function-type] ((long (*) (bfd *, long, asymbol **, long, asymbol **, asymbol **)) _bfd_n1) ^ bfd.h:7839:3: note: in expansion of macro ‘_bfd_nodynamic_get_synthetic_symtab’ NAME##_get_synthetic_symtab, \ ^~~~ srec.c:1394:3: note: in expansion of macro ‘BFD_JUMP_TABLE_DYNAMIC’ BFD_JUMP_TABLE_DYNAMIC (_bfd_nodynamic), ^~~~~~~~~~~~~~~~~~~~~~

If I find the time over the weekend I might try to create a different build environment for it. Like Raspbian.

chris-bishop commented 3 years ago

@wieck

Ugh, yea those look very similar to issues I kept running in to when I last tried cross compiling for my RPi.

Given all the "math" going on there, most likely more of the same juxtaposition of some toolchain submodules compiling without any issue and others which speak a completely different language than our ARM SoCs 😝

I'm hoping to get some time to try again soon. Maybe between the two of us we can submit some Pull Requests that may help others down the line.

Cheers!

PiotrParysek commented 3 years ago

Servus,

I have found a "dirty" workaround for this problem: In .config file change the CT_EXPAT_VERSION line to: CT_EXPAT_VERSION="2.4.1" the solution is "dirty" because there is/may appear a warning: [WARN ] Not verifying 'expat-2.4.1.tar.xz': digest missing Due to the message displayed within version 2.3.0: expat-2.3.0-RENAMED-VULNERABLE-PLEASE-USE-2.4.1-INSTEAD I have inserted the newest version - 2.4.1

With this solution, I have successfully compiled the tool.

Saludos.

PS. If you compile tool on Raspberry Pi 3+/4 you should also change the value of CT_EXTRA_CFLAGS_FOR_HOST flag to: CT_EXTRA_CFLAGS_FOR_HOST="-march=armv7-a -mfloat-abi=hard -mtune=cortex-a5"

wieck commented 3 years ago

@PiotrParysek

this should be the proper way to fix it:

https://github.com/espressif/crosstool-NG/pull/16

wieck commented 3 years ago

@PiotrParysek

PS. If you compile tool on Raspberry Pi 3+/4 you should also change the value of CT_EXTRA_CFLAGS_FOR_HOST flag to: CT_EXTRA_CFLAGS_FOR_HOST="-march=armv7-a -mfloat-abi=hard -mtune=cortex-a5"

That depends on the distro you are using on the PI. Some (like Raspbian) aim for maximum compatibility with whatever and allow to run a lot, with sometimes interesting results. Others (like CentOS) aim for maximum correctness and almost nothing will work, unless it was explicitly compiled for it.

wieck commented 3 years ago

@chris-bishop

If it rains again all weekend and I am stuck in my bat cave I plan to take another crack at building a proper cross compiler for the esp toolchain on my x86-x64 laptop targeting my aarch64 RPi4B+ Ubuntu 21.04 box, because as you have also noted unfortunately some of the dependency in toolchain still will not properly install on aarch64.

FYI, I performed a canadian build of this and it worked. But I had to throw 32 cores at it to get the build time down from 2.5 hours on a PI4 to 1 hour on a dual-Xeon-Silver. Very few people have this sort of resources, so I'm not going to continue with that.

What I am aiming for now is to put the build of each IDF component into a docker/podman container, then extract the resulting binaries. I often build X86-64 RPMs that way, so nothing new there. If they are linked static they should work on any PI3/4. At least that is my current theory.