search

JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.54k stars 5.47k forks source link

Building Julia-1.7.1 in Debian-64 SID, errors are given #43624

Open domr2695 opened 2 years ago

domr2695 commented 2 years ago 
Downloaded git clone git://github.com/JuliaLang/julia.git

Created in /media/u1/s1/0/julia-1.7.1 Make.user file:
    USE_BINARYBUILDER=0
    prefix=/usr/local/
    -j=$(nproc)

cd /media/u1/s1/0/julia-1.7.1
make

When building Julia-1.7.1 in Debian-64 SID, errors are given:

/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:1724:41: error: argument 2 of type 'unsigned char[36]' with mismatched bound [-Werror=array-parameter=]
 1724 |                           unsigned char hash[36],
      |                           ~~~~~~~~~~~~~~^~~~~~~~
/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:684:63: note: previously declared as 'unsigned char *'
  684 | atic void ssl_calc_verify_tls( const mbedtls_ssl_context *, unsigned char *, size_t * );
      |                                                             ^~~~~~~~~~~~~~~

/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:1756:48: error: argument 2 of type 'unsigned char[32]' with mismatched bound [-Werror=array-parameter=]
 1756 |                                  unsigned char hash[32],
      |                                  ~~~~~~~~~~~~~~^~~~~~~~
/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:691:69: note: previously declared as 'unsigned char *'
  691 | oid ssl_calc_verify_tls_sha256( const mbedtls_ssl_context *,unsigned char *, size_t * );
      |                                                             ^~~~~~~~~~~~~~~

/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:1805:48: error: argument 2 of type 'unsigned char[48]' with mismatched bound [-Werror=array-parameter=]
 1805 |                                  unsigned char hash[48],
      |                                  ~~~~~~~~~~~~~~^~~~~~~~
/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:697:70: note: previously declared as 'unsigned char *'
  697 | id ssl_calc_verify_tls_sha384( const mbedtls_ssl_context *, unsigned char *, size_t * );
      |                                                             ^~~~~~~~~~~~~~~

/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c: In function 'ssl_calc_finished_tls_sha384':
/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:3260:5: error: 'mbedtls_sha512_finish_ret' accessing 64 bytes in a region of size 48 [-Werror=stringop-overflow=]
 3260 |     mbedtls_sha512_finish_ret( &sha512, padbuf );
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:3260:5: note: referencing argument 2 of type 'unsigned char *'
In file included from /media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/include/mbedtls/ssl_internal.h:51,
                 from /media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:41:
/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/include/mbedtls/sha512.h:145:5: note: in a call to function 'mbedtls_sha512_finish_ret'
  145 | int mbedtls_sha512_finish_ret( mbedtls_sha512_context *ctx,
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c: In function 'ssl_calc_verify_tls_sha384':
/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:1839:5: error: 'mbedtls_sha512_finish_ret' accessing 64 bytes in a region of size 48 [-Werror=stringop-overflow=]
 1839 |     mbedtls_sha512_finish_ret( &sha512, hash );
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:1839:5: note: referencing argument 2 of type 'unsigned char *'
In file included from /media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/include/mbedtls/ssl_internal.h:51,
                 from /media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/library/ssl_tls.c:41:
/media/u1/s1/0/julia-1.7.1/deps/srccache/mbedtls-2.24.0/include/mbedtls/sha512.h:145:5: note: in a call to function 'mbedtls_sha512_finish_ret'
  145 | int mbedtls_sha512_finish_ret( mbedtls_sha512_context *ctx,
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make[4]: *** [library/CMakeFiles/mbedtls.dir/build.make:202: library/CMakeFiles/mbedtls.dir/ssl_tls.c.o] Error 1
make[3]: *** [CMakeFiles/Makefile2:514: library/CMakeFiles/mbedtls.dir/all] Error 2
make[2]: *** [Makefile:146: all] Error 2
make[1]: *** [/media/u1/s1/0/julia-1.7.1/deps/mbedtls.mk:52: scratch/mbedtls-2.24.0/build-compiled] Error 2
make: *** [Makefile:60: julia-deps] Error 2
giordano commented 2 years ago

That's an error in mbedtls, not Julia itself. Here is a patch to fix the error: https://github.com/ARMmbed/mbedtls/pull/3848

DilumAluthge commented 2 years ago

Presumably, when doing source builds (i.e. USE_BINARYBUILDER=0), we should vendor that patch and automatically apply it?

domr2695 commented 2 years ago

Many thanks to giordano for the link: That's an error in mbedtls, not Julia itself. Here is a patch to fix the error: ARMmbed/mbedtls#3848

And it led to the link ARMmbed/mbedtls#7303

And it led to the idea that if you need to install a different version of the component from "./deps/Versions.make", then it is better to copy the "julia" folder from git to the working folder. Next, change the version of the component to "./deps/Versions.make", delete folders and files with the previous version of the component from "./deps" and run "make -j$(nproc)" in the root of the working folder (if a single-core processor, then just "make"), but the Internet must be for this. Running "make clean" or "make cleanall" does not help, because reassembling with a component of a different version mostly fails. The reasons for this are not yet clear.

The Julia build with mbedtls 2.28.0 or 2.16.12 goes fine, and with 2.24.0 it completes with errors.

Usage in "./deps/mbedtls.mk":

$(SRCCACHE)/$(MBEDTLS_SRC)/mbedtls-cmake-findpy.patch-applied: $(SRCCACHE)/$(MBEDTLS_SRC)/source-extracted

    # This patch merged upstream shortly after MBedTLS's 2.25.0 minor release, so chances
    # are it will be included at least in their next minor release (2.26.0?).
    cd $(SRCCACHE)/$(MBEDTLS_SRC) && \
    patch -p1 -f < $(SRCDIR)/patches/mbedtls-cmake-findpy.patch
    echo 1 > $@

patch "./deps/patches/mbedtls-cmake-findpy.patch" for mbedtls 2.28.0 or 2.16.12 leads to build errors. The last 3 lines should be commented out, then the building will pass without errors.

The developers of the Julia build system can be respected, since you can change the version of the component and this needs to be done only in one place and remove the old version from

    ./deps/checksums
    ./deps/scratch
    ./deps/srccache
mkitti commented 2 years ago

You may have just needed make distclean or make distcleanall

mkitti commented 2 years ago

Also note that in Yggdrasil we just staged Mbed TLS 2.28: https://github.com/JuliaPackaging/Yggdrasil/tree/master/M/MbedTLS/MbedTLS%402.28.0

and that the PR to update Julia for Mbed TLS 2.28 is here: https://github.com/JuliaLang/julia/pull/42311

domr2695 commented 2 years ago

Julia 1.6.4 used to be going fine. Julia 1.6.5 used to be going fine. Julia 1.7.0 used to be going fine. Julia 1.7.1 used to be going fine.

Now all versions give the same error when building:

JULIA usr/lib/julia/sys-o.a WARNING: Error during initialization of module LinearAlgebra: ErrorException("could not load library "/media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64.so" /media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64.so: ELF load command address/offset not properly aligned")

WARNING: Error during initialization of module LinearAlgebra: ErrorException("could not load library "/media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64.so" /media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64.so: ELF load command address/offset not properly aligned")

Generating REPL precompile statements... 36/36 Executing precompile statements... 1499/1536

Precompilation complete. Summary: Total ─────── 125.683829 seconds Generation ── 96.765243 seconds 76.991% Execution ─── 28.918586 seconds 23.009% LINK usr/lib/julia/sys.so WARNING: Error during initialization of module LinearAlgebra: ErrorException("could not load library "/media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64.so" /media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64.so: ELF load command address/offset not properly aligned")

Building with mbedtls 2.24.0 leads to errors. I had to switch to 2.28.0. The build from 2.16.12 is also going fine.

Using OpenBLAS 0.3.13 or 0.3.19 does not solve anything.

$ julia -tauto WARNING: Error during initialization of module LinearAlgebra: ErrorException("could not load library "/usr/local/bin/../lib/libopenblas64.so" /usr/local/bin/../lib/libopenblas64.so: адрес/смещение команды загрузки ELF не выровнено правильно") () | Documentation: https://docs.julialang.org () | () () | | |_ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ ` | | | | || | | | (| | | Version 1.7.1 (2021-12-22) / |_'|||_'_| | |/ |

julia> Threads.nthreads() 8

mkitti commented 2 years ago

Are you still using the same command to build? It sounds like something has gone wrong with your build environment as opposed to the Julia sources.

Just to make sure, clone a fresh repository.

domr2695 commented 2 years ago

From time to time I rename the folder with git to "julia.old" and run git clone https://github.com/JuliaLang/julia.git In between "clone" I update git by git fetch -ft https://github.com/JuliaLang/julia.git Then I copy the git folder to the working folder and copy the files to it ./Make.user: USE_BINARYBUILDER=0 prefix=/usr/local/

Added in first lines «./deps/checksums/mbedtls»: mbedtls-2.28.0.tar.gz/md5/d64054513df877458493dbb28e2935fa mbedtls-2.28.0.tar.gz/sha512/907867edf532ba3b099f4fb7ce31f5773ceceb072a8d067b1d830e879d541f92f401d64f13bbe6b4eb0845e58bb765d7d28896be414bb0fc7ac5b3876066be5f mbedtls-2.24.0.tar.gz/md5/9d1adcec4aa6729ae1dc56c3a24cb7d2 mbedtls-2.24.0.tar.gz/sha512/a51e80cedfa5c1772c79cba2dacd33f551516debf083803f7a5c1f4817c928e3bfb343fbe0c2e70ed591d0eba8fdc1bc46d11de7c3d12f50826de8f2f2ece279 mbedtls-2.16.12.tar.gz/md5/f3a7b041c43b35c883632a1773bf61a6 mbedtls-2.16.12.tar.gz/sha512/8d96d8cd906cc0999134320e4e1f550631426d166eab5da6e65469ee7286093810fcc6ac4bd5500ee55972d159f8bef7f9e53245f7f0eec72f72c35265b4313b

Replace some lines in «./media/u1/s1/0/julia-1.7.1/deps/Versions.make»: #MbedTLS MBEDTLS_VER := 2.16.12 # or 2.28.0 MBEDTLS_JLL_NAME := MbedTLS

Comments some lines in «./deps/mbedtls.mk»: # This patch merged upstream shortly after MBedTLS's 2.25.0 minor release, so chances # are it will be included at least in their next minor release (2.26.0?). #| cd $(SRCCACHE)/$(MBEDTLS_SRC) && \ #| patch -p1 -f < $(SRCDIR)/patches/mbedtls-cmake-findpy.patch #| echo 1 > $@ The last 3 lines should be commented out, then the building will pass without errors. And in the terminal window I run 'make -j$(nproc)`

A week ago Julia was going to be fine. And the previous versions, and 1.7.0, and 1.7.1. But now the installation of Julia 1.8-Dev is being completed: JULIA usr/lib/julia/sys-o.a WARNING: Error during initialization of module LinearAlgebra: ErrorException("could not load library "/media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64_.so" /media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64_.so: ELF load command address/offset not properly aligned") WARNING: Error during initialization of module LinearAlgebra: ErrorException("could not load library "/media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64_.so" /media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64_.so: ELF load command address/offset not properly aligned") Generating REPL precompile statements... 36/36 Executing precompile statements... 1314/1345 Precompilation complete. Summary: Total ─────── 118.162285 seconds Generation ── 87.270455 seconds 73.8564% Execution ─── 30.891830 seconds 26.1436% LINK usr/lib/julia/sys.so WARNING: Error during initialization of module LinearAlgebra: ErrorException("could not load library "/media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64_.so" /media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64_.so: ELF load command address/offset not properly aligned")

I have already reinstalled Debian-64 Sid 2 times, but nothing helps. The problem is "libopenblas64_.so ". Except for her, everything works.

mkitti commented 2 years ago

You may want to take a look at the BinaryBuilder script to see if that produces a better build: https://github.com/JuliaPackaging/Yggdrasil/blob/master/O/OpenBLAS/common.jl https://github.com/JuliaPackaging/Yggdrasil/blob/master/O/OpenBLAS/OpenBLAS32%400.3.13/build_tarballs.jl

vtjnash commented 2 years ago

That sounds like the patchelf-related issue we used to see on FreeBSD. @staticfloat any idea if glibc is adding new checks that are now stricter like FreeBSD was?

staticfloat commented 2 years ago

@domr2695 what version of glibc are you running? You can tell by looking at the target of the /lib64/ld-linux-x86-64.so.2 symlink:

$ ls -la /lib64/ld-linux-x86-64.so.2 
lrwxrwxrwx 1 root root 32 Dec 16  2020 /lib64/ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.31.so

So on my machine, I'm running glibc version 2.31.

giordano commented 2 years ago

Debian Sid should have glibc 2.33: https://packages.debian.org/source/sid/glibc, same as Arch Linux:

% /usr/lib/libc.so.6 
GNU C Library (GNU libc) release release version 2.33.
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 11.1.0.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://bugs.archlinux.org/>.
domr2695 commented 2 years ago

I have glibc 2.33-2.

If in "./Make.user": USE_BINARYBUILDER=1

Then everything is going fine. But, at the same time, all libraries are downloaded from the Internet, but from another (unknown to me) place and nothing is builded on the computer. But Julia starts and works fine.

And libopenblas64_.0.3.13.so 30.2 Mb. And there are 33 segments in the library.

libopenblas64_.0.3.13.so: file format elf64-x86-64 architecture: i386:x86-64, flags 0x00000150: HAS_SYMS, DYNAMIC, D_PAGED starting address 0x0000000000118000

If this library is replaced in the Julia assembly with a broken library, then Julia starts and works normally.

At the same time, if in "./Make.user": USE_BINARYBUILDER=0

Then all libraries are assembled from source texts on the computer.

And libopenblas64_.so 29.5 Mb, and it doesn't work.

LINK usr/lib/julia/says.so WARNING: Error during initialization of module LinearAlgebra: Error Exception("could not load library "/media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64.so" /media/u1/s1/0/julia-1.7.1/usr/bin/../lib/libopenblas64.so: ELF load command address/offset not properly aligned") libopenblas64_.so: file format elf64-x86-64 architecture: i386:x86-64, flags 0x00000150: HAS_SYMS, DYNAMIC, D_PAGED starting address 0x0000000000000000

And there are 26 segments in the library.

mkitti commented 2 years ago

If you look at https://github.com/JuliaLang/julia/blob/32abe71df48dcf32703f8bbcffa2f35a54152467/deps/openblas.mk#L210 you will see that it invokes bb-install. That's defined here:

https://github.com/JuliaLang/julia/blob/da28c17d9f9401ff4e9bd1b6e29e09ef735091d1/deps/tools/bb-install.mk

which says to download a tar ball from:

https://github.com/JuliaLang/julia/blob/da28c17d9f9401ff4e9bd1b6e29e09ef735091d1/deps/tools/bb-install.mk#L32

In the case of openblas that leads to one of the Assets here:

https://github.com/JuliaBinaryWrappers/OpenBLAS_jll.jl/releases

which leads to one of the build recipes here:

https://github.com/JuliaPackaging/Yggdrasil/tree/master/O/OpenBLAS

My suggestion is to try one of the build recipes above. In particular you might want to build 0.3.17 which is what is currently on the master branch: https://github.com/JuliaPackaging/Yggdrasil/blob/master/O/OpenBLAS/OpenBLAS32%400.3.17/build_tarballs.jl

That uses julia functions defined here:

https://github.com/JuliaPackaging/Yggdrasil/blob/master/O/OpenBLAS/common.jl

staticfloat commented 2 years ago

Yeah, by default the binaries are downloaded from here, we host a community build tree where binary dependencies for Julia packages and Julia itself are built for all the platforms Julia supports (Windows, macOS, Linux, FreeBSD, etc...) and hosted as tarballs on GitHub. If you're interested, you can actually find the precise build steps used to create the OpenBLAS binaries.

So if the Yggdrasil-provided binaries are working, that makes me think that there might be something fishy going on with your particular build toolchain. This is part of the reason why we prefer to use precompiled binaries; there is an enormous amount of surface area across all of these different platforms, and if users use a toolchian that behaves ever so slightly differently than most other toolchains, there can be compilation issues. And, in this case, it's in a dependency, rather than our own code!

I don't immediately see any reason why this would be happening..... especially since the steps taken on the local machine should be more or less the same as what happens in Yggdrasil.

@domr2695 can you build OpenBLAS manually? Do something like the following:

git clone -b v0.3.13 https://github.com/xianyi/OpenBLAS
cd OpenBLAS
export LDFLAGS="${LDFLAGS} '-Wl,-rpath,\$\$ORIGIN' -Wl,-z,origin"
make -j$(nproc) USE_THREAD=1 GEMM_MULTITHREADING_THRESHOLD=50 NO_AFFINITY=1 NO_STATIC=1 INTERFACE64=1 SYMBOLSUFFIX=64_ LIBPREFIX=libopenblas64_ NUM_THREADS=32 BINARY=64 DYNAMIC_ARCH=1 TARGET=GENERIC

Once that's done, it'll have a libopenblas64_.so file in the current directory. Try copying that over the one in your Julia build tree, and see if we still have a problem. My guess is that it may work, but if we then try to alter the binary (either through strip or patchelf or something similar) it may then refuse to work.

It would also be good to find out the version of patchelf and strip you have on your computer.

domr2695 commented 2 years ago

./julia-1.7.1/deps/Versions.make:

# Patchelf (we don't ship this or even use a JLL, we just always build it)

PATCHELF_VER := 0.13
./deps/scratch/patchelf-0.13/patchelf.spec:
Name: patchelf
Version: 0.13.20210805.a949ff2
Release: 1
URL: http://nixos.org/patchelf.html
$ strip -V
GNU strip (GNU Binutils for Debian) 2.37.50.20220106

Now (on 2022-01-13 22:17 GMT+3) when installing Julia 1.7.1 during: 'make -j$(nproc)` at the end is issued: make: *** [Makefile:60: julia-deps] Error 2

I had to add the "-i" parameter to line 60 in "./Makefile"`: julia-deps: | $(DIRS) $(build_datarootdir)/julia/base $(build_datarootdir)/julia/test @$(MAKE) $(QUIET_MAKE) -i -C $(BUILDROOT)/deps

Also in "./Make.user"

OPENBLAS_DYNAMIC_ARCH = 0"

leads to premature termination of the make build or to the issue of make at the end of the build.

I had to specify, because my processor architecture is SANDYBRIDGE:

OPENBLAS_USE_THREAD = 32 OPENBLAS_TARGET_ARCH=SANDYBRIDGE OPENBLAS_DYNAMIC_ARCH = 0 # to disable compiling multiple architectures in a single binary. OPENBLAS_NO_AVX2 = 1 # disables AVX2 instructions, allowing OpenBLAS to compile

At the end of the assembly , it began to be issued:

JULIA usr/lib/julia/sys-o.a WARNING: Error during initialization of module LinearAlgebra: ErrorException("could not load library "/media/u1/s1/0/julia-1.8.0-DEV/usr/bin/../lib/libopenblas64.so" /media/u1/s1/0/julia-1.8.0-DEV/usr/bin/../lib/libopenblas64.so: ELF load command address/offset not properly aligned") WARNING: Error during initialization of module LinearAlgebra: ErrorException("could not load library "/media/u1/s1/0/julia-1.8.0-DEV/usr/bin/../lib/libopenblas64.so" /media/u1/s1/0/julia-1.8.0-DEV/usr/bin/../lib/libopenblas64.so: ELF load command address/offset not properly aligned") Generating REPL precompile statements... 36/36 Executing precompile statements... 1314/1345 Precompilation complete. Summary: Total ─────── 123.209477 seconds Generation ── 91.286695 seconds 74.0906% Execution ─── 31.922782 seconds 25.9094% LINK usr/lib/julia/sys.so WARNING: Error during initialization of module LinearAlgebra: ErrorException("could not load library "/media/u1/s1/0/julia-1.8.0-DEV/usr/bin/../lib/libopenblas64.so" /media/u1/s1/0/julia-1.8.0-DEV/usr/bin/../lib/libopenblas64.so: ELF load command address/offset not properly aligned")

Solved the problem:


file: ./deps/srccache/curl-7.73.0/configure
------------------------------------------------------------------------------------
line 27197 for Julia 1.7.0, 1.7.1 for CURL 7.73.0
# Added a framing if for the metalink support condition, otherwise build
# fails with an error.

# Check whether --with-libmetalink was given.
if test ! ${without-libmetalink}
then :
      if test "${with_libmetalink+set}" = set; then :
            withval=$with_libmetalink; OPT_LIBMETALINK=$withval
      fi
fi
-------------------------------------------------------------------------------------
line 27197 for Julia 1.8.0-DEV for CURL 7.73.0
# Added a framing if for the metalink support condition, otherwise the build
# fails with an error.

# Check whether --with-libmetalink was given.
if test ! ${without-libmetalink}
then :
      if test ${with_libmetalink+y}
      then :
            withval=$with_libmetalink; as_fn_error $? "--with-libmetalink no longer works!" "$LINENO" 5
      fi
fi
--------------------------------------------------------------------------------
line 28871 for Julia 1.8.0-DEV CURL 7.81.0

# Added a framing if for the metalink support condition, otherwise the build
# fails with an error.

# Check whether --with-libmetalink was given.
if test ! ${without-libmetalink}
then :
      if test ${with_libmetalink+y}
      then :
            withval=$with_libmetalink; as_fn_error $? "--with-libmetalink no longer works!" "$LINENO" 5
      fi
fi
------------------------------------------------------------------------------------
Created for CURL 7.73.0
diff --normal -a ./origin/configure ./new/configure > curl-configure.patch
cp ./origin/configure ./patched/
patch -p1 ./patched/configure ./curl-configure.patch
Copy to "/deps/patches/ file ./curl-configure.patch":
27198,27199c27198,27202
< if test "${with_libmetalink+set}" = set; then :
< withval=$with_libmetalink; OPT_LIBMETALINK=$withval
---
> if test ! ${without-libmetalink}
> then :
> if test "${with_libmetalink+set}" = set; then :
> withval=$with_libmetalink; OPT_LIBMETALINK=$withval
> fi
------------------------------------------------------------------------------------

I draw your attention to the fact that in "./deps/Versions.make" for Julia 1.7.0-1.8.0-DEV is:

# mbedTLS
MBEDTLS_VER := 2.24.0
MBEDTLS_JLL_NAME := mbedTLS

using mbedTLS 2.24.0 results in the output of the messages that I gave in the first message, and the completion of make with an error message:

[mbedtls compilation error with GCC >= 11: 'mbedtls_sha512_finish_ret' accessing 64 bytes in a region of size 48 · Issue #7302 · micropython/micropython · GitHub

[lib/mbedtls: Switch to currently latest commit of LTS branch v2.16. by dpgeorge · Pull Request #7303 · micropython/micropython · GitHub)

In versions 2.16.12 and 2.28.0, this problem has been fixed, so they should be used.

And now it is enough to copy for each version of Julia: 1.7.0, 1.7.1, 1.8.0-DEV the corresponding files in the corresponding folder with the Julia source texts and run make -j$(nproc). All versions are installed normally and you no longer need to write the "-i" parameter in line 60 in "./Makefile"`:

julia-deps: | $(DIRS) $(build_datarootdir)/julia/base $(build_datarootdir)/julia/test @$(MAKE) $(QUIET_MAKE) -i -C $(BUILDROOT)/deps

Julia 1.7.0\
      Make.user:
USE_BINARYBUILDER=0
prefix=/usr/local/
OPENBLAS_USE_THREAD = 32
OPENBLAS_TARGET_ARCH = SANDYBRIDGE
# to disable compiling multiple architectures in a single binary.
OPENBLAS_DYNAMIC_ARCH = 0
# disables AVX2 instructions, allowing OpenBLAS to compile
OPENBLAS_NO_AVX2 = 1

    deps\
          checksums\
               mbedtls:
mbedtls-2.28.0.tar.gz/md5/d64054513df877458493dbb28e2935fa
mbedtls-2.28.0.tar.gz/sha512/907867edf532ba3b099f4fb7ce31f5773ceceb072a8d067b1d830e879d541f92f401d64f13bbe6b4eb0845e58bb765d7d28896be414bb0fc7ac5b3876066be5f

          patches\
              curl-configure.patch:
27198,27199c27198,27202
< if test "${with_libmetalink+set}" = set; then :
<   withval=$with_libmetalink; OPT_LIBMETALINK=$withval
---
> if test ! ${without-libmetalink}
> then :
>   if test "${with_libmetalink+set}" = set; then :
>     withval=$with_libmetalink; OPT_LIBMETALINK=$withval
>   fi

Versions.make:
# MbedTLS
MBEDTLS_VER := 2.28.0
MBEDTLS_JLL_NAME := MbedTLS 

curl.mk:
$(SRCCACHE)/curl-$(CURL_VER)/source-extracted: $(SRCCACHE)/curl-$(CURL_VER).tar.bz2
    $(JLCHECKSUM) $<
    cd $(dir $<) && $(TAR) jxf $(notdir $<)
    cp $(SRCDIR)/patches/config.sub $(SRCCACHE)/curl-$(CURL_VER)/config.sub
    touch -c $(SRCCACHE)/curl-$(CURL_VER)/configure # old target
    cd $(SRCCACHE)/curl-$(CURL_VER) && \
    patch -p1 -f ./configure $(SRCDIR)/patches/curl-configure.patch
    echo 1 > $@   

mbedtls.mk:
$(SRCCACHE)/$(MBEDTLS_SRC)/mbedtls-cmake-findpy.patch-applied: $(SRCCACHE)/$(MBEDTLS_SRC)/source-extracted
    # Apply workaround for CMake 3.18.2 bug (https://github.com/ARMmbed/mbedtls/pull/3691).
    # This patch merged upstream shortly after MBedTLS's 2.25.0 minor release, so chances
    # are it will be included at least in their next minor release (2.26.0?).
        # my comment next 3 lines 2021-01-09 10:38
    #| cd $(SRCCACHE)/$(MBEDTLS_SRC) && \
    #|  patch -p1 -f < $(SRCDIR)/patches/mbedtls-cmake-findpy.patch
    #| echo 1 > $@
------------------------------------------------------------------------------------
Julia 1.7.1\
      Make.user:
USE_BINARYBUILDER=0
prefix=/usr/local/
OPENBLAS_USE_THREAD = 32
OPENBLAS_TARGET_ARCH = SANDYBRIDGE
# to disable compiling multiple architectures in a single binary.
OPENBLAS_DYNAMIC_ARCH = 0
# disables AVX2 instructions, allowing OpenBLAS to compile
OPENBLAS_NO_AVX2 = 1

    deps\
          checksums\
               mbedtls:
mbedtls-2.28.0.tar.gz/md5/d64054513df877458493dbb28e2935fa
mbedtls-2.28.0.tar.gz/sha512/907867edf532ba3b099f4fb7ce31f5773ceceb072a8d067b1d830e879d541f92f401d64f13bbe6b4eb0845e58bb765d7d28896be414bb0fc7ac5b3876066be5f

          patches\
              curl-configure.patch:
27198,27199c27198,27202
< if test "${with_libmetalink+set}" = set; then :
<   withval=$with_libmetalink; OPT_LIBMETALINK=$withval
---
> if test ! ${without-libmetalink}
> then :
>   if test "${with_libmetalink+set}" = set; then :
>     withval=$with_libmetalink; OPT_LIBMETALINK=$withval
>   fi

Versions.make:
# MbedTLS
MBEDTLS_VER := 2.28.0
MBEDTLS_JLL_NAME := MbedTLS 

curl.mk:
$(SRCCACHE)/curl-$(CURL_VER)/source-extracted: $(SRCCACHE)/curl-$(CURL_VER).tar.bz2
    $(JLCHECKSUM) $<
    cd $(dir $<) && $(TAR) jxf $(notdir $<)
    cp $(SRCDIR)/patches/config.sub $(SRCCACHE)/curl-$(CURL_VER)/config.sub
    touch -c $(SRCCACHE)/curl-$(CURL_VER)/configure # old target
    cd $(SRCCACHE)/curl-$(CURL_VER) && \
    patch -p1 -f ./configure $(SRCDIR)/patches/curl-configure.patch
    echo 1 > $@   

mbedtls.mk:
$(SRCCACHE)/$(MBEDTLS_SRC)/mbedtls-cmake-findpy.patch-applied: $(SRCCACHE)/$(MBEDTLS_SRC)/source-extracted
    # Apply workaround for CMake 3.18.2 bug (https://github.com/ARMmbed/mbedtls/pull/3691).
    # This patch merged upstream shortly after MBedTLS's 2.25.0 minor release, so chances
    # are it will be included at least in their next minor release (2.26.0?).
        # my comment next 3 lines 2021-01-09 10:38
    #| cd $(SRCCACHE)/$(MBEDTLS_SRC) && \
    #|  patch -p1 -f < $(SRCDIR)/patches/mbedtls-cmake-findpy.patch
    #| echo 1 > $@
------------------------------------------------------------------------------------
Julia 1.8.0-DEV\
      Make.user:
USE_BINARYBUILDER=0
prefix=/usr/local/
OPENBLAS_USE_THREAD = 32
OPENBLAS_TARGET_ARCH = SANDYBRIDGE
# to disable compiling multiple architectures in a single binary.
OPENBLAS_DYNAMIC_ARCH = 0
# disables AVX2 instructions, allowing OpenBLAS to compile
OPENBLAS_NO_AVX2 = 1

    deps\
          checksums\
               mbedtls:
mbedtls-2.28.0.tar.gz/md5/d64054513df877458493dbb28e2935fa
mbedtls-2.28.0.tar.gz/sha512/907867edf532ba3b099f4fb7ce31f5773ceceb072a8d067b1d830e879d541f92f401d64f13bbe6b4eb0845e58bb765d7d28896be414bb0fc7ac5b3876066be5f

          patches\
              curl-configure.patch:
27198,27199c27198,27202
< if test "${with_libmetalink+set}" = set; then :
<   withval=$with_libmetalink; OPT_LIBMETALINK=$withval
---
> if test ! ${without-libmetalink}
> then :
>   if test "${with_libmetalink+set}" = set; then :
>     withval=$with_libmetalink; OPT_LIBMETALINK=$withval
>   fi

Versions.make:
# MbedTLS
MBEDTLS_VER := 2.28.0
MBEDTLS_JLL_NAME := MbedTLS 

curl.mk:
$(SRCCACHE)/curl-$(CURL_VER)/source-extracted: $(SRCCACHE)/curl-$(CURL_VER).tar.bz2
    $(JLCHECKSUM) $<
    cd $(dir $<) && $(TAR) jxf $(notdir $<)
    cp $(SRCDIR)/patches/config.sub $(SRCCACHE)/curl-$(CURL_VER)/config.sub
    touch -c $(SRCCACHE)/curl-$(CURL_VER)/configure # old target
    cd $(SRCCACHE)/curl-$(CURL_VER) && \
    patch -p1 -f ./configure $(SRCDIR)/patches/curl-configure.patch
    echo 1 > $@   

mbedtls.mk:
$(SRCCACHE)/$(MBEDTLS_SRC)/mbedtls-cmake-findpy.patch-applied: $(SRCCACHE)/$(MBEDTLS_SRC)/source-extracted
    # Apply workaround for CMake 3.18.2 bug (https://github.com/ARMmbed/mbedtls/pull/3691).
    # This patch merged upstream shortly after MBedTLS's 2.25.0 minor release, so chances
    # are it will be included at least in their next minor release (2.26.0?).
        # my comment next 3 lines 2021-01-09 10:38
    #| cd $(SRCCACHE)/$(MBEDTLS_SRC) && \
    #|  patch -p1 -f < $(SRCDIR)/patches/mbedtls-cmake-findpy.patch
    #| echo 1 > $@
vtjnash commented 2 years ago

That upgrade work was ongoing in https://github.com/JuliaLang/julia/pull/42311. It looks like mbedTLS violated semver, so it is being a bit more challenging to push through that particular update and needs some additional help to upgrade the additional dependencies.

domr2695 commented 2 years ago

Began to emerge: make[1]: [/media/u1/s1/0/julia-1.8.0-DEV/deps/p7zip.mk:6: /media/u1/s1/0/julia-1.8.0-DEV/deps/srccache/p7zip-16.2.0.tar.bz2] Error 28 make[1]: Waiting for unfinished jobs.…

Before that, there was often a message about exceeding TIMEOUT = 15 seconds. and the termination of the "make" execution.

/deps/p7zip.mk:6: $($(JL DOWNLOAD) $@ https://downloads.sourceforge.net/project/p7zip/p7zip/16.02/p7zip_16.02_src_all.tar.bz2

The $(JLDOWNLOAD) variable is defined in "./Make.inc": Line 402: JL DOWNLOAD := $(JULIA HOME)/tips/tools/jl download JLCHECKSUM := $(JULIAHOME)/deps/tools/jlchecksum

././deps/tools/jl download: Line 13: set: TIMEOUT=30 # seconds. Old value=15

Everything started downloading normally.

martin-frbg commented 2 years ago

The "ELF load command offset/address not properly aligned" error has now been traced to a bug in binutils 2.38, see https://github.com/xianyi/OpenBLAS/issues/3708 and subsequent https://sourceware.org/bugzilla/show_bug.cgi?id=29435 with a source patch for the binutils ld

mkitti commented 2 years ago

Given that the error has been traced to a 3rd party, can this issue now be closed?