Open mkitti opened 2 years ago
There is currently a comment referencing ucrt
here:
https://github.com/JuliaLang/julia/blob/8bdb04526d9ca78f0efaff871a5859bd66ed6c32/src/jlapi.c#L505-L516
It would be interesting to hear if someone can get --with-default-msvcrt=ucrt
working. But it appears to require rebuilding gcc from scratch, so it may be rather a challenge.
But it appears to require rebuilding gcc from scratch, so it may be rather a challenge.
It looks like the MSYS2 UCRT64 environment has a gcc, so perhaps they have already addressed that part? https://www.msys2.org/docs/environments/
We have not supported msys2 in a couple years, but that is probably a good starting point
Migrating MinGW to UCRT requires 5 steps on Arch linux (should be similar for Cygwin and other Linux):
mingw-w64-gcc
and mingw-w64-binutils
to bootstrapmingw-w64-header
by adding the configure option --with-default-msvcrt=ucrt
, build and installmingw-w64-crt
by adding the configure option --with-default-msvcrt=ucrt
, build and installmingw-w64-winpthreads
and installmingw-w64-gcc
and installThen we can use the new mingw-w64-gcc
to cross compile and the produced executable links to UCRT.
UCRT is definitely better than old msvcrt because it supports more modern C features, so it can reduce a lot of cross-platform hacks especially for Windows.
However, I am afraid that the whole Windows toolchain and all Windows artifacts in BinaryBuilder
ecosystem need to be rebuilt to use UCRT. It should be regarded as a MAJOR incompatible ABI change for Windows platform.
The near term objective is just to get Julia to build with UCRT.
How should we deal with prebuilt third-party dependency? They are built by BinaryBuilder and should switch to UCRT too.
Yes, I understand. I'm not suggesting that we start distributing Julia linked to UCRT. I'm just suggesting that the first goal is to develop the ability to build Julia linked to UCRT.
On Linux, BinaryBuilder currently supports both glibc and musl, so perhaps we eventually develop a similar capability.
I agree with you. UCRT can be added as a new build configuration on Windows.
We need a new triplet for BinaryBuilder
.
Maybe: x86_64-w64-ucrt64
which mean x86_64
on windows
with gcc + ucrt
ref: Environments - MSYS2
We need a new triplet for
BinaryBuilder
.Maybe:
x86_64-w64-ucrt64
which meanx86_64
onwindows
withgcc + ucrt
ref: Environments - MSYS2
Yes, it is incompatible with old binary artifacts depending on msvcrt.dll.
Another thing is that we may have to fix the MinGW/MSYS2 compilation environment first.
It looks like setting up a cygwin+UCRT
compile environment is a bit more complicated.
Another thing is that we may have to fix the MinGW/MSYS2 compilation environment first. It looks like setting up a
cygwin+UCRT
compile environment is a bit more complicated.
I think cygwin + ucrt
is easier because once we obtain a ucrt-enabled cross gcc toolchain in cygwin and replace current toolchain like I mentioned above, we can compile Julia in cygwin as usual (maybe without any code change).
I will give it a try when I have some spare time this month. I am struggling to build cygwin packages with cygport last week.
(You can also use a ucrt-enabled cross gcc toolchain on Linux to compile Julia.)
According to https://github.com/msys2/MSYS2-packages/issues/3040
Msys2 does not encourage the usage of cross gcc toolchain, which does not align with the build system of Julia designed around cross compilation. Therefore, supporting Msys2 would be great, but it may require more effort than cygwin + ucrt
approach.
Patch that enabled building under MSys2 UCRT64.
From 7031e5e84d5d7e0ba088f88a1a4d10a9527f4a30 Mon Sep 17 00:00:00 2001
From: Tim Stahlhut <stahta01@gmail.com>
Date: Mon, 20 Feb 2023 19:01:18 -0500
Subject: ucrt: Replace "msvcrt" with "ucrtbase"
Replace "-lmsvcrt" with "-lucrtbase" in base/linking.jl
Set CRTDLL_BASENAME to "ucrtbase".
Replace libmsvcrt.a with libucrtbase.a
---
Makefile | 2 +-
base/linking.jl | 2 +-
src/dlload.c | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/Makefile b/Makefile
index 8010088bd1..2358fb405c 100644
--- a/Makefile
+++ b/Makefile
@@ -261,7 +261,7 @@ endif
# These are files from mingw32 and required for creating shared libraries like our caches.
-$(INSTALL_M) $(build_libdir)/libgcc_s.a $(DESTDIR)$(libdir)/
-$(INSTALL_M) $(build_libdir)/libgcc.a $(DESTDIR)$(libdir)/
- -$(INSTALL_M) $(build_libdir)/libmsvcrt.a $(DESTDIR)$(libdir)/
+ -$(INSTALL_M) $(build_libdir)/libucrtbase.a $(DESTDIR)$(libdir)/
else
# Copy over .dSYM directories directly for Darwin
diff --git a/base/linking.jl b/base/linking.jl
index 288279347f..957a7f8820 100644
--- a/base/linking.jl
+++ b/base/linking.jl
@@ -133,7 +133,7 @@ function link_image_cmd(path, out)
SHLIBDIR = "-L$(shlibdir())"
LIBS = is_debug() ? ("-ljulia-debug", "-ljulia-internal-debug") : ("-ljulia", "-ljulia-internal")
@static if Sys.iswindows()
- LIBS = (LIBS..., "-lopenlibm", "-lssp", "-lgcc_s", "-lgcc", "-lmsvcrt")
+ LIBS = (LIBS..., "-lopenlibm", "-lssp", "-lgcc_s", "-lgcc", "-lucrtbase")
end
V = VERBOSE[] ? "--verbose" : ""
diff --git a/src/dlload.c b/src/dlload.c
index dd5d75da31..5ab350f31c 100644
--- a/src/dlload.c
+++ b/src/dlload.c
@@ -60,7 +60,7 @@ static int endswith_extension(const char *path) JL_NOTSAFEPOINT
}
#ifdef _OS_WINDOWS_
-#define CRTDLL_BASENAME "msvcrt"
+#define CRTDLL_BASENAME "ucrtbase"
JL_DLLEXPORT const char *jl_crtdll_basename = CRTDLL_BASENAME;
const char *jl_crtdll_name = CRTDLL_BASENAME ".dll";
--
Needed a version of PR 48640
$ cat Make.user
DEPS_GIT = 1
USE_BINARYBUILDER_CSL = 0
USE_SYSTEM_CSL = 1
USE_BINARYBUILDER_LLVM = 0
USE_BINARYBUILDER_GMP = 0
USE_BINARYBUILDER_MPFR = 0
Edit: Had test failures in file, misc, Dates/io, and Mmap that looked like a common cause might exist. Also, had failures in SparseArrays/cholmod along with the normal REPL, InteractiveUtils, and FileWatching failures.
Tim S.
It seems that only msys2 currently has support for ucrt (why don't they upstream it?) so we can't support it in binary builder and change the ecosystem to it. USE_BINARYBUILDER=0
might help, if you are willing to work through all of the dependencies and make sure they get build correctly.
Currently trying USE_BINARYBUILDER=0 and USE_SYSTEM_CSL = 1 with the below because of build errors; most already know and reported by others. Edit in issue 45645 Edit2: LIBSUITESPARSE error is because it failed to find BLASTRAMPOLINE.
echo 'USE_BINARYBUILDER_BLASTRAMPOLINE = 1' >> Make.user && \
echo 'USE_BINARYBUILDER_LIBSUITESPARSE = 1' >> Make.user && \
echo 'USE_BINARYBUILDER_ZLIB = 1' >> Make.user && \
echo 'USE_BINARYBUILDER_P7ZIP = 1' >> Make.user && \
It seems that only msys2 currently has support for ucrt (why don't they upstream it?) so we can't support it in binary builder and change the ecosystem to it.
USE_BINARYBUILDER=0
might help, if you are willing to work through all of the dependencies and make sure they get build correctly.
The support for ucrt already exists in upstream mingw64, it is turned on by rebuilding the whole toolchain from scratch with --with-default-msvcrt=ucrt
passed to mingw-w64-headers
and mingw-w64-crt
when configuring. (https://github.com/msys2/MINGW-packages/issues/6901)
A viable solution is to create a new triplet for ucrt just like musl vs glibc on Linux.
Apparently to add a new triplet we should send a pull request to modify this file: https://github.com/JuliaLang/julia/blob/e3d366f1966595ba737220df49e220610823b331/base/binaryplatforms.jl#L640-L644
Another question about this platform, are we sticking with gcc or is anyone thinking about clang?
My inclination would be to stick with gcc at the moment. Thus the UCRT64 msys2 environment looks like the target.
Apparently to add a new triplet we should send a pull request to modify this file:
Another question about this platform, are we sticking with gcc or is anyone thinking about clang?
My inclination would be to stick with gcc at the moment. Thus the UCRT64 msys2 environment looks like the target.
CLANG64 gives the below output
$ clang -dumpmachine
x86_64-w64-windows-gnu
After building most deps from source tests fail below plus the normal three of InteractiveUtils, FileWatching, and REPL.
Error in testset file:
Error in testset misc:
Error in testset SparseArrays/cholmod:
Error in testset Dates/io:
Error in testset LibSSH2_jll:
Error in testset LibGit2_jll:
Error in testset Mmap:
Error in testset PCRE2_jll:
Error in testset LinearAlgebra/pinv
I am building Cygwin git master branch with llvm copy patch using USE_BINARYBUILDER=0. And, I thought I would look for UCRT import libs I found under UCRT64 and I found them
H:\cygwin64\usr\x86_64-w64-mingw32\sys-root\mingw\lib
libucrt.a
libucrtapp.a
libucrtbase.a
I picked libucrtbase.a to build UCRT64 because google searches implied it might be more correct. One of the other two might be better.
Once, the current Cygwin build finishes could take a day or so. Esp. if I need to re-start it. I plan to see if the same ucrt patch results in a good build using Cygwin.
Edit: Turns out linking to ucrt with an Toolchain not designed to use UCRT results in linker errors most often and when it does not seems to have run-time errors.
Tim S.
xref:
Background: We need high-quality implementations of strtof
and strtod
.
The implementation in msvcrt is of poor quality, and the corresponding export functions are available in UCRT.
strtod directly from msvcrt
This one is significantly worse quality, and does not conform to the current C standards.
Originally posted by
@vtjnash
in https://github.com/JuliaLang/julia/issues/49699#issuecomment-1545785317
POC(Proof Of Concept) for compiling from source code using the UCRT64 environment.
MINGW64_NT-10.0-22631 A309-Y9000P 3.4.10.x86_64 2023-11-30 06:09 UTC x86_64 Msys
julia souce with patchs: based on master, https://github.com/inkydragon/julia/tree/cyhan/win-ucrt64
diff: https://github.com/JuliaLang/julia/compare/1b183b93...inkydragon:julia:cyhan/win-ucrt64
deps:
pacman -S cmake diffutils git m4 make patch tar p7zip curl python
pacman -S mingw-w64-ucrt-x86_64-gcc
pacman -S unzip mingw-w64-ucrt-x86_64-gcc-fortran mingw-w64-ucrt-x86_64-nasm
# pacman -S mingw-w64-ucrt-x86_64-cmake
Make.user
# https://github.com/JuliaLang/julia/issues/51740
override HAVE_SSP := 0
# new flag
USE_WIN_UCRT := 1
USE_BINARYBUILDER=0
# TODO: build p7zip from source
USE_BINARYBUILDER_P7ZIP:=1
make -C deps/ extract
I get tar
unpacking errors when building libgit2
and llvm
.
You need to repeat the unpacking operation manually and create the corresponding source-extracted
file
Then you can continue to build.
make -C deps/ -j
make -j
NOTE: build llvm takes a loooong time, for me, it's 30~40 min.
If you use ldd to check the shared libraries...
EXCEPTION_ACCESS_VIOLATION when test threads
.
I can reproduce this error by: make -C test/ threads
Need for further analysis.
threads (1) | started at 2023-12-31T22:41:17.415
Test Summary: | Pass Total Time
threads_exec.jl with JULIA_NUM_THREADS == 1 | 40353 40353 26.1s
Test Summary: | Pass Total Time
threads_exec.jl with JULIA_NUM_THREADS == 2 | 48759 48759 24.2s
Test Summary: | Pass Total Time
threads_exec.jl with JULIA_NUM_THREADS == 4 | 47656 47656 22.8s
Test Summary: | Pass Total Time
threads_exec.jl with JULIA_NUM_THREADS == 4 | 46870 46870 25.3s
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffa13471120 -- ┌ Error: A "spawn and wait lots of tasks" test failed
│ n = 2000000
│ proc.exitcode = 1
│ proc.termsignal = 15
│ success(proc) = false
│ timeout = true
└ @ Main.Test61Main_threads D:\jl\julia\test\threads.jl:313
threads (1) | 228.09 | 0.03 | 0.0 | 113.49 | 467.34
Please add "MSYS2" label.
This UCRT seem like good to support (not just the float parsing bug, that has a better/easier workaround), it's the future, for Windows10+, i.e. all supported Julia platforms, available for Vista+ (also latest version still available there?), though I see "Runtime library support for Windows XP is no longer available in the latest Visual C++ Redistributable for Visual Studio 2015, 2017, 2019 and 2022. The last redistributable to support Windows XP is version 16.7 (file version 14.27.29114.0)." that I think concerns no one...
On Microsoft Windows, there is a new C runtime "libc" called "ucrt". Many of the changes support ISO C99 conformance.. https://devblogs.microsoft.com/cppblog/introducing-the-universal-crt/ https://docs.microsoft.com/en-us/cpp/porting/upgrade-your-code-to-the-universal-crt?view=msvc-170
UCRT deployment was backported to Windows XP, although Windows XP is no longer supported: https://docs.microsoft.com/en-us/cpp/windows/universal-crt-deployment?view=msvc-170
MSYS2 provides a UCRT64 build environment: https://www.msys2.org/docs/environments/
MSYS2 currently has a julia package, with many packages: https://github.com/msys2/MINGW-packages/tree/master/mingw-w64-julia
For the near term, I propose that we develop build instructions and improve compatibility with the MSYS2 UCRT64 build environment. This may be beneficial by also making Julia buildable on other MSYS2 platforms such as aarch64.