search

jeroen / curl

A Modern and Flexible Web Client for R
https://jeroen.r-universe.dev/curl
Other
216 stars 71 forks source link

Windows ARM64 source build - fails if parallel make jobs used (winlibs download race condition) #340

Open andrjohns opened 2 months ago

andrjohns commented 2 months ago

Currently experiencing this on Windows ARM64 (so not sure if present in x86) with all packages that need winlibs static libs (where the library isn't already in rtools).

If I have MAKEFLAGS=-j2 (or any value > 1) set in my .Renviron, the installation will fail at the linking stage due to libraries not being found:

install.packages("curl", repos = "https://cloud.r-project.org")
...
"C:/PROGRA~1/R-AARC~1/R-devel/bin/Rscript.exe" "../tools/winlibs.R"
clang -shared -s -static-libgcc -o curl.dll tmp.def callbacks.o curl.o download.o escape.o fetch.o findport.o form.o getdate.o handle.o ieproxy.o init.o interrupt.o multi.o nslookup.o options.o reflist.o split.o ssl.o typechecking.o utils.o version.o winidn.o writer.o -L../windows/libcurl/libclang-17.0.6 -L../windows/libcurl/lib -lwinhttp -lcurl -lnghttp2 -lssh2 -lz -lssl -lcrypto -pthread -lgdi32 -lws2_32 -lcrypt32 -lbcrypt -lwldap32 -L/aarch64-w64-mingw32.static.posix/lib -L/aarch64-w64-mingw32.static.posix/lib -LC:/PROGRA~1/R-AARC~1/R-devel/bin -lR
lld: error: unable to find library -lnghttp2
clang-17: error: linker command failed with exit code 1 (use -v to see invocation)
[1] TRUE
echo '#include <curl/curl.h>' | clang  -I../windows/libcurl/include -DCURL_STATICLIB -DSTRICT_R_HEADERS -DR_NO_REMAP -std=gnu99 -E -xc - | grep "^[ \t]*CURLOPT_.*," | sed s/,// > ../tools/option_table.txt
no DLL was created
ERROR: compilation failed for package 'curl'

Which appears to be caused by $(SHLIB) target reaching the linking stage before the winlibs target has completed. I'd guess that this hasn't been seen to-date on x86_64 because compilation is generally slower than under ARM, so the winlibs target has usually finished before $(SHLIB) reaches linking.

From local testing, an easy fix appears to be just making the dependency explicit in Makevars.win:

- all: clean winlibs
+ $(SHLIB): $(OBJECTS)
+ $(OBJECTS): winlibs

But I'm not sure whether there's an easier fix which doesn't require updating all winlibs R packages. Thanks!

jeroen commented 2 months ago

Yes I think that is the only solution. FWIW parallel make on Windows was never really supported because there were many other issues with it, so we didn't bother to test it.