Add support for new build layout in 24.1

[jerboaa]

The new location is, for example, linux-amd64/glibc/liblibchelper.a when it used to be amd64/liblibchelper.a.

This also removes an MX 5.313 conditional which is useless in master of mandrel-packaging which is supposed to build latest graal/master (needing mx 6+).

This fixes build issues seen in CI such as this one: https://github.com/graalvm/mandrel/actions/runs/7442144672/job/20245208110#step:8:475

[jerboaa]

Mac OS build fails with:

Downloading NINJA from ['https://github.com/ninja-build/ninja/releases/download/v1.10.2/ninja-mac.zip']
JvmFuncsFallbacksBuildTask svm-jvmfuncs-fallback-builder: Failed due to error: 1
Error: Process completed with exit code 1.

I have no idea how to fix that and don't have a mac build machine. @zakkak probably something for you?

I'm looking at the windows build failure...

[jerboaa]

Windows build failure seems to be:

2024-01-08T17:34:40.6448804Z ninja -j 1 -v 
2024-01-08T17:34:40.6449054Z [6876: started subprocess 4660: ['ninja', '-j', '1', '-v']]
2024-01-08T17:34:40.6449800Z ninja: error: mkdir(D_/a/mandrel-packaging/mandrel-packaging/mandrel/substratevm/mxbuild/jdk23/svm-jvmfuncs-fallback-builder): No such file or directory
2024-01-08T17:34:40.6451788Z [1/3] cl -nologo -showIncludes -ID:\a\mandrel-packaging\mandrel-packaging\mandrel\substratevm\src\com.oracle.svm.native.jvm.windows\src -ID:\a\mandrel-packaging\mandrel-packaging\jdk\include -ID:\a\mandrel-packaging\mandrel-packaging\jdk\include\win32  -MD -Zi -O2 -DJNIEXPORT= -c D:\a\mandrel-packaging\mandrel-packaging\mandrel\substratevm\src\com.oracle.svm.native.jvm.windows\src\JvmFuncs.c -Fosrc\JvmFuncs.obj
2024-01-08T17:34:40.6451882Z ninja: build stopped: .
2024-01-08T17:34:40.6452335Z Building com.oracle.svm.native.jvm.windows_windows-amd64-default with Ninja: Failed due to error: 1

[jerboaa]

Windows build failure seems to be:

2024-01-08T17:34:40.6448804Z ninja -j 1 -v 
2024-01-08T17:34:40.6449054Z [6876: started subprocess 4660: ['ninja', '-j', '1', '-v']]
2024-01-08T17:34:40.6449800Z ninja: error: mkdir(D_/a/mandrel-packaging/mandrel-packaging/mandrel/substratevm/mxbuild/jdk23/svm-jvmfuncs-fallback-builder): No such file or directory
2024-01-08T17:34:40.6451788Z [1/3] cl -nologo -showIncludes -ID:\a\mandrel-packaging\mandrel-packaging\mandrel\substratevm\src\com.oracle.svm.native.jvm.windows\src -ID:\a\mandrel-packaging\mandrel-packaging\jdk\include -ID:\a\mandrel-packaging\mandrel-packaging\jdk\include\win32  -MD -Zi -O2 -DJNIEXPORT= -c D:\a\mandrel-packaging\mandrel-packaging\mandrel\substratevm\src\com.oracle.svm.native.jvm.windows\src\JvmFuncs.c -Fosrc\JvmFuncs.obj
2024-01-08T17:34:40.6451882Z ninja: build stopped: .
2024-01-08T17:34:40.6452335Z Building com.oracle.svm.native.jvm.windows_windows-amd64-default with Ninja: Failed due to error: 1

or maybe not. A different run has:

Caused by: java.nio.file.NoSuchFileException: D:\a\mandrel\mandrel\mandrel\substratevm\mxbuild\windows-amd64\com.oracle.svm.native.libchelper\windows-amd64\glibc\libchelper.lib
    at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:85)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
    at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
    at java.base/sun.nio.fs.WindowsFileCopy.copy(WindowsFileCopy.java:108)
    at java.base/sun.nio.fs.WindowsFileSystemProvider.copy(WindowsFileSystemProvider.java:282)
    at java.base/java.nio.file.Files.copy(Files.java:1305)
    at FileSystem.copy(build.java:1606)
    at build.main(build.java:173)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher.execute(SourceLauncher.java:264)
    at jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher.run(SourceLauncher.java:153)
    at jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher.main(SourceLauncher.java:78)

[jerboaa]

https://github.com/graalvm/mx/commit/f49c56f846f8656ce3a174c22631234b8de8fdea seems to be the mx commit which caused this new build failure.

[jerboaa]

Not sure what's wrong with the windows PR tester build, but it built fine on windows here: https://github.com/graalvm/mandrel/actions/runs/7453024302/job/20277533723#step:8:595

[zakkak]

I have no idea how to fix that and don't have a mac build machine. @zakkak probably something for you?

On it.

[zakkak]

The MacOS issue is caused by https://github.com/adoptium/temurin-build/issues/3602, note however that even after manually renaming the folder the build still fails because in darwin mx uses default in the static lib paths, just like it does in windows.

I also see some formatting issues (tabs vs spaces) causing indentation inconsistencies.

[zakkak]

or maybe not. A different run has:

Caused by: java.nio.file.NoSuchFileException: D:\a\mandrel\mandrel\mandrel\substratevm\mxbuild\windows-amd64\com.oracle.svm.native.libchelper\windows-amd64\glibc\libchelper.lib

This looks like a build using master instead of your branch/patch.

The actual issue seems to indeed be some failure to create a directory.

Not sure what's wrong with the windows PR tester build, but it built fine on windows here: https://github.com/graalvm/mandrel/actions/runs/7453024302/job/20277533723#step:8:595

I see that you use a different mandrel-packaging commit in the passing run (294f4415c6536f109ea3a701a3159f3460c94076 instead of cec47827490e5b5940b4e58633736bad22bf054b) the only difference however appears to be the exclusion of --verbose in the latter :shrug:, see https://github.com/graalvm/mandrel-packaging/compare/294f4415c6536f109ea3a701a3159f3460c94076...cec47827490e5b5940b4e58633736bad22bf054b

[jerboaa]

The MacOS issue is caused by adoptium/temurin-build#3602, note however that even after manually renaming the folder the build still fails because in darwin mx uses default in the static lib paths, just like it does in windows.

Thanks. I'll fix it.

I also see some formatting issues (tabs vs spaces) causing indentation inconsistencies.

OK. I'll fix those.

[jerboaa]

or maybe not. A different run has: Caused by: java.nio.file.NoSuchFileException: D:\a\mandrel\mandrel\mandrel\substratevm\mxbuild\windows-amd64\com.oracle.svm.native.libchelper\windows-amd64\glibc\libchelper.lib

This looks like a build using master instead of your branch/patch.

OK. Ignore that then.

The actual issue seems to indeed be some failure to create a directory.

Yes, the failing build without --verbose (which is too noisy to see the build failure clearly) is:

(ninja: error: mkdir(D_/a/mandrel-packaging/mandrel-packaging/mandrel/substratevm/mxbuild/jdk23/svm-jvmfuncs-fallback-builder): No such file or directory

[1/3] CC src\JvmFuncs.obj

ninja: build stopped: .

, -2)
Building com.oracle.svm.native.jvm.windows_windows-amd64-default with Ninja: Failed due to error: 1
java.net.SocketException: Connection reset
java.net.SocketException: Connection reset

Not sure what's wrong with the windows PR tester build, but it built fine on windows here: https://github.com/graalvm/mandrel/actions/runs/7453024302/job/20277533723#step:8:595

I see that you use a different mandrel-packaging commit in the passing run (294f441 instead of cec4782) the only difference however appears to be the exclusion of --verbose in the latter 🤷, see 294f441...cec4782

Yes. Only --verbose is the difference. But somehow the manual workflow doesn't have the mkdir issue, while the PR tester has.

[jerboaa]

The MacOS issue is caused by adoptium/temurin-build#3602, note however that even after manually renaming the folder

Do you have a patch for the rename until https://github.com/adoptium/temurin-build/issues/3602 is fixed?

[zakkak]

Do you have a patch for the rename until adoptium/temurin-build#3602 is fixed?

No, I only tested it locally by doing the rename by hand.

[jerboaa]

Mac build is OK now. https://github.com/graalvm/mandrel/actions/runs/7460492845/job/20298676477 is a run verifying the windows build (which looks good so far). Something is strange on the PR tester, but I'm not sure what it is so I'll stop there and let somebody with a windows box look into it. I'll clean up the commits, re-test and would then like to merge so as to unbreak the windows build in CI. @zakkak How does that sound?

[zakkak]

Sounds good. FWIW, some differences I see in the mandrel-packaging CI vs the mandrel CI:

1. mandrel-packaging uses python 3.10 while mandrel uses 3.8
2. mandrel-packaging checks out mx master, while mandrel checks out the version defined in graal/common.json

[jerboaa]

@zakkak Please review, thanks!

[jerboaa]

Sounds good. FWIW, some differences I see in the mandrel-packaging CI vs the mandrel CI:

1. mandrel-packaging uses python 3.10 while mandrel uses 3.8

2. mandrel-packaging checks out mx `master`, while mandrel checks out the version defined in `graal/common.json`

FWIW, https://github.com/jerboaa/graal/actions/runs/7461810722/job/20302721226 is a build with python 3.10.11 (as the PR tester uses). It built fine, so that's not it.

[jerboaa]

mandrel-packaging checks out mx master, while mandrel checks out the version defined in graal/common.json

We now also have PR tester runs with mx version from graal/common.json and that shows the same windows issue. That's not it either.