search

oracle / graal

GraalVM compiles Java applications into native executables that start instantly, scale fast, and use fewer compute resources 🚀
https://www.graalvm.org
Other
20.32k stars 1.63k forks source link

Build fails due to missing polybench #7384

Closed oroppas closed 1 year ago

oroppas commented 1 year ago

Describe the issue Archiving fails since sdk/mxbuild/linux-amd64/polybench.image/polybench is not generated during the build process.

Steps to reproduce the issue Please include both build steps as well as run steps

  1. JAVA_HOME=labsjdk-ce-21-jvmci-23.1-b15 mx -v --max-cpus 2 --dynamicimports /substratevm,/tools,/sulong,/graal-nodejs,graalpython,/wasm,/espresso build

Describe GraalVM and your environment:

More details Add any other information about the problem here. Especially important are stack traces or log output. Feel free to link to gists or to screenshots if necessary.

Archiving GRAALVM_F3A1E87DD6_JAVA21: Failed due to error: [Errno 2] No such file or directory: '/home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/polybench'
Exception in thread Thread-792 (executeTask):
Traceback (most recent call last):
  File "/usr/lib64/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 92, in run
    super(_DummyProcess, self).run()
  File "/usr/lib64/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 15219, in executeTask
    task.execute()
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 5244, in execute
    _built = self.build()
             ^^^^^^^^^^^^
  File "/home/ryuta/packages/graalvm/graalvm/graal/sdk/mx.sdk/mx_sdk_vm_impl.py", line 1083, in build
    super(GraalVmLayoutDistributionTask, self).build()
  File "/home/ryuta/packages/graalvm/graalvm/graal/sdk/mx.sdk/mx_sdk_vm_impl.py", line 887, in build
    super(BaseGraalVmLayoutDistributionTask, self).build()
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 5615, in build
    self.subject.make_archive()
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 6451, in make_archive
    self._install_source(source, output, destination, arc)
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 6214, in _install_source
    _install_source_files([next(d.getArchivableResults(single=True))], archive=archive)
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 6198, in _install_source_files
    merge_recursive(_source_file, _dst, _arcname, excludes, archive=archive)
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 6166, in merge_recursive
    archiver.add(src, dst, provenance)
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 15788, in add
    self._add_f(filename, archive_name, provenance)
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 15717, in _add_tar
    self.zf.add(filename, archive_name, filter=self._tarinfo_filter, recursive=False)
  File "/usr/lib64/python3.11/tarfile.py", line 2157, in add
    tarinfo = self.gettarinfo(name, arcname)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/tarfile.py", line 2030, in gettarinfo
    statres = os.lstat(name)
              ^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/polybench'
Archiving GRAALVM_F3A1E87DD6_JAVA21 failed
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx_enter.py", line 44, in <module>
    mx._main_wrapper()
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 18796, in _main_wrapper
    main()
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 18782, in main
    retcode = c(command_args)
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx_commands.py", line 145, in __call__
    return self.command_function(*args, **kwargs)
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 15289, in build
    abort(f'{len(failed)} build tasks failed')
  File "/home/ryuta/packages/graalvm/graalvm/mx/mx.py", line 4262, in abort
    traceback.print_stack()
1 build tasks failed
oroppas commented 1 year ago

Noticeably absent is sdk/mxbuild/linux-amd64/polybench.image/polybench (executable)

[polybench:824534] Produced artifacts:
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/build-artifacts.json (build_info)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/graal_isolate.h (c_header)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/graal_isolate_dynamic.h (c_header)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/polybench.h (c_header)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/polybench.so (shared_library)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/polybench.so.debug (debug_info)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/polybench_dynamic.h (c_header)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/sources (debug_info)

Full log:

Building polybench.image... [dependencies updated: POLYBENCH, GRAALVM_AF29D570EF_JAVA21_STAGE1]
env MX_HOME=/home/ryuta/packages/graalvm/graalvm/mx MX__SUITEMODEL=sibling MX_PRIMARY_SUITE_PATH=/home/ryuta/packages/graalvm/graalvm/graal/vm MX_SUBPROCESS_COMMAND_FILE=/tmp/mx_subprocess_command.lgwg1b2x \
/home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/GRAALVM_AF29D570EF_JAVA21_STAGE1/graalvm-af29d570ef-java21-24.0.0-dev/lib/svm/bin/native-image --parallelism=2 -H:+UnlockExperimentalVMOptions -H:+BuildOutputPrefix -H:+GenerateBuildArtifactsFile -H:-UnlockExperimentalVMOptions --macro:polybench-launcher --macro:truffle --language:all -H:+UnlockExperimentalVMOptions -H:Path=/home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image -H:-UnlockExperimentalVMOptions
Warning: Using a deprecated option --rerun-class-initialization-at-runtime= from 'META-INF/native-image/org.graalvm.shadowed.icu4j/native-image.properties' in 'file:///home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/GRAALVM_AF29D570EF_JAVA21_STAGE1/graalvm-af29d570ef-java21-24.0.0-dev/languages/icu4j/truffle-icu4j.jar'. Currently there is no replacement for this option. Try using --initialize-at-run-time or use the non-API option -H:ClassInitialization directly.
Warning: Invalid option --add-exports org.graalvm.nativeimage/org.graalvm.nativeimage.impl=org.graalvm.py provided by 'META-INF/native-image/org.graalvm.py/native-image.properties' in 'file:///home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/GRAALVM_AF29D570EF_JAVA21_STAGE1/graalvm-af29d570ef-java21-24.0.0-dev/languages/python/graalpython.jar'. Specified target-module 'org.graalvm.py' is unknown.
[polybench:824534] ========================================================================================================================
[polybench:824534] GraalVM Native Image: Generating 'polybench' (shared library)...
[polybench:824534] ========================================================================================================================
[polybench:824534] [1/8] Initializing...                                                                 (5.5s @ 0.32GB)
[polybench:824534]  Java version: 21+35, vendor version: GraalVM CE 21-dev+35.1
[polybench:824534]  Graal compiler: optimization level: 2, target machine: compatibility
[polybench:824534]  C compiler: gcc (redhat, x86_64, 12.3.1)
[polybench:824534]  Garbage collector: Serial GC (max heap size: 80% of RAM)
[polybench:824534]  11 user-specific feature(s):
[polybench:824534]  - com.oracle.graal.python.BouncyCastleFeature
[polybench:824534]  - com.oracle.svm.thirdparty.gson.GsonFeature
[polybench:824534]  - com.oracle.svm.truffle.TruffleBaseFeature: Provides support for Truffle languages
[polybench:824534]  - com.oracle.svm.truffle.TruffleFeature: Provides support for Truffle runtime compilation
[polybench:824534]  - com.oracle.svm.truffle.TruffleJFRFeature: Provides JFR flight recording for Truffle events.
[polybench:824534]  - com.oracle.svm.truffle.nfi.TruffleNFIFeature
[polybench:824534]  - com.oracle.svm.truffle.nfi.posix.PosixTruffleNFIFeature
[polybench:824534]  - com.oracle.truffle.espresso.ref.FinalizationFeature
[polybench:824534]  - org.graalvm.home.HomeFinderFeature: Finds GraalVM paths and its version number
[polybench:824534]  - org.graalvm.launcher.JLine3FeatureModuleAccessFeature
[polybench:824534]  - org.graalvm.launcher.PolyglotLauncherFeature: Registers launchers for AOT compilation
[polybench:824534] ------------------------------------------------------------------------------------------------------------------------
[polybench:824534]  8 experimental option(s) unlocked:
[polybench:824534]  - '-H:-ParseRuntimeOptions' (origin(s): macro option 'polybench-launcher@file:///home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/GRAALVM_AF29D570EF_JAVA21_STAGE1/graalvm-af29d570ef-java21-24.0.0-dev/lib/svm/macros/polybench-launcher/@user')
[polybench:824534]  - '-H:Name' (alternative API option(s): -o polybench; origin(s): command line)
[polybench:824534]  - '-H:+EnforceMaxRuntimeCompileMethods' (origin(s): macro option 'polybench-launcher@file:///home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/GRAALVM_AF29D570EF_JAVA21_STAGE1/graalvm-af29d570ef-java21-24.0.0-dev/lib/svm/macros/polybench-launcher/@user')
[polybench:824534]  - '-H:Path' (origin(s): command line)
[polybench:824534]  - '-H:+GenerateBuildArtifactsFile' (origin(s): command line)
[polybench:824534]  - '-H:+DumpRuntimeCompilationOnSignal' (origin(s): macro option 'polybench-launcher@file:///home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/GRAALVM_AF29D570EF_JAVA21_STAGE1/graalvm-af29d570ef-java21-24.0.0-dev/lib/svm/macros/polybench-launcher/@user')
[polybench:824534]  - '-H:+BuildOutputPrefix' (origin(s): command line)
[polybench:824534]  - '-H:+AssertInitializationSpecifiedForAllClasses' (origin(s): macro option 'polybench-launcher@file:///home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/GRAALVM_AF29D570EF_JAVA21_STAGE1/graalvm-af29d570ef-java21-24.0.0-dev/lib/svm/macros/polybench-launcher/@user')
[polybench:824534] ------------------------------------------------------------------------------------------------------------------------
[polybench:824534] Build resources:
[polybench:824534]  - 13.80GB of memory (22.1% of 62.44GB system memory, determined at start)
[polybench:824534]  - 2 thread(s) (12.5% of 16 available processor(s), set via '--parallelism=2')
[To redirect Truffle log output to a file use one of the following options:
* '--log.file=<path>' if the option is passed using a guest language launcher.
* '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher.
* Configure logging using the polyglot embedding API.]
[java] WARNING: Thread.start() called on java.lang.ref.Reference$ReferenceHandler / Reference Handler but thread support is disabled: polyglot context does not allow thread creation (`allowCreateThread(false)`)
[java] WARNING: Thread.start() called on java.lang.ref.Finalizer$FinalizerThread / Finalizer but thread support is disabled: polyglot context does not allow thread creation (`allowCreateThread(false)`)
[polybench:824534] [2/8] Performing analysis...  [********]                                            (200.9s @ 7.09GB)
[polybench:824534]    54,213 reachable types   (97.4% of   55,639 total)
[polybench:824534]    90,315 reachable fields  (63.5% of  142,188 total)
[polybench:824534]   252,072 reachable methods (74.9% of  336,727 total)
[polybench:824534]    40,701 runtime compiled methods (12.1% of  336,727 total)
[polybench:824534]    15,484 types,   280 fields, and 4,718 methods registered for reflection
[polybench:824534]        74 types,    81 fields, and   593 methods registered for JNI access
[polybench:824534]         4 native libraries: dl, pthread, rt, z
[polybench:824534] [3/8] Building universe...                                                           (34.9s @ 6.60GB)
[polybench:824534] [4/8] Parsing methods...      [*****]                                                (22.7s @ 7.17GB)
[polybench:824534] [5/8] Inlining methods...     [****]                                                 (21.5s @ 5.92GB)
[polybench:824534] [6/8] Compiling methods...    [****************]                                    (273.7s @ 8.48GB)
[polybench:824534] [7/8] Layouting methods...    [*****]                                                (31.5s @ 9.52GB)
[polybench:824534] [8/8] Creating image...       [**********]                                          (96.7s @ 10.71GB)
[polybench:824534]  207.70MB (21.78%) for code area:   213,313 compilation units
[polybench:824534]  308.83MB (32.39%) for image heap:2,927,940 objects and 4,524 resources
[polybench:824534]  418.56MB (43.89%) for debug info generated in 29.5s
[polybench:824534]   18.52MB ( 1.94%) for other data
[polybench:824534]  953.61MB in total
[polybench:824534] ------------------------------------------------------------------------------------------------------------------------
[polybench:824534] Top 10 origins of code area:                                Top 10 object types in image heap:
[polybench:824534]   65.32MB graalpython.jar                                     73.78MB byte[] for code metadata
[polybench:824534]   37.37MB graaljs.jar                                         33.41MB byte[] for graph encodings
[polybench:824534]   21.95MB espresso.jar                                        25.66MB byte[] for embedded resources
[polybench:824534]   21.82MB sulong-core.jar                                     18.37MB byte[] for java.lang.String
[polybench:824534]   13.48MB java.base                                           17.07MB java.lang.Class
[polybench:824534]    8.62MB org.graalvm.truffle                                 11.25MB byte[] for general heap data
[polybench:824534]    8.21MB bouncycastle-provider.jar                           10.59MB java.lang.String
[polybench:824534]    5.79MB jdk.internal.vm.compiler                             5.82MB com.oracle.svm.truffle.api.SubstrateTruffleMethod
[polybench:824534]    3.57MB truffle-icu4j.jar                                    5.64MB java.lang.Object[]
[polybench:824534]    2.89MB java.xml                                             5.25MB java.util.concurrent.ConcurrentHashMap$Node
[polybench:824534]   17.17MB for 59 more packages                               101.99MB for 14153 more object types
[polybench:824534] ------------------------------------------------------------------------------------------------------------------------
[polybench:824534] Recommendations:
[polybench:824534]  HEAP: Set max heap for improved and more predictable memory usage.
[polybench:824534] ------------------------------------------------------------------------------------------------------------------------
[polybench:824534]                        62.8s (9.0% of total time) in 619 GCs | Peak RSS: 16.70GB | CPU load: 2.93
[polybench:824534] ------------------------------------------------------------------------------------------------------------------------
[polybench:824534] Produced artifacts:
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/build-artifacts.json (build_info)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/graal_isolate.h (c_header)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/graal_isolate_dynamic.h (c_header)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/polybench.h (c_header)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/polybench.so (shared_library)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/polybench.so.debug (debug_info)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/polybench_dynamic.h (c_header)
[polybench:824534]  /home/ryuta/packages/graalvm/graalvm/graal/sdk/mxbuild/linux-amd64/polybench.image/sources (debug_info)
[polybench:824534] ========================================================================================================================
[polybench:824534] Finished generating 'polybench' in 11m 35s.
Finished Building polybench.image
Building org.graalvm.launcher.native.js_amd64 with Ninja... [dependencies updated: libjsvm.so.image, LLVM_NINJA_TOOLCHAIN]
Building org.graalvm.launcher.native.llvm_amd64 with Ninja... [dependencies updated: libllvmvm.so.image, LLVM_NINJA_TOOLCHAIN]
Composing classpath for ['graal-js:GRAALJS_LAUNCHER']. Entries:
- sdk:COLLECTIONS
- sdk:WORD
- sdk:NATIVEIMAGE
- sdk:POLYGLOT
- sdk:JLINE3
- sdk:LAUNCHER_COMMON
- graal-js:GRAALJS_LAUNCHERComposing classpath for ['sulong:SULONG_LAUNCHER']. Entries:
- sdk:COLLECTIONS
- sdk:WORD
- sdk:NATIVEIMAGE
- sdk:POLYGLOT
- sdk:JLINE3
- sdk:LAUNCHER_COMMON
- sulong:SULONG_LAUNCHER
oubidar-Abderrahim commented 1 year ago

Hi, Thank you for reaching out about this, we'll take a look into it shortly

oroppas commented 1 year ago

One thing I found is that the build succeeds without graal-nodejs on --dynamicimports. It's likely something to do with graaljs.

oubidar-Abderrahim commented 1 year ago

I apologize but after some investigation, this seems more like an mx-related configuration, we do not intend to spend time fixing this issue.