search

beehive-lab / TornadoVM

TornadoVM: A practical and efficient heterogeneous programming framework for managed languages
https://www.tornadovm.org
Apache License 2.0
1.18k stars 113 forks source link

New Iteration for achieving Multi-Threaded Execution Plans #389

Closed jjfumero closed 5 months ago

jjfumero commented 5 months ago

Description

This PR fixes an issue sharing the dataObjectState with multiple Java threads.

Problem description

The problem was that this state must be per thread (private), otherwise, when a buffer is allocated/removed, the device buffer taken might end-up being deallocated twice, thus, provoking a seg-fault from the driver side. This PR solves this issue.

Disclaimer: I can run without errors using the OpenCL and SPIR-V backends. However, when running with the PTX backend, I still get random errors in the JNI side, mostly related to events. But IMO, this is not related to this fix.

Backend/s tested

Mark the backends affected by this PR.

OS tested

Mark the OS where this PR is tested.

Did you check on FPGAs?

If it is applicable, check your changes on FPGAs.

How to test the new patch?

$ make 
$ make tests 
$ tornado-test --jvm="-Dtornado.device.memory=4GB" --debug -V --fast uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans

# Pass also the batch processing examples or each backend:
$ tornado-test -V --fast uk.ac.manchester.tornado.unittests.batches.TestBatches

Additional context

mairooni commented 5 months ago

I get this for OpenCL 

tornado -ea  --jvm "-Xmx6g -Dtornado.recover.bailout=False -Dtornado.unittests.verbose=True  -Dtornado.device.memory=4GB"  -m  tornado.unittests/uk.ac.manchester.tornado.unittests.tools.TornadoTestRunner  --params "uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans"
WARNING: Using incubator modules: jdk.incubator.vector

Running thread t0Running thread t1[TornadoVM-OCL-JNI] ERROR : clEnqueueNDRangeKernel -> Returned: -4
[TornadoVM-OCL-JNI] ERROR : clWaitForEvents -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[JNI] uk.ac.manchester.tornado.drivers.opencl> notify error:
[JNI] uk.ac.manchester.tornado.drivers.opencl> [TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfoCL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_NDRANGE_KERNEL on NVIDIA GeForce RTX 3050 Ti Laptop GPU (Device 0).
 -> Returned: 
-58
[JNI] uk.ac.manchester.tornado.drivers.opencl> notify error:
[JNI] uk.ac.manchester.tornado.drivers.opencl> CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_READ_BUFFER on NVIDIA GeForce RTX 3050 Ti Laptop GPU (Device 0).

[ERROR] clEnqueueReadBuffer, code = -4 n[TornadoVM-OCL-JNI] ERROR : clEnqueueReadBuffer -> Returned: -4
[TornadoVM-OCL-JNI] ERROR : clWaitForEvents -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clEnqueueNDRangeKernel -> Returned: -4
[JNI] uk.ac.manchester.tornado.drivers.opencl> notify error:
[JNI] uk.ac.manchester.tornado.drivers.opencl> CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_NDRANGE_KERNEL on NVIDIA GeForce RTX 3050 Ti Laptop GPU (Device 0).

[TornadoVM-OCL-JNI] ERROR : clWaitForEvents -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[ERROR] clEnqueueReadBuffer, code = -4 n[TornadoVM-OCL-JNI] ERROR : clEnqueueReadBuffer -> Returned: -4
[TornadoVM-OCL-JNI] ERROR : clWaitForEvents -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[JNI] uk.ac.manchester.tornado.drivers.opencl> notify error:
[JNI] uk.ac.manchester.tornado.drivers.opencl> CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_READ_BUFFER on NVIDIA GeForce RTX 3050 Ti Laptop GPU (Device 0).

[TornadoVM-OCL-JNI] ERROR : clEnqueueNDRangeKernel -> Returned: -4
[TornadoVM-OCL-JNI] ERROR : clWaitForEvents -> Returned: -58
[JNI] uk.ac.manchester.tornado.drivers.opencl> notify error:
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[JNI] uk.ac.manchester.tornado.drivers.opencl> [TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_NDRANGE_KERNEL on NVIDIA GeForce RTX 3050 Ti Laptop GPU (Device 0).
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58

[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[ERROR] clEnqueueReadBuffer, code = -4 n[TornadoVM-OCL-JNI] ERROR : clEnqueueReadBuffer -> Returned: -4
[TornadoVM-OCL-JNI] ERROR : clWaitForEvents -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58[JNI] uk.ac.manchester.tornado.drivers.opencl> notify error:
[JNI] uk.ac.manchester.tornado.drivers.opencl> CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_READ_BUFFER on NVIDIA GeForce RTX 3050 Ti Laptop GPU (Device 0).

[TornadoVM-OCL-JNI] ERROR : clEnqueueNDRangeKernel -> Returned: -4
[TornadoVM-OCL-JNI] ERROR : clWaitForEvents -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[JNI] uk.ac.manchester.tornado.drivers.opencl> notify error:
[JNI] uk.ac.manchester.tornado.drivers.opencl> CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_NDRANGE_KERNEL on NVIDIA GeForce RTX 3050 Ti Laptop GPU (Device 0).

[ERROR] clEnqueueReadBuffer, code = -4 n[TornadoVM-OCL-JNI] ERROR : clEnqueueReadBuffer -> Returned: -4
[TornadoVM-OCL-JNI] ERROR : clWaitForEvents -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[JNI] uk.ac.manchester.tornado.drivers.opencl> notify error:
[JNI] uk.ac.manchester.tornado.drivers.opencl> CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_READ_BUFFER on NVIDIA GeForce RTX 3050 Ti Laptop GPU (Device 0).

[TornadoVM-OCL-JNI] ERROR : clEnqueueNDRangeKernel -> Returned: -4
[JNI] uk.ac.manchester.tornado.drivers.opencl> notify error:
[JNI] uk.ac.manchester.tornado.drivers.opencl> CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_NDRANGE_KERNEL on NVIDIA GeForce RTX 3050 Ti Laptop GPU (Device 0).

[TornadoVM-OCL-JNI] ERROR : clWaitForEvents -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[TornadoVM-OCL-JNI] ERROR : clGetEventProfilingInfo -> Returned: -58
[ERROR] clEnqueueReadBuffer, code = -4 n[TornadoVM-OCL-JNI] ERROR : clEnqueueReadBuffer -> Returned: [JNI] uk.ac.manchester.tornado.drivers.opencl> notify error:
-4[JNI] uk.ac.manchester.tornado.drivers.opencl> CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_READ_BUFFER on NVIDIA GeForce RTX 3050 Ti Laptop GPU (Device 0).
...

This happens both when I run the test individually and through the make tests

mairooni commented 5 months ago

PTX: 

tornado-test --jvm="-Dtornado.device.memory=4GB" --debug -V --fast uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans
tornado --jvm "-Xmx6g -Dtornado.recover.bailout=False -Dtornado.unittests.verbose=True -Dtornado.debug=True -Dtornado.device.memory=4GB"  -m  tornado.unittests/uk.ac.manchester.tornado.unittests.tools.TornadoTestRunner  --params "uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans"
WARNING: Using incubator modules: jdk.incubator.vector
Running thread t0Running thread t1      [TornadoVM-PTX-JNI] ERROR : cuMemAlloc -> Returned: 2
        [TornadoVM-PTX-JNI] ERROR : cuMemcpyHtoDAsyncMemSeg -> Returned: 1
        [TornadoVM-PTX-JNI] ERROR : cuEventSynchronize -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventElapsedTime -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventElapsedTime -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventSynchronize -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventElapsedTime -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventElapsedTime -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventCreate (beforeEvent) -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventCreate (afterEvent) -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventRecord -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuMemcpyDtoHMemSeg -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventRecord -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventCreate (beforeEvent) -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventCreate (afterEvent) -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventRecord -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuMemcpyDtoHMemSeg -> Returned: 716
        [TornadoVM-PTX-JNI] ERROR : cuEventRecord -> Returned: 716
[thread 6688 also had an error]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007b707dafe705, pid=6432, tid=6687
#
# JRE version: Java(TM) SE Runtime Environment (21.0+35) (build 21+35-LTS-2513)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (21+35-LTS-2513, mixed mode, tiered, jvmci, parallel gc, linux-amd64)
# Problematic frame:
# C  [libcuda.so.1+0x4fe705]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/mary/Projects/Juan/TornadoVM/core.6432)
#
# An error report file with more information is saved as:
# /home/mary/Projects/Juan/TornadoVM/hs_err_pid6432.log
[5,914s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted (core dumped)
mairooni commented 5 months ago

For SPIR-V the tests pass: 

Test: class uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans
        Running test: test01                     ................  [PASS] 
        Running test: test02                     ................  [PASS] 
        Running test: test03                     ................  [PASS] 
        Running test: test04                     ................  [PASS] 
Test ran: 4, Failed: 0, Unsupported: 0
stratika commented 5 months ago

In Linux OS with SPIR-V BACKEND when I run make tests, I get: 

tornado -ea  --jvm "-Xmx6g -Dtornado.recover.bailout=False -Dtornado.unittests.verbose=True  -Dtornado.device.memory=4GB"  -m  tornado.unittests/uk.ac.manchester.tornado.unittests.tools.TornadoTestRunner  --params "uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans"
WARNING: Using incubator modules: jdk.incubator.vector
Exception in thread "Thread-17" java.lang.NullPointerException: Cannot load from object array because "this.commandQueueGroupProperties" is null
    at beehive.levelzero.jni@0.1.3/uk.ac.manchester.tornado.drivers.spirv.levelzero.LevelZeroDevice.getCommandQueueGroupProperties(LevelZeroDevice.java:115)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVCommandQueueTable$ThreadCommandQueueTable.getCommandQueueOrdinal(SPIRVCommandQueueTable.java:125)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVCommandQueueTable$ThreadCommandQueueTable.createCommandQueue(SPIRVCommandQueueTable.java:88)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVCommandQueueTable$ThreadCommandQueueTable.get(SPIRVCommandQueueTable.java:73)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVCommandQueueTable.get(SPIRVCommandQueueTable.java:56)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVLevelZeroContext.getCommandQueueForDevice(SPIRVLevelZeroContext.java:153)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVLevelZeroContext.enqueueWriteBuffer(SPIRVLevelZeroContext.java:558)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVDeviceContext.enqueueWriteBuffer(SPIRVDeviceContext.java:346)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.mm.SPIRVMemorySegmentWrapper.enqueueWrite(SPIRVMemorySegmentWrapper.java:159)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.runtime.SPIRVTornadoDevice.streamIn(SPIRVTornadoDevice.java:398)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.interpreter.TornadoVMInterpreter.transferHostToDeviceAlways(TornadoVMInterpreter.java:480)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.interpreter.TornadoVMInterpreter.execute(TornadoVMInterpreter.java:305)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.interpreter.TornadoVMInterpreter.execute(TornadoVMInterpreter.java:855)
    at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
    at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.TornadoVM.executeInterpreterSingleThreaded(TornadoVM.java:125)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.TornadoVM.execute(TornadoVM.java:112)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph.scheduleInner(TornadoTaskGraph.java:859)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph.execute(TornadoTaskGraph.java:1366)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph.execute(TornadoTaskGraph.java:1378)
    at tornado.api@1.0.4-dev/uk.ac.manchester.tornado.api.TaskGraph.execute(TaskGraph.java:777)
    at tornado.api@1.0.4-dev/uk.ac.manchester.tornado.api.ImmutableTaskGraph.execute(ImmutableTaskGraph.java:49)
    at tornado.api@1.0.4-dev/uk.ac.manchester.tornado.api.TornadoExecutionPlan$TornadoExecutor.lambda$execute$0(TornadoExecutionPlan.java:406)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
    at tornado.api@1.0.4-dev/uk.ac.manchester.tornado.api.TornadoExecutionPlan$TornadoExecutor.execute(TornadoExecutionPlan.java:406)
    at tornado.api@1.0.4-dev/uk.ac.manchester.tornado.api.TornadoExecutionPlan.execute(TornadoExecutionPlan.java:117)
    at tornado.unittests@1.0.4-dev/uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans.compute(TestMultiThreadedExecutionPlans.java:183)
    at tornado.unittests@1.0.4-dev/uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans.lambda$test03$4(TestMultiThreadedExecutionPlans.java:193)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Exception in thread "Thread-263" java.lang.NullPointerException: Cannot load from object array because "this.commandQueueGroupProperties" is null
    at beehive.levelzero.jni@0.1.3/uk.ac.manchester.tornado.drivers.spirv.levelzero.LevelZeroDevice.getCommandQueueGroupProperties(LevelZeroDevice.java:115)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVCommandQueueTable$ThreadCommandQueueTable.getCommandQueueOrdinal(SPIRVCommandQueueTable.java:125)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVCommandQueueTable$ThreadCommandQueueTable.createCommandQueue(SPIRVCommandQueueTable.java:88)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVCommandQueueTable$ThreadCommandQueueTable.get(SPIRVCommandQueueTable.java:73)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVCommandQueueTable.get(SPIRVCommandQueueTable.java:56)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVLevelZeroContext.getCommandQueueForDevice(SPIRVLevelZeroContext.java:153)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVLevelZeroContext.enqueueWriteBuffer(SPIRVLevelZeroContext.java:558)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.SPIRVDeviceContext.enqueueWriteBuffer(SPIRVDeviceContext.java:346)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.mm.SPIRVMemorySegmentWrapper.enqueueWrite(SPIRVMemorySegmentWrapper.java:159)
    at tornado.drivers.spirv@1.0.4-dev/uk.ac.manchester.tornado.drivers.spirv.runtime.SPIRVTornadoDevice.streamIn(SPIRVTornadoDevice.java:398)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.interpreter.TornadoVMInterpreter.transferHostToDeviceAlways(TornadoVMInterpreter.java:480)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.interpreter.TornadoVMInterpreter.execute(TornadoVMInterpreter.java:305)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.interpreter.TornadoVMInterpreter.execute(TornadoVMInterpreter.java:855)
    at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1024)
    at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.TornadoVM.executeInterpreterSingleThreaded(TornadoVM.java:125)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.TornadoVM.execute(TornadoVM.java:112)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph.scheduleInner(TornadoTaskGraph.java:859)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph.execute(TornadoTaskGraph.java:1366)
    at tornado.runtime@1.0.4-dev/uk.ac.manchester.tornado.runtime.tasks.TornadoTaskGraph.execute(TornadoTaskGraph.java:1378)
    at tornado.api@1.0.4-dev/uk.ac.manchester.tornado.api.TaskGraph.execute(TaskGraph.java:777)
    at tornado.api@1.0.4-dev/uk.ac.manchester.tornado.api.ImmutableTaskGraph.execute(ImmutableTaskGraph.java:49)
    at tornado.api@1.0.4-dev/uk.ac.manchester.tornado.api.TornadoExecutionPlan$TornadoExecutor.lambda$execute$0(TornadoExecutionPlan.java:406)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
    at tornado.api@1.0.4-dev/uk.ac.manchester.tornado.api.TornadoExecutionPlan$TornadoExecutor.execute(TornadoExecutionPlan.java:406)
    at tornado.api@1.0.4-dev/uk.ac.manchester.tornado.api.TornadoExecutionPlan.execute(TornadoExecutionPlan.java:117)
    at tornado.unittests@1.0.4-dev/uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans.compute(TestMultiThreadedExecutionPlans.java:183)
    at tornado.unittests@1.0.4-dev/uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans.lambda$test04$6(TestMultiThreadedExecutionPlans.java:230)
    at java.base/java.lang.Thread.run(Thread.java:1583)

Running thread t0Running thread t1Test: class uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans
    Running test: test01                     ................  [PASS] 
    Running test: test02                     ................  [PASS] 
    Running test: test03                     ................  [PASS] 
    Running test: test04                     ................  [PASS] 
Test ran: 4, Failed: 0, Unsupported: 0

However, when I run this test as standalone, it does not throw that exception. Has anyone observed similar behaviour?

mikepapadim commented 5 months ago

OpenCL on Popos:


tornado -ea  --jvm "-Xmx6g -Dtornado.recover.bailout=False -Dtornado.unittests.verbose=True  -Dtornado.device.memory=4GB"  -m  tornado.unittests/uk.ac.manchester.tornado.unittests.tools.TornadoTestRunner  --params "uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans"
WARNING: Using incubator modules: jdk.incubator.vector

Running thread t1Running thread t0Test: class uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans
    Running test: test01                     ................  [PASS] 
    Running test: test02                     ................  [PASS] 
    Running test: test03                     ................  [PASS] 
    Running test: test04                     ................  [PASS] 
Test ran: 4, Failed: 0, Unsupported: 0
mikepapadim commented 5 months ago

In macOS (OpenCL):

tornado-test --jvm="-Dtornado.device.memory=4GB" --debug -V --fast uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans
tornado --jvm "-Xmx6g -Dtornado.recover.bailout=False -Dtornado.unittests.verbose=True -Dtornado.debug=True -Dtornado.device.memory=4GB"  -m  tornado.unittests/uk.ac.manchester.tornado.unittests.tools.TornadoTestRunner  --params "uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans"
WARNING: Using incubator modules: jdk.incubator.vector
Running thread t0Running thread t1Context leak detected, msgtracer returned -1
Context leak detected, msgtracer returned -1
Context leak detected, msgtracer returned -1
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001ddb1a170, pid=155, tid=43875
#
# JRE version: OpenJDK Runtime Environment Microsoft-9388422 (21.0.3+9) (build 21.0.3+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM Microsoft-9388422 (21.0.3+9-LTS, mixed mode, tiered, jvmci, parallel gc, bsd-aarch64)
# Problematic frame:
# C  [OpenCL+0x22170]  clLogMessagesToStderrAPPLE+0x2dc
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/thanos/repositories/TornadoVM-stratika/hs_err_pid155.log
[2.208s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
#   https://github.com/microsoft/openjdk/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

I see you run it with Microsoft jdk, can you try a different one?

stratika commented 5 months ago

In macOS (OpenCL):

tornado-test --jvm="-Dtornado.device.memory=4GB" --debug -V --fast uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans
tornado --jvm "-Xmx6g -Dtornado.recover.bailout=False -Dtornado.unittests.verbose=True -Dtornado.debug=True -Dtornado.device.memory=4GB"  -m  tornado.unittests/uk.ac.manchester.tornado.unittests.tools.TornadoTestRunner  --params "uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans"
WARNING: Using incubator modules: jdk.incubator.vector
Running thread t0Running thread t1Context leak detected, msgtracer returned -1
Context leak detected, msgtracer returned -1
Context leak detected, msgtracer returned -1
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001ddb1a170, pid=155, tid=43875
#
# JRE version: OpenJDK Runtime Environment Microsoft-9388422 (21.0.3+9) (build 21.0.3+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM Microsoft-9388422 (21.0.3+9-LTS, mixed mode, tiered, jvmci, parallel gc, bsd-aarch64)
# Problematic frame:
# C  [OpenCL+0x22170]  clLogMessagesToStderrAPPLE+0x2dc
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/thanos/repositories/TornadoVM-stratika/hs_err_pid155.log
[2.208s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
#   https://github.com/microsoft/openjdk/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

I see you run it with Microsoft jdk, can you try a different one?

Sure, I also ran with OpenJDK 21:

tornado-test --jvm="-Dtornado.device.memory=4GB" --debug -V --fast uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans
tornado --jvm "-Xmx6g -Dtornado.recover.bailout=False -Dtornado.unittests.verbose=True -Dtornado.debug=True -Dtornado.device.memory=4GB"  -m  tornado.unittests/uk.ac.manchester.tornado.unittests.tools.TornadoTestRunner  --params "uk.ac.manchester.tornado.unittests.multithreaded.TestMultiThreadedExecutionPlans"
WARNING: Using incubator modules: jdk.incubator.vector
Running thread t0Running thread t1Context leak detected, msgtracer returned -1
Context leak detected, msgtracer returned -1
Context leak detected, msgtracer returned -1
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001ddb1a170, pid=7096, tid=90907
#
# JRE version: Java(TM) SE Runtime Environment (21.0.2+13) (build 21.0.2+13-LTS-58)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (21.0.2+13-LTS-58, mixed mode, tiered, jvmci, parallel gc, bsd-aarch64)
# Problematic frame:
# C  [OpenCL+0x22170]  clLogMessagesToStderrAPPLE+0x2dc
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/thanos/repositories/TornadoVM-stratika/hs_err_pid7096.log
[2.732s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
jjfumero commented 5 months ago

Thank you all for the extensive testing. This PR needs to be combined with #387. So let's merge that first and then we tested again.

jjfumero commented 5 months ago

Tests are passing in my configuration. Since we merged with #387, can we all check if this fixes the issue?

jjfumero commented 5 months ago

Great, I will merge.