Open syamajala opened 8 months ago
I'm running S3D through cuda-memcheck and I'm seeing the following error before the main task even starts:
========= Program hit CUDA_ERROR_INVALID_VALUE (error 1) due to "invalid argument" on CUDA API call to cuMemHostGetDevicePointer_v2. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2c60b3] ========= in /lib/x86_64-linux-gnu/libcuda.so.1 ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/realm/cuda/cuda_module.cc:5666:Realm::Cuda::GPUAllocation::register_allocation(Realm::Cuda::GPU*, void*, unsigned long, bool) [0x1238be1] ========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/librealm.so.1 ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/realm/cuda/cuda_module.cc:4927:Realm::Cuda::CudaModule::create_dma_channels(Realm::RuntimeImpl*) [0x12347ea] ========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/librealm.so.1 ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/realm/runtime_impl.cc:2149:Realm::RuntimeImpl::finish_configure() [0x118ff21] ========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/librealm.so.1 ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/realm/runtime_impl.cc:489:Realm::Runtime::finish_configure() [0x11873d2] ========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/librealm.so.1 ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/legion/runtime.cc:30212:Legion::Internal::Runtime::start(int, char**, bool, bool, bool) [0x31c1998] ========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/liblegion.so.1 ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/legion/runtime/legion/legion.cc:7041:Legion::Runtime::start(int, char**, bool, bool, bool) [0x2caa8fc] ========= in /scratch2/seshu/legion_s3d_tdb//legion/language/build/lib/liblegion.so.1 ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/rhst/s3d_rank_mpi.cc:280:S3DRank::start_legion() [0xa2da4] ========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/librhsf.so ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/rhst/rhst_fortran.cc:133:initialize_rhsf_legion_ [0xa1246] ========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/librhsf.so ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/s3d/source/drivers/solve_driver.f90:196:solve_driver_ [0x1ddaee] ========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/s3d.x ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/s3d/source/drivers/main.f90:131:MAIN__ [0x1dd3e8] ========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/s3d.x ========= Host Frame:/scratch2/seshu/legion_s3d_tdb/s3d/source/drivers/main.f90:8:main [0x682f] ========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/s3d.x ========= Host Frame:../csu/libc-start.c:342:__libc_start_main [0x24083] ========= in /lib/x86_64-linux-gnu/libc.so.6 ========= Host Frame:_start [0x686e] ========= in /scratch2/seshu/legion_s3d_tdb/Shell1D/DBO_Test_1/s3d.x
I'm running S3D through cuda-memcheck and I'm seeing the following error before the main task even starts: