Hi
an error of cudaErrorIllegalAddress was occurred when I tried to decode utterances with my own customized batched-wav-nnet3-cude.cc. the only difference between my customized code and the original is that my code is a JNI toy, and all config variables are assigned in the code than command arguments. the sequence code runs ok until the line
valid = cuda_pipeline->GetLattice(key, &clat);
any help is appreciate.
logs
WARNING ([5.5]:SelectGpuId():cu-device.cc:221) Not in compute-exclusive mode. Suggestion: use 'nvidia-smi -c 3' to set compute exclusive mode
LOG ([5.5]:SelectGpuIdAuto():cu-device.cc:349) Selecting from 6 GPUs
LOG ([5.5]:SelectGpuIdAuto():cu-device.cc:364) cudaSetDevice(0): Tesla V100-PCIE-32GB free:20292M, used:12210M, total:32502M, free/total:0.624334
LOG ([5.5]:SelectGpuIdAuto():cu-device.cc:364) cudaSetDevice(1): Tesla V100-PCIE-32GB free:20276M, used:12226M, total:32502M, free/total:0.623841
LOG ([5.5]:SelectGpuIdAuto():cu-device.cc:364) cudaSetDevice(2): Tesla V100-PCIE-32GB free:20292M, used:12210M, total:32502M, free/total:0.624334
LOG ([5.5]:SelectGpuIdAuto():cu-device.cc:364) cudaSetDevice(3): Tesla V100-PCIE-32GB free:20292M, used:12210M, total:32502M, free/total:0.624334
LOG ([5.5]:SelectGpuIdAuto():cu-device.cc:364) cudaSetDevice(4): Tesla V100-PCIE-32GB free:32070M, used:432M, total:32502M, free/total:0.986709
LOG ([5.5]:SelectGpuIdAuto():cu-device.cc:364) cudaSetDevice(5): Tesla V100-PCIE-32GB free:32070M, used:432M, total:32502M, free/total:0.986709
LOG ([5.5]:SelectGpuIdAuto():cu-device.cc:411) Trying to select device: 4 (automatically), mem_ratio: 0.986709
LOG ([5.5]:SelectGpuIdAuto():cu-device.cc:430) Success selecting device 4 free mem ratio: 0.986709
LOG ([5.5]:FinalizeActiveGpu():cu-device.cc:284) The active GPU is [4]: Tesla V100-PCIE-32GB free:31974M, used:528M, total:32502M, free/total:0.983755 version 7.0
LOG ([5.5]:Initialize():batched-threaded-nnet3-cuda-pipeline.cc:32) BatchedThreadedNnet3CudaPipeline Initialize with 1 control threads, 5 worker threads and batch size 50
LOG ([5.5]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG ([5.5]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG ([5.5]:CheckAndFixConfigs():nnet-batch-compute.cc:363) Increasing --frames-per-chunk from 50 to 51 to make it a multiple of --frame-subsampling-factor=3
******** initialize cuda_pipeline ********
**** Read Wav ****
**** scp:/data1/wjb/kaldi/egs/aishell/s5/data/test/wav-100.scp ****
**** Enter In While ****
**** Start GetLattice ****
LOG ([5.5]:ExecuteWorker():batched-threaded-nnet3-cuda-pipeline.cc:611) Error unrecoverable cuda decoder error 'cuda-decoder-kernels.cu:1698 :cudaErrorIllegalAddress'
ASSERTION_FAILED ([5.5]:ExecuteWorker():batched-threaded-nnet3-cuda-pipeline.cc:613) Assertion failed: (UNRECOVERABLE_EXCEPTION)
[ Stack-Trace: ]
kaldi::MessageLogger::LogMessage() const
kaldi::KaldiAssertFailure_(char const*, char const*, int, char const*)
kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::ExecuteWorker(int)
std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::*)(int)> (kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline*, int)> >::_M_run()
clone
Aborted (core dumped)
Hi an error of
cudaErrorIllegalAddress
was occurred when I tried to decode utterances with my own customizedbatched-wav-nnet3-cude.cc
. the only difference between my customized code and the original is that my code is a JNI toy, and all config variables are assigned in the code than command arguments. the sequence code runs ok until the linevalid = cuda_pipeline->GetLattice(key, &clat);
any help is appreciate.logs
and the codes