Closed r-barnes closed 6 years ago
This line
INFO - Processing 250- vs 500-sequences
Also seems odd, both input files are pointing to the same file, which should lead to a 250 v 250 comparison.
Switching the end of _execute_calculate_score_kernel
in SmithWatermanCuda.py
to read:
except Exception as exception:
self.logger.error('Warning: {0}\nContinuing calculation...'.format(exception))
raise exception
gives:
DEBUG - Compiling cuda code.
DEBUG - Converting score to string...
DEBUG - Allocated: 15215.931701660156MB of memory
DEBUG - At sequence: 0 of 250, length = 28
DEBUG - Calculating scores.
ERROR - Warning: cuCtxSynchronize failed: an illegal memory access was encountered
Continuing calculation...
ERROR - cuCtxSynchronize failed: an illegal memory access was encountered
Traceback (most recent call last):
File "./implementations/warris2018/pypaswas.py", line 11, in <module>
ppw.run()
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/pypaswasall.py", line 235, in run
results.extend(self.program.process(query_sequences, target_sequences, self))
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/Core/Programs.py", line 82, in process
results = self.smith_waterman.align_sequences(records_seqs, targets, target_index)
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/Core/SmithWaterman.py", line 530, in align_sequences
self._calculate_score()
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/Core/SmithWaterman.py", line 609, in _calculate_score
self._execute_calculate_score_kernel(number_of_blocks, idx, idy)
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/Core/SmithWatermanCuda.py", line 226, in _execute_calculate_score_kernel
raise exception
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/Core/SmithWatermanCuda.py", line 222, in _execute_calculate_score_kernel
driver.Context.synchronize() #@UndefinedVariable @IgnorePep8
pycuda._driver.LogicError: cuCtxSynchronize failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuModuleUnload failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFreeHost failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFreeHost failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
Thanks for the information! I have never seen this problem before.
But first the easy part:
INFO - Processing 250- vs 500-sequences
With DNA sequences the target sequences will automatically be aligned in reverse complement too. This is located in pyPaSWAS/pypaswasall.py lines 117/118 if you'd like to disable this. At some point I need to make a CL switch for this.
The memory issue is new. The first thing that stands out is the amount of memory you allocated: ~15Gb. The cards I used have never exceeded 4Gb. Although I did use ~64GB on an Intel CPU system. I'm not sure all pointers in the CUDA code are 64bits... Could you check two things for me?
Thanks, Sven
Thanks @swarris!
I tried using --maximum_memory_usage=0.1
, but didn't see a change in behaviour.
python3 ./implementations/warris2018/pypaswas.py -o zout --loglevel=DEBUG --outputformat=TXT -p aligner --filetype1=fasta --filetype2=fasta -O OVERRIDE_OUTPUT -M DNA-RNA --device=$device --recompile=T --short_sequences=F --framework=CUDA --maximum_memory_usage=0.1 $queryfile $databasefile
INFO - Initializing application...
DEBUG - Initializing Score...
DEBUG - Initializing score finished.
DEBUG - Initializing DnaRnaScore...
DEBUG - Creating matrix with parameters:
match_score: 5,
mismatch_score: -3,
gap_score: -5.0,
other_score: -1,
any_score: 0
DEBUG - Initializing DnaRnaScore finished.
INFO - Application initialized.
INFO - Setting program...
DEBUG - Initializing aligner...
DEBUG - Initializing hitlist...
DEBUG - Initializing hitlist OK.
DEBUG - Setting SW...
DEBUG - Using CUDA implementation
DEBUG - Initializing SmithWaterman.
INFO - No gap extension penalty detected: using original PaSWAS scoring algorithm
DEBUG - Going to initialize device... with number 0
DEBUG - Initializing device 0
DEBUG - Aligner initialized.
INFO - Program set.
DEBUG - Initializing hitlist...
DEBUG - Initializing hitlist OK.
INFO - Reading query sequences 0 1000000...
DEBUG - Initializing reader
path = /autofs/nccs-svm1_home1/spinyfan/crd-swgpu/data/ant-500.fasta
limitlength = 100000...
DEBUG - Initializing reader finished.
DEBUG - Reading from fasta file...
DEBUG - 250 sequences read.
DEBUG - Sorting records on length...
INFO - Query sequences OK.
INFO - Reading target sequences 0, 100000000...
DEBUG - Initializing reader
path = /autofs/nccs-svm1_home1/spinyfan/crd-swgpu/data/ant-500.fasta
limitlength = 100000...
DEBUG - Initializing reader finished.
DEBUG - Reading from fasta file...
DEBUG - 250 sequences read.
DEBUG - Sorting records on length...
INFO - Target sequences OK.
INFO - Processing 250- vs 500-sequences
DEBUG - Aligner processing...
DEBUG - At target: 0 of 500
DEBUG - Total memory on Device: 15983.75
DEBUG - Total memory on Device: 15983.75
DEBUG - Initializing hitlist...
DEBUG - Initializing hitlist OK.
DEBUG - Clearing device memory.
DEBUG - Total memory on Device: 15983.75
DEBUG - Compiling cuda code.
DEBUG - Converting score to string...
DEBUG - Allocated: 1746.7488174438477MB of memory
DEBUG - At sequence: 0 of 250, length = 10
DEBUG - Calculating scores.
ERROR - Warning: cuCtxSynchronize failed: an illegal memory access was encountered
Continuing calculation...
ERROR - cuCtxSynchronize failed: an illegal memory access was encountered
Traceback (most recent call last):
File "./implementations/warris2018/pypaswas.py", line 11, in <module>
ppw.run()
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/pypaswasall.py", line 235, in run
results.extend(self.program.process(query_sequences, target_sequences, self))
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/Core/Programs.py", line 82, in process
results = self.smith_waterman.align_sequences(records_seqs, targets, target_index)
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/Core/SmithWaterman.py", line 530, in align_sequences
self._calculate_score()
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/Core/SmithWaterman.py", line 609, in _calculate_score
self._execute_calculate_score_kernel(number_of_blocks, idx, idy)
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/Core/SmithWatermanCuda.py", line 226, in _execute_calculate_score_kernel
raise exception
File "/autofs/nccs-svm1_home1/spinyfan/crd-swgpu/implementations/warris2018/pyPaSWAS/Core/SmithWatermanCuda.py", line 222, in _execute_calculate_score_kernel
driver.Context.synchronize() #@UndefinedVariable @IgnorePep8
pycuda._driver.LogicError: cuCtxSynchronize failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFreeHost failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFreeHost failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuModuleUnload failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
Working on trying the OpenCL framework now.
It doesn't appear as though OpenCL is available on my machine :-/
It doesn't appear as though OpenCL is available on my machine :-/
That is odd. For NVIDIA GPUs OpenCL should be available by default: https://developer.nvidia.com/opencl Did you try to run another CUDA or OpenCL programs? Does the example from pyCUDA run for instance?
I'll try to update my system to cuda 9.2.
The pyCUDA example runs without issue. I'm inquiring about OpenCL with the sysadmins.
My VM is connected to an GRID K2, but this GPU is not supported for cuda 9... So I had to revert to cuda 7.5. I'll try to find another machine to test it on.
Could you share your data with me so I can test it better?
Thanks!
On a fresh install Ubuntu 18.04, CUDA 9.1, GeForce GTX 650 and pyCUDA/pyOpenCL I can use the data found in the 'data' folder without a glitch.
Has it been resolved?
I'm running PyPaSWAS as follows:
I get several error messages:
I'm running with CUDA 9.0.69, Anaconda Python 3.5.5, numpy 1.14.3, biopython 1.71. The system has a Tesla P100 and 15215.931701660156MB of 15983.75MB are being allocated (maybe there's an issue with trying to allocate a millionth of a byte? ;-) )
Any thoughts as to what might be going on?