search

ContinuumIO / anaconda-issues

Anaconda issue tracking
646 stars 220 forks source link

Fatal Python error: Aborted #11446

Open rohitlal125555 opened 4 years ago

rohitlal125555 commented 4 years ago

Actual Behavior

Python crashes with Fatal Python error: Aborted

Expected Behavior

Should run continuously forever (at least for months without any restart required) with no crash errors.

Detailed Description

I've developed an image processing script that does gamma correction (on cpu), sobel(on nvidia gpu). But, strangely the program is crashing frequently where the frequency lies anywhere between 4 hours to 14 hours. I've no clue how to debug this error as it gives only one error message in stack trace "Fatal Python Error". Packages/Libraries used: numba.cuda, numpy, sklearn.cluster, cv2, multiprocessing.Pool Any idea how to debug/resolve this python crash error. Please see the below stack trace of the error. ``` ======= Backtrace: ========= INFO:cuda_img_process:53760 /lib64/libc.so.6(+0x816b9)[0x7f7d483f96b9] /opt/anaconda3/lib/python3.6/site-packages/zmq/backend/cython/../../../../../libzmq.so.5(+0x328ee)[0x7f7d2808a8ee] /opt/anaconda3/lib/python3.6/site-packages/zmq/backend/cython/../../../../../libzmq.so.5(+0x6631a)[0x7f7d280be31a] /opt/anaconda3/lib/python3.6/site-packages/zmq/backend/cython/../../../../../libzmq.so.5(+0x5407a)[0x7f7d280ac07a] /opt/anaconda3/lib/python3.6/site-packages/zmq/backend/cython/../../../../../libzmq.so.5(+0x27561)[0x7f7d2807f561] /opt/anaconda3/lib/python3.6/site-packages/zmq/backend/cython/../../../../../libzmq.so.5(+0x5df86)[0x7f7d280b5f86] /lib64/libpthread.so.0(+0x7ea5[2019-11-11 20:39:43,323] -- INFO - cuda_roi_image_process_normalize_writeimage_api2.py -- receive_image_for_image_process -- Line no - 370 -- CUDA Device : 2 /lib64/libc.so.6(clone+0x6d)[0x7f7d484768cd] ======= Memory map: ======== INFO:cuda_img_process:CUDA Device : 2 [2019-11-11 20:39:43,323] -- INFO - cuda_roi_image_process_normalize_writeimage_api2.py -- receive_image_for_image_process -- Line no - 382 -- Image Recieved for Image Processing : 53761 : 111129_1_261.bmp INFO:cuda_img_process:Image Recieved for Image Processing : 53761 : 111129_1_261.bmp 200000000-200200000 rw-s 00000000 00:05 92844 /dev/nvidiactl 200200000-200600000 ---p 00000000 00:00 0 200600000-200800000 rw-s 00000000 00:05 92844 /dev/nvidiactl 200800000-200a00000 rw-s 00000000 00:05 92847 /dev/nvidia2 200a00000-205200000 rw-s 00000000 00:05 92844 /dev/nvidiactl 205200000-205400000 rw-s 00000000 00:05 92847 /dev/nvidia2 205400000-206400000 ---p 00000000 00:00 0 206400000-206600000 rw-s 00000000 00:05 92844 /dev/nvidiactl 206600000-206800000 rw-s 00000000 00:05 92844 /dev/nvidiactl 206800000-206a00000 rw-s 206800000 00:05 99322 /dev/nvidia-uvm 206a00000-206c00000 ---p 00000000 00:00 0 206c00000-206e00000 rw-s 00000000 00:05 92844 /dev/nvidiactl 206e00000-207000000 rw-s 00000000 00:04 142114037 /dev/zero (deleted) 207000000-600200000 ---p 00000000 00:00 0 10000000000-10004000000 ---p 00000000 00:00 0 55858734d000-5585873a4000 r--p 00000000 fd:00 843121090 /opt/anaconda3/bin/python3.6 5585873a4000-55858756b000 r-xp 00057000 fd:00 843121090 /opt/anaconda3/bin/python3.6 55858756b000-558587608000 r--p 0021e000 fd:00 843121090 /opt/anaconda3/bin/python3.6 558587609000-55858760c000 r--p 002bb000 fd:00 843121090 /opt/anaconda3/bin/python3.6 55858760c000-55858766f000 rw-p 002be000 fd:00 843121090 /opt/anaconda3/bin/python3.6 55858766f000-5585876a0000 rw-p 00000000 00:00 0 5585886f6000-55858ab8b000 rw-p 00000000 00:00 0 [heap] 55858ab8b000-55858abab000 rw-p 00000000 00:00 0 [heap] 55858abab000-55858b5da000 rw-p 00000000 00:00 0 [heap] 55858b5da000-55858b5fa000 rw-p 00000000 00:00 0 [heap] 55858b5fa000-55858c05f000 rw-p 00000000 00:00 0 [heap] 55858c05f000-55858c07b000 rw-p 00000000 00:00 0 [heap] 55858c07b000-55858cb32000 rw-p 00000000 00:00 0 [heap] 55858cb32000-55858d5b1000 rw-p 00000000 00:00 0 [heap] 55858d5b1000-55858de72000 rw-p 00000000 00:00 0 [heap] 55858de72000-55858e00b000 rw-p 00000000 00:00 0 [heap] 55858e00b000-55858e039000 rw-p 00000000 00:00 0 [heap] 55858e039000-55858ea94000 rw-p 00000000 00:00 0 [heap] 55858ea94000-55858ea98000 rw-p 00000000 00:00 0 [heap] 55858ea98000-55858ffcd000 rw-p 00000000 00:00 0 [heap] 55858ffcd000-55858ffdf000 rw-p 00000000 00:00 0 [heap] 55858ffdf000-558590b16000 rw-p 00000000 00:00 0 [heap] 558590b16000-558592466000 rw-p 00000000 00:00 0 [heap] 558592466000-55859273b000 rw-p 00000000 00:00 0 [heap] 55859273b000-558593969000 rw-p 00000000 00:00 0 [heap] 558593969000-5585943c9000 rw-p 00000000 00:00 0 [heap] 5585943c9000-558594e70000 rw-p 00000000 00:00 0 [heap] 558594e70000-55859529c000 rw-p 00000000 00:00 0 [heap] 55859529c000-55859635b000 rw-p 00000000 00:00 0 [heap] 55859635b000-558596d9d000 rw-p 00000000 00:00 0 [heap] 558596d9d000-558598287000 rw-p 00000000 00:00 0 [heap] 558598287000-558599132000 rw-p 00000000 00:00 0 [heap] 558599132000-55859aedd000 rw-p 00000000 00:00 0 [heap] 55859aedd000-55859aefd000 rw-p 00000000 00:00 0 [heap] 7f77b8000000-7f77b8001000 rw-p 00000000 00:00 0 7f77b8001000-7f77b94fb000 rw-p 00000000 00:00 0 7f77b94fb000-7f77ba3cd000 rw-p 00000000 00:00 0 7f77ba3cd000-7f77bb29f000 rw-p 00000000 00:00 0 [2019-11-11 20:39:43,324] -- INFO - cuda_roi_image_process_normalize_writeimage_api2.py -- cuda_process_roi_extract -- Line no - 766 -- Splitted image received for processing : 111129_1_261_0_0_0.bmp 7f77bb29f000-7f77bb49c000 rw-p 00000000 00:00 0 7f77bb49c000-7f77bbf39000 rw-p 00000000 00:00 0 7f77bbf39000-7f77bbf93000 rw-p 00000000 00:00 0 7f77bbf93000-7f77bc000000 ---p 00000000 00:00 0 7f77c8000000-7f77c8021000 rw-p 00000000 00:00 0 7f77c8021000-7f77cc000000 ---p 00000000 00:00 0 7f77cc000000-7f77cc021000 rw-p 00000000 00:00 0 7f77cc021000-7f77d0000000 ---p 00000000 00:00 0 7f77d0000000-7f77d0021000 rw-p 00000000 00:00 0 7f77d0021000-7f77d4000000 ---p 00000000 00:00 0 7f77d4000000-7f77d4021000 rw-p 00000000 00:00 0 7f77d4021000-7f77d8000000 ---p 00000000 00:00 0 7f77d8000000-7f77d8021000 rw-p 00000000 00:00 0 7f77d8021000-7f77dc000000 ---p 00000000 00:00 0 7f77dc000000-7f77dc021000 rw-p 00000000 00:00 0 7f77dc021000-7f77e0000000 ---p 00000000 00:00 0 7f77e1ff0000-7f77e1ff1000 ---p 00000000 00:00 0 7f77e1ff1000-7f77e27f4000 rwxp 00000000 00:00 0 [stack:398300] 7f77e27f4000-7f77e27f5000 ---p 00000000 00:00 0 7f77e27f5000-7f77e2ff8000 rwxp 00000000 00:00 0 INFO:cuda_img_process:Splitted image received for processing : 111129_1_261_0_0_0.bmp [stack:398299] 7f77e2ff8000-7f77e2ff9000 ---p 00000000 00:00 0 7f77e2ff9000-7f77e37fc000 rwxp 00000000 00:00 0 [stack:398298] 7f77e37fc000-7f77e37fd000 ---p 00000000 00:00 0 7f77e37fd000-7f77e4000000 rwxp 00000000 00:00 0 [stack:398297] 7f77e4000000-7f77e4021000 rw-p 00000000 00:00 0 7f77e4021000-7f77e8000000 ---p 00000000 00:00 0 7f77e8000000-7f77e8021000 rw-p 00000000 00:00 0 7f77e8021000-7f77ec000000 ---p 00000000 00:00 0 7f77ec000000-7f77ec021000 rw-p 00000000 00:00 0 7f77ec021000-7f77f0000000 ---p 00000000 00:00 0 7f77f0000000-7f77f0021000 rw-p 00000000 00:00 0 7f77f0021000-7f77f4000000 ---p 00000000 00:00 0 7f77f4000000-7f77f4021000 rw-p 00000000 00:00 0 7f77f4021000-7f77f8000000 ---p 00000000 00:00 0 7f77f87e4000-7f77f87e5000 ---p 00000000 00:00 0 7f77f87e5000-7f77f8fe8000 rwxp 00000000 00:00 0 [stack:398296] 7f77f8fe8000-7f77f8fe9000 ---p 00000000 00:00 0 7f77f8fe9000-7f77f97ec000 rwxp 00000000 00:00 0 [stack:398295] 7f77f97ec000-7f77f97ed000 ---p 00000000 00:00 0 7f77f97ed000-7f77f9ff0000 rwxp 00000000 00:00 0 [stack:398294] 7f77f9ff0000-7f77f9ff1000 ---p 00000000 00:00 0 7f77f9ff1000-7f77fa7f4000 rwxp 00000000 00:00 0 [stack:398290] 7f77fa7f4000-7f77fa7f5000 ---p 00000000 00:00 0 7f77fa7f5000-7f77faff8000 rwxp 00000000 00:00 0 [stack:398287] 7f77faff8000-7f77faff9000 ---p 00000000 00:00 0 7f77faff9000-7f77fb7fc000 rwxp 00000000 00:00 0 [stack:398284] 7f77fb7fc000-7f77fb7fd000 ---p 00000000 00:00 0 7f77fb7fd000-7f77fc000000 rwxp 00000000 00:00 0 [stack:398281] 7f77fc000000-7f77fc021000 rw-p 00000000 00:00 0 7f77fc021000-7f7800000000 ---p 00000000 00:00 0 7f7800000000-7f7800021000 rw-p 00000000 00:00 0 7f7800021000-7f7804000000 ---p 00000000 00:00 0 7f7804000000-7f7804021000 rw-p 00000000 00:00 0 7f7804021000-7f7808000000 ---p 00000000 00:00 0 7f7808000000-7f7808021000 rw-p 00000000 00:00 0 7f7808021000-7f780c000000 ---p 00000000 00:00 0 7f780c000000-7f780c021000 rw-p 00000000 00:00 0 7f780c021000-7f7810000000 ---p 00000000 00:00 0 7f7810000000-7f7810021000 rw-p 00000000 00:00 0 7f7810021000-7f7814000000 ---p 00000000 00:00 0 7f7814000000-7f7814021000 rw-p 00000000 00:00 0 7f7814021000-7f7818000000 ---p 00000000 00:00 0 7f7818000000-7f7818021000 rw-p 00000000 00:00 0 7f7818021000-7f781c000000 ---p 00000000 00:00 0 7f781c7e4000-7f781c7e5000 ---p 00000000 00:00 0 7f781c7e5000-7f781cfe8000 rwxp 00000000 00:00 0 [stack:398278] 7f781cfe8000-7f781cfe9000 ---p 00000000 00:00 0 7f781cfe9000-7f781d7ec000 rwxp 00000000 00:00 0 [stack:398275] 7f781d7ec000-7f781d7ed000 ---p 00000000 00:00 0 7f781d7ed000-7f781dff0000 rwxp 00000000 00:00 0 [stack:398273] 7f781dff0000-7f781dff1000 ---p 00000000 00:00 0 7f781dff1000-7f781e7f4000 rwxp 00000000 00:00 0 [stack:398269] 7f781e7f4000-7f781e7f5000 ---p 00000000 00:00 0 7f781e7f5000-7f781eff8000 rwxp 00000000 00:00 0 [stack:398265] 7f781eff8000-7f781eff9000 ---p 00000000 00:00 0 7f781eff9000-7f781f7fc000 rwxp 00000000 00:00 0 [stack:398260] 7f781f7fc000-7f781f7fd000 ---p 00000000 00:00 0 7f781f7fd000-7f7820000000 rwxp 00000000 00:00 0 [stack:398257] 7f7820000000-7f7820021000 rw-p 00000000 00:00 0 7f7820021000-7f7824000000 ---p 00000000 00:00 0 7f7824000000-7f7824021000 rw-p 00000000 00:00 0 7f7824021000-7f7828000000 ---p 00000000 00:00 0 7f7828000000-7f7828021000 rw-p 00000000 00:00 0 7f7828021000-7f782c000000 ---p 00000000 00:00 0 7f782c000000-7f782c021000 rw-p 00000000 00:00 0 7f782c021000-7f7830000000 ---p 00000000 00:00 0 7f7830000000-7f7830021000 rw-p 00000000 00:00 0 7f7830021000-7f7834000000 ---p 00000000 00:00 0 7f7834000000-7f7834021000 rw-p 00000000 00:00 0 7f7834021000-7f7838000000 ---p 00000000 00:00 0 7f7838000000-7f7838021000 rw-p 00000000 00:00 0 7f7838021000-7f783c000000 ---p 00000000 00:00 0 7f783c000000-7f783c021000 rw-p 00000000 00:00 0 7f783c021000-7f7840000000 ---p 00000000 00:00 0 7f7840000000-7f7840021000 rw-p 00000000 00:00 0 7f7840021000-7f7844000000 ---p 00000000 00:00 0 7f7844000000-7f7844021000 rw-p 00000000 00:00 0 7f7844021000-7f7848000000 ---p 00000000 00:00 0 7f7848000000-7f7848021000 rw-p 00000000 00:00 0 7f7848021000-7f784c000000 ---p 00000000 00:00 0 7f784c000000-7f784c021000 rw-p 00000000 00:00 0 7f784c021000-7f7850000000 ---p 00000000 00:00 0 7f7850000000-7f7850021000 rw-p 00000000 00:00 0 7f7850021000-7f7854000000 ---p 00000000 00:00 0 7f7854000000-7f7854021000 rw-p 00000000 00:00 0 7f7854021000-7f7858000000 ---p 00000000 00:00 0 7f7858000000-7f7858021000 rw-p 00000000 00:00 0 7f7858021000-7f785c000000 ---p 00000000 00:00 0 7f785c000000-7f785c021000 rw-p 00000000 00:00 0 7f785c021000-7f7860000000 ---p 00000000 00:00 0 7f7860000000-7f7860021000 rw-p 00000000 00:00 0 7f7860021000-7f7864000000 ---p 00000000 00:00 0 7f7864000000-7f7864021000 rw-p 00000000 00:00 0 7f7864021000-7f7868000000 ---p 00000000 00:00 0 7f7868000000-7f7868021000 rw-p 00000000 00:00 0 7f7868021000-7f786c000000 ---p 00000000 00:00 0 7f786c000000-7f786c021000 rw-p 00000000 00:00 0 7f786c021000-7f7870000000 ---p 00000000 00:00 0 7f7870000000-7f7870021000 rw-p 00000000 00:00 0 7f7870021000-7f7874000000 ---p 00000000 00:00 0 7f7874000000-7f7874021000 rw-p 00000000 00:00 0 7f7874021000-7f7878000000 ---p 00000000 00:00 0 7f7878000000-7f7878021000 rw-p 00000000 00:00 0 7f7878021000-7f787c000000 ---p 00000000 00:00 0 7f787c000000-7f787c021000 rw-p 00000000 00:00 0 7f787c021000-7f7880000000 ---p 00000000 00:00 0 7f7880000000-7f7880021000 rw-p 00000000 00:00 0 7f7880021000-7f7884000000 ---p 00000000 00:00 0 7f7884000000-7f7884021000 rw-p 00000000 00:00 0 7f7884021000-7f7888000000 ---p 00000000 00:00 0 7f7888000000-7f7888021000 rw-p 00000000 00:00 0 7f7888021000-7f788c000000 ---p 00000000 00:00 0 7f788c000000-7f788c021000 rw-p 00000000 00:00 0 7f788c021000-7f7890000000 ---p 00000000 00:00 0 7f7890000000-7f7890021000 rw-p 00000000 00:00 0 7f7890021000-7f7894000000 ---p 00000000 00:00 0 7f7894000000-7f7894021000 rw-p 00000000 00:00 0 7f7894021000-7f7898000000 ---p 00000000 00:00 0 7f7898000000-7f7898021000 rw-p 00000000 00:00 0 7f7898021000-7f789c000000 ---p 00000000 00:00 0 7f789c000000-7f789c021000 rw-p 00000000 00:00 0 7f789c021000-7f78a0000000 ---p 00000000 00:00 0 7f78a0000000-7f78a0021000 rw-p 00000000 00:00 0 7f78a0021000-7f78a4000000 ---p 00000000 00:00 0 7f78a4000000-7f78a4021000 rw-p 00000000 00:00 0 7f78a4021000-7f78a8000000 ---p 00000000 00:00 0 7f78a8000000-7f78a8021000 rw-p 00000000 00:00 0 7f78a8021000-7f78ac000000 ---p 00000000 00:00 0 7f78ac000000-7f78ac021000 rw-p 00000000 00:00 0 7f78ac021000-7f78b0000000 ---p 00000000 00:00 0 7f78b0000000-7f78b0021000 rw-p 00000000 00:00 0 7f78b0021000-7f78b4000000 ---p 00000000 00:00 0 7f78b4000000-7f78b4021000 rw-p 00000000 00:00 0 7f78b4021000-7f78b8000000 ---p 00000000 00:00 0 7f78b8000000-7f78b8021000 rw-p 00000000 00:00 0 7f78b8021000-7f78bc000000 ---p 00000000 00:00 0 7f78bc000000-7f78bc021000 rw-p 00000000 00:00 0 7f78bc021000-7f78c0000000 ---p 00000000 00:00 0 7f78c0000000-7f78c0021000 rw-p 00000000 00:00 0 7f78c0021000-7f78c4000000 ---p 00000000 00:00 0 7f78c4000000-7f78c4021000 rw-p 00000000 00:00 0 7f78c4021000-7f78c8000000 ---p 00000000 00:00 0 7f78c8000000-7f78c8021000 rw-p 00000000 00:00 0 7f78c8021000-7f78cc000000 ---p 00000000 00:00 0 7f78cc000000-7f78cc021000 rw-p 00000000 00:00 0 7f78cc021000-7f78d0000000 ---p 00000000 00:00 0 7f78d0000000-7f78d0021000 rw-p 00000000 00:00 0 7f78d0021000-7f78d4000000 ---p 00000000 00:00 0 7f78d4000000-7f78d4021000 rw-p 00000000 00:00 0 7f78d4021000-7f78d8000000 ---p 00000000 00:00 0 7f78d8000000-7f78d8021000 rw-p 00000000 00:00 0 7f78d8021000-7f78dc000000 ---p 00000000 00:00 0 7f78dc000000-7f78dc021000 rw-p 00000000 00:00 0 7f78dc021000-7f78e0000000 ---p 00000000 00:00 0 7f78e0000000-7f78e0021000 rw-p 00000000 00:00 0 7f78e0021000-7f78e4000000 ---p 00000000 00:00 0 7f78e4000000-7f78e4021000 rw-p 00000000 00:00 0 7f78e4021000-7f78e8000000 ---p 00000000 00:00 0 7f78e8000000-7f78e8021000 rw-p 00000000 00:00 0 7f78e8021000-7f78ec000000 ---p 00000000 00:00 0 7f78ec7da000-7f78ec7db000 ---p 00000000 00:00 0 7f78ec7db000-7f78ecfde000 rwxp 00000000 00:00 0 [stack:398250] 7f78ecfde000-7f78ecfdf000 ---p 00000000 00:00 0 7f78ecfdf000-7f78ed7e2000 rwxp 00000000 00:00 0 [stack:398244] 7f78ed7e2000-7f78ed7e3000 ---p 00000000 00:00 0 7f78ed7e3000-7f78edfe6000 rwxp 00000000 00:00 0 [stack:398243] 7f78edfe6000-7f78edfe7000 ---p 00000000 00:00 0 7f78edfe7000-7f78ee7ea000 rwxp 00000000 00:00 0 [stack:398242] 7f78ee7ea000-7f78ee7eb000 ---p 00000000 00:00 0 7f78ee7eb000-7f78eefee000 rwxp 00000000 00:00 0 [stack:398241] 7f78eefee000-7f78eefef000 ---p 00000000 00:00 0 7f78eefef000-7f78ef7ef000 rwxp 00000000 00:00 0 [stack:397864] 7f78ef7ef000-7f78ef7f0000 ---p 00000000 00:00 0 7f78ef7f0000-7f78efff0000 rwxp 00000000 00:00 0 [stack:397863] 7f78efff0000-7f78efff1000 ---p 00000000 00:00 0 7f78efff1000-7f78f07f1000 rwxp 00000000 00:00 0 [stack:397862] 7f78f07f1000-7f78f07f2000 ---p 00000000 00:00 0 7f78f07f2000-7f78f0ff2000 rwxp 00000000 00:00 0 [stack:397861] 7f78f0ff2000-7f78f0ff3000 ---p 00000000 00:00 0 7f78f0ff3000-7f78f17f3000 rwxp 00000000 00:00 0 [stack:397860] 7f78f17f3000-7f78f17f4000 ---p 00000000 00:00 0 7f78f17f4000-7f78f1ff4000 rwxp 00000000 00:00 0 [stack:397859] 7f78f1ff4000-7f78f1ff5000 ---p 00000000 00:00 0 7f78f1ff5000-7f78f27f5000 rwxp 00000000 00:00 0 [stack:397858] 7f78f27f5000-7f78f27f6000 ---p 00000000 00:00 0 7f78f27f6000-7f78f2ff6000 rwxp 00000000 00:00 0 [stack:397857] 7f78f2ff6000-7f78f2ff7000 ---p 00000000 00:00 0 7f78f2ff7000-7f78f37f7000 rwxp 00000000 00:00 0 [stack:397856] 7f78f37f7000-7f78f37f8000 ---p 00000000 00:00 0 7f78f37f8000-7f78f3ff8000 rwxp 00000000 00:00 0 [stack:397855] 7f78f3ff8000-7f78f3ff9000 ---p 00000000 00:00 0 7f78f3ff9000-7f78f47f9000 rwxp 00000000 00:00 0 [stack:397854] 7f78f47f9000-7f78f47fa000 ---p 00000000 00:00 0 7f78f47fa000-7f78f4ffa000 rwxp 00000000 00:00 0 [stack:397853] 7f78f4ffa000-7f78f4ffb000 ---p 00000000 00:00 0 7f78f4ffb000-7f78f57fb000 rwxp 00000000 00:00 0 [stack:397852] 7f78f57fb000-7f78f57fc000 ---p 00000000 00:00 0 7f78f57fc000-7f78f5ffc000 rwxp 00000000 00:00 0 [stack:397851] 7f78f5ffc000-7f78f5ffd000 ---p 00000000 00:00 0 7f78f5ffd000-7f78f67fd000 rwxp 00000000 00:00 0 [stack:397850] 7f78f67fd000-7f78f67fe000 ---p 00000000 00:00 0 7f78f67fe000-7f78f6ffe000 rwxp 00000000 00:00 0 [stack:397849] 7f78f6ffe000-7f78f6fff000 ---p 00000000 00:00 0 7f78f6fff000-7f78f77ff000 rwxp 00000000 00:00 0 [stack:397848] 7f78f77ff000-7f78f7800000 ---p 00000000 00:00 0 7f78f7800000-7f78f8000000 rwxp 00000000 00:00 0 [stack:397847] 7f78f8000000-7f78f8021000 rw-p 00000000 00:00 0 7f78f8021000-7f78fc000000 ---p 00000000 00:00 0 7f78fc000000-7f78fc021000 rw-p 00000000 00:00 0 7f78fc021000-7f7900000000 ---p 00000000 00:00 0 7f7900000000-7f7900021000 rw-p 00000000 00:00 0 7f7900021000-7f7904000000 ---p 00000000 00:00 0 7f7904000000-7f7904021000 rw-p 00000000 00:00 0 7f7904021000-7f7908000000 ---p 00000000 00:00 0 7f7908000000-7f7908021000 rw-p 00000000 00:00 0 7f7908021000-7f790c000000 ---p 00000000 00:00 0 7f790c000000-7f790c001000 rw-p 00000000 00:00 0 7f790c001000-7f790ced3000 rw-p 00000000 00:00 0 7f790ced3000-7f790cede000 rw-p 00000000 00:00 0 7f790cede000-7f790ddb1000 rw-p 00000000 00:00 0 7f790ddb1000-7f790ec83000 rw-p 00000000 00:00 0 7f790ec83000-7f790fcb6000 rw-p 00000000 00:00 0 7f790fcb6000-7f790ff76000 rw-p 00000000 00:00 0 7f790ff76000-7f7910000000 ---p 00000000 00:00 0 7f7910000000-7f7910021000 rw-p 00000000 00:00 0 7f7910021000-7f7914000000 ---p 00000000 00:00 0 7f7914000000-7f7914021000 rw-p 00000000 00:00 0 7f7914021000-7f7918000000 ---p 00000000 00:00 0 7f7918000000-7f7918021000 rw-p 00000000 00:00 0 7f7918021000-7f791c000000 ---p 00000000 00:00 0 7f791c000000-7f791c021000 rw-p 00000000 00:00 0 7f791c021000-7f7920000000 ---p 00000000 00:00 0 7f7920000000-7f7920021000 rw-p 00000000 00:00 0 7f7920021000-7f7924000000 ---p 00000000 00:00 0 7f7924000000-7f7924021000 rw-p 00000000 00:00 0 7f7924021000-7f7928000000 ---p 00000000 00:00 0 7f7928000000-7f7928021000 rw-p 00000000 00:00 0 7f7928021000-7f792c000000 ---p 00000000 00:00 0 7f792c000000-7f792c021000 rw-p 00000000 00:00 0 7f792c021000-7f7930000000 ---p 00000000 00:00 0 7f7930000000-7f7930021000 rw-p 00000000 00:00 0 7f7930021000-7f7934000000 ---p 00000000 00:00 0 7f7934000000-7f7934021000 rw-p 00000000 00:00 0 7f7934021000-7f7938000000 ---p 00000000 00:00 0 7f7938000000-7f7938021000 rw-p 00000000 00:00 0 7f7938021000-7f793c000000 ---p 00000000 00:00 0 7f793c000000-7f793c021000 rw-p 00000000 00:00 0 7f793c021000-7f7940000000 ---p 00000000 00:00 0 7f7940000000-7f7940021000 rw-p 00000000 00:00 0 7f7940021000-7f7944000000 ---p 00000000 00:00 0 7f7944000000-7f7944021000 rw-p 00000000 00:00 0 7f7944021000-7f7948000000 ---p 00000000 00:00 0 7f7948000000-7f7948021000 rw-p 00000000 00:00 0 7f7948021000-7f794c000000 ---p 00000000 00:00 0 7f794c000000-7f794c021000 rw-p 00000000 00:00 0 7f794c021000-7f7950000000 ---p 00000000 00:00 0 7f7950000000-7f7950021000 rw-p 00000000 00:00 0 7f7950021000-7f7954000000 ---p 00000000 00:00 0 7f7954000000-7f7954021000 rw-p 00000000 00:00 0 7f7954021000-7f7958000000 ---p 00000000 00:00 0 7f7958000000-7f7958021000 rw-p 00000000 00:00 0 7f7958021000-7f795c000000 ---p 00000000 00:00 0 7f795c7f9000-7f795c7fa000 ---p 00000000 00:00 0 7f795c7fa000-7f795cffa000 rwxp 00000000 00:00 0 [stack:397846] 7f795cffa000-7f795cffb000 ---p 00000000 00:00 0 7f795cffb000-7f795d7fb000 rwxp 00000000 00:00 0 [stack:397845] 7f795d7fb000-7f795d7fc000 ---p 00000000 00:00 0 7f795d7fc000-7f795dffc000 rwxp 00000000 00:00 0 [stack:397844] 7f795dffc000-7f795dffd000 ---p 00000000 00:00 0 7f795dffd000-7f795e7fd000 rwxp 00000000 00:00 0 [stack:397843] 7f795e7fd000-7f795e7fe000 ---p 00000000 00:00 0 7f795e7fe000-7f795effe000 rwxp 00000000 00:00 0 [stack:397842] 7f795effe000-7f795efff000 ---p 00000000 00:00 0 Fatal Python error: Aborted Thread 0x00007f7d00ffd700 (most recent call first): File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 293 in safe_cuda_api_call File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 610 in clear File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 599 in add_item File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 964 in core File "/opt/anaconda3/lib/python3.6/site-packages/numba/utils.py", line 669 in __call__ File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1268 in free File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1382 in deref File "/opt/anaconda3/lib/python3.6/site-packages/numba/utils.py", line 669 in __call__ File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 477 in split_image_for_cuda File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 385 in receive_image_for_image_process File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 243 in on_post File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 440 in execute File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 172 in service File "/opt/anaconda3/lib/python3.6/site-packages/waitress/channel.py", line 356 in service File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 85 in handler_thread File "/opt/anaconda3/lib/python3.6/threading.py", line 864 in run File "/opt/anaconda3/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/opt/anaconda3/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x00007f7d017fe700 (most recent call first): File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 293 in safe_cuda_api_call File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 610 in clear File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 599 in add_item File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 964 in core File "/opt/anaconda3/lib/python3.6/site-packages/numba/utils.py", line 669 in __call__ File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1268 in free File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1382 in deref File "/opt/anaconda3/lib/python3.6/site-packages/numba/utils.py", line 669 in __call__ File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 477 in split_image_for_cuda File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 385 in receive_image_for_image_process File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 243 in on_post File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 440 in execute File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 172 in service File "/opt/anaconda3/lib/python3.6/site-packages/waitress/channel.py", line 356 in service File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 85 in handler_thread File "/opt/anaconda3/lib/python3.6/threading.py", line 864 in run File "/opt/anaconda3/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/opt/anaconda3/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x00007f7d01fff700 (most recent call first): File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 293 in safe_cuda_api_call File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 756 in allocator File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 741 in _attempt_allocation File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 758 in memalloc File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 102 in __init__ File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 631 in from_array_like File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/devicearray.py", line 693 in auto_device File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/api.py", line 110 in to_device File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/devices.py", line 225 in _require_cuda_context File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 786 in cuda_process_roi_extract File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 477 in split_image_for_cuda File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 385 in receive_image_for_image_process File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 243 in on_post File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 440 in execute File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 172 in service File "/opt/anaconda3/lib/python3.6/site-packages/waitress/channel.py", line 356 in service File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 85 in handler_thread File "/opt/anaconda3/lib/python3.6/threading.py", line 864 in run File "/opt/anaconda3/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/opt/anaconda3/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x00007f7d14f8f700 (most recent call first): File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 293 in safe_cuda_api_call File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 610 in clear File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 599 in add_item File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 964 in core File "/opt/anaconda3/lib/python3.6/site-packages/numba/utils.py", line 669 in __call__ File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1268 in free File "/opt/anaconda3/lib/python3.6/site-packages/numba/cuda/cudadrv/driver.py", line 1382 in deref File "/opt/anaconda3/lib/python3.6/site-packages/numba/utils.py", line 669 in __call__ File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 477 in split_image_for_cuda File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 385 in receive_image_for_image_process File "/home/admin/Downloads/falcon/SCRIPTS/cuda_roi_image_process_normalize_writeimage_api2.py", line 243 in on_post File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 440 in execute File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 172 in service File "/opt/anaconda3/lib/python3.6/site-packages/waitress/channel.py", line 356 in service File "/opt/anaconda3/lib/python3.6/site-packages/waitress/task.py", line 85 in handler_thread File "/opt/anaconda3/lib/python3.6/threading.py", line 864 in run File "/opt/anaconda3/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/opt/anaconda3/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x00007f7d177fe700 (most recent call first): File "/opt/anaconda3/lib/python3.6/multiprocessing/connection.py", line 379 in _recv File "/opt/anaconda3/lib/python3.6/multiprocessing/connection.py", line 407 in _recv_bytes File "/opt/anaconda3/lib/python3.6/multiprocessing/connection.py", line 250 in recv File "/opt/anaconda3/lib/python3.6/multiprocessing/pool.py", line 463 in _handle_results File "/opt/anaconda3/lib/python3.6/threading.py", line 864 in run File "/opt/anaconda3/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/opt/anaconda3/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x00007f7d17fff700 (most recent call first): File "/opt/anaconda3/lib/python3.6/threading.py", line 295 in wait File "/opt/anaconda3/lib/python3.6/queue.py", line 164 in get File "/opt/anaconda3/lib/python3.6/multiprocessing/pool.py", line 415 in _handle_tasks File "/opt/anaconda3/lib/python3.6/threading.py", line 864 in run File "/opt/anaconda3/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/opt/anaconda3/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x00007f7d1c833700 (most recent call first): File "/opt/anaconda3/lib/python3.6/multiprocessing/pool.py", line 406 in _handle_workers File "/opt/anaconda3/lib/python3.6/threading.py", line 864 in run File "/opt/anaconda3/lib/python3.6/threading.py", line 916 in _bootstrap_inner File "/opt/anaconda3/lib/python3.6/threading.py", line 884 in _bootstrap Thread 0x00007f7d48b5f740 (most recent call first): File "/opt/anaconda3/lib/python3.6/site-packages/waitress/wasyncore.py", line 152 in poll File "/opt/anaconda3/lib/python3.6/site-packages/waitress/wasyncore.py", line 222 in loop File "/opt/anaconda3/lib/python3.6/site-packages/waitress/server.py", line 323 in run File "/opt/anaconda3/lib/python3.6/site-packages/waitress/__init__.py", line 17 in serve File "/opt/anaconda3/lib/python3.6/site-packages/waitress/runner.py", line 279 in run File "/opt/anaconda3/bin/waitress-serve", line 10 in Aborted (core dumped) ```
Operating System:

Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo) Kernel: Linux 3.10.0-693.el7.x86_64 Architecture: x86-64

conda info
``` active environment : None user config file : /home/admin/.condarc populated config files : conda version : 4.7.12 conda-build version : 3.0.27 python version : 3.6.9.final.0 virtual packages : __cuda=10.1 base environment : /opt/anaconda3 (read only) channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /opt/anaconda3/pkgs /home/admin/.conda/pkgs envs directories : /home/admin/.conda/envs /opt/anaconda3/envs platform : linux-64 user-agent : conda/4.7.12 requests/2.22.0 CPython/3.6.9 Linux/3.10. 0-693.el7.x86_64 rhel/7.6 glibc/2.17 UID:GID : 1000:1000 netrc file : None offline mode : False ```
conda list --show-channel-urls
``` # packages in environment at /opt/anaconda3: # # Name Version Build Channel _tflow_select 2.1.0 gpu anaconda absl-py 0.8.0 py36_0 anaconda anaconda custom py36hbbc8b67_0 anaconda asn1crypto 0.24.0 py36_0 anaconda astor 0.8.0 py36_0 anaconda backports 1.0 py_2 anaconda backports.functools_lru_cache 1.5 py_2 anaconda backports.tempfile 1.0 py_1 anaconda backports.weakref 1.0.post1 py_1 anaconda beautifulsoup4 4.8.0 py36_0 anaconda blas 1.0 mkl anaconda bzip2 1.0.8 h7b6447c_0 anaconda c-ares 1.15.0 h7b6447c_1001 anaconda ca-certificates 2019.10.16 0 anaconda cairo 1.14.12 h8948797_3 anaconda certifi 2019.9.11 py36_0 anaconda cffi 1.12.3 py36h2e261b9_0 anaconda chardet 3.0.4 py36_1003 anaconda click 7.0 py36_0 anaconda cloudpickle 1.2.2 py_0 anaconda conda 4.7.12 py36_0 anaconda conda-build 3.0.27 py36h940a66d_0 defaults conda-package-handling 1.6.0 py36h7b6447c_0 anaconda conda-verify 3.4.2 py_1 anaconda cryptography 2.7 py36h1ba5d50_0 anaconda cudatoolkit 10.1.168 0 anaconda cudnn 7.6.0 cuda10.1_0 anaconda cupti 10.1.168 0 anaconda cx_oracle 7.0.0 py36h7b6447c_0 anaconda cycler 0.10.0 py36_0 anaconda cytoolz 0.10.0 py36h7b6447c_0 anaconda dask-core 2.5.0 py_0 anaconda dbus 1.13.6 h746ee38_0 anaconda decorator 4.4.0 py36_1 anaconda django 2.2.1 py36_0 anaconda django-cors-headers 3.0.2 py_0 conda-forge eventlet 0.23.0 py36_1000 conda-forge expat 2.2.6 he6710b0_0 anaconda falcon 2.0.0 py36h516909a_1 conda-forge ffmpeg 4.0 hcdf2ecd_0 anaconda filelock 3.0.12 py_0 anaconda fontconfig 2.13.0 h9420a91_0 anaconda freeglut 3.0.0 hf484d3e_5 anaconda freetype 2.9.1 h8a8886c_1 anaconda future 0.17.1 py36_0 anaconda gast 0.3.2 py_0 anaconda gevent 1.2.2 py36h2fe25dc_0 anaconda gevent-websocket 0.10.1 py36_1 anaconda glib 2.56.2 hd408876_0 anaconda glob2 0.7 py_0 anaconda google-pasta 0.1.7 py_0 anaconda graphite2 1.3.13 h23475e2_0 anaconda greenlet 0.4.15 py36h7b6447c_0 anaconda grpcio 1.16.1 py36hf8bcb03_1 anaconda gst-plugins-base 1.14.0 hbbd80ab_1 anaconda gstreamer 1.14.0 hb453b48_1 anaconda h5py 2.8.0 py36h989c5e5_3 anaconda harfbuzz 1.8.8 hffaf4a1_0 anaconda hdf5 1.10.2 hba1933b_1 anaconda icu 58.2 h211956c_0 anaconda idna 2.8 py36_0 anaconda imageio 2.5.0 py36_0 anaconda intel-openmp 2019.5 281 anaconda jasper 2.0.14 h07fcdf6_1 anaconda jinja2 2.10.1 py36_0 anaconda joblib 0.13.2 py36_0 anaconda jpeg 9b habf39ab_1 anaconda keras-applications 1.0.8 py_0 anaconda keras-preprocessing 1.1.0 py_1 anaconda kiwisolver 1.1.0 py36he6710b0_0 anaconda libedit 3.1.20181209 hc058e9b_0 anaconda libffi 3.2.1 h4deb6c0_3 anaconda libgcc-ng 9.1.0 hdf63c60_0 anaconda libgfortran-ng 7.3.0 hdf63c60_0 anaconda libglu 9.0.0 hf484d3e_1 anaconda libopencv 3.4.2 hb342d67_1 anaconda libopus 1.3 h7b6447c_0 anaconda libpng 1.6.37 hbc83047_0 anaconda libprotobuf 3.9.2 hd408876_0 anaconda libsodium 1.0.16 h1bed415_0 anaconda libstdcxx-ng 9.1.0 hdf63c60_0 anaconda libtiff 4.0.10 h2733197_2 anaconda libuuid 1.0.3 h1bed415_2 anaconda libvpx 1.7.0 h439df22_0 anaconda libxcb 1.13 h1bed415_1 anaconda libxml2 2.9.9 hea5a465_1 anaconda llvmlite 0.29.0 py36hf484d3e_0 numba markdown 3.1.1 py36_0 anaconda markupsafe 1.1.1 py36h7b6447c_0 anaconda matplotlib 3.1.1 py36h5429711_0 anaconda mkl 2019.5 281 anaconda mkl-service 2.3.0 py36he904b0f_0 anaconda mkl_fft 1.0.14 py36ha843d7b_0 anaconda mkl_random 1.1.0 py36hd6b4f25_0 anaconda ncurses 6.1 he6710b0_1 anaconda networkx 2.3 py_0 anaconda numba 0.45.1 py36h962f231_0 defaults numpy 1.17.2 py36haad9e8e_0 anaconda numpy-base 1.17.2 py36hde5b4d6_0 anaconda olefile 0.46 py36_0 anaconda opencv 3.4.2 py36h6fd60c2_1 defaults openssl 1.1.1 h7b6447c_0 anaconda pandas 0.25.2 py36he6710b0_0 anaconda patchelf 0.9 he6710b0_3 anaconda pcre 8.43 he6710b0_0 anaconda pillow 6.1.0 py36h34e0f95_0 anaconda pip 19.2.3 pypi_0 pypi pixman 0.38.0 h7b6447c_0 anaconda pkginfo 1.5.0.1 py36_0 anaconda protobuf 3.9.2 py36he6710b0_0 anaconda psutil 5.6.3 py36h7b6447c_0 anaconda py-opencv 3.4.2 py36hb342d67_1 anaconda pycosat 0.6.3 py36h14c3975_0 anaconda pycparser 2.19 py36_0 anaconda pyopenssl 19.0.0 py36_0 anaconda pyparsing 2.4.2 py_0 anaconda pyqt 5.9.2 py36h22d08a2_1 anaconda pysocks 1.7.1 py36_0 anaconda python 3.6.9 h265db76_0 anaconda python-dateutil 2.8.0 py36_0 anaconda pytz 2019.2 py_0 anaconda pywavelets 1.0.3 py36hdd07704_1 anaconda pyyaml 5.1.2 py36h7b6447c_0 anaconda pyzmq 18.1.0 py36he6710b0_0 anaconda qt 5.9.7 h5867ecd_1 anaconda readline 7.0 h7b6447c_5 anaconda requests 2.22.0 py36_0 anaconda ruamel_yaml 0.15.46 py36h14c3975_0 anaconda scikit-image 0.15.0 py36he6710b0_0 anaconda scikit-learn 0.21.3 py36hd81dba3_0 anaconda scipy 1.3.1 py36h7c811a0_0 anaconda setuptools 41.2.0 py36_0 anaconda sip 4.19.13 py36he6710b0_0 anaconda six 1.12.0 py36_0 anaconda soupsieve 1.9.3 py36_0 anaconda sqlalchemy 1.2.14 py36h7b6447c_0 anaconda sqlite 3.29.0 h7b6447c_0 anaconda tbb 2019.4 hfd86e86_0 defaults tensorboard 1.14.0 py36hf484d3e_0 anaconda tensorflow 1.14.0 gpu_py36h3fb9ad6_0 anaconda tensorflow-base 1.14.0 gpu_py36he45bfe2_0 anaconda tensorflow-estimator 1.14.0 py_0 anaconda tensorflow-gpu 1.14.0 h0d30ee6_0 anaconda tensorflow-serving-api 1.14.0 pypi_0 pypi termcolor 1.1.0 py36_1 anaconda tk 8.6.8 hbc83047_0 anaconda toolz 0.10.0 py_0 anaconda tornado 6.0.3 py36h7b6447c_0 anaconda tqdm 4.36.1 py_0 anaconda urllib3 1.24.2 py36_0 anaconda waitress 1.3.1 pypi_0 pypi werkzeug 0.16.0 py_0 anaconda wheel 0.33.6 py36_0 anaconda wrapt 1.11.2 py36h7b6447c_0 anaconda xz 5.2.4 h14c3975_4 anaconda yaml 0.1.7 h96e3832_1 anaconda zeromq 4.3.1 he6710b0_3 anaconda zlib 1.2.11 h7b6447c_3 anaconda zstd 1.3.7 h0b5b093_0 anaconda ```
sklam commented 4 years ago

This looks like an issue due to numba or features related to numba. The traceback seems to be indicating that memory allocation failed. I'd suggest checking the GPU-memory utilization. Also, try to debug it with environment variable NUMBA_CUDA_MAX_PENDING_DEALLOCS_COUNT=0 to disable the deferring of deallocation (see http://numba.pydata.org/numba-doc/latest/cuda/memory.html?highlight=numba_cuda_max_pending_deallocs_count#deallocation-behavior for details).

If you have further problem with numba, I'd suggest opening a ticket at https://github.com/numba/numba/issues.

sklam commented 4 years ago

Also, it would be useful to know how the threads using the GPUs. Are all threads using the same GPU? Or, is each thread assigned to a GPU?

rohitlal125555 commented 4 years ago

@sklam I'd suggest checking the GPU-memory utilization. Also, try to debug it with environment variable NUMBA_CUDA_MAX_PENDING_DEALLOCS_COUNT=0 to disable the deferring of deallocation

I've monitored the GPU memory utilization but it is not much. I'm having 4 GPUs of 32GB each on my machine and there is hardly 4 to 5 GBs of memory used & at max 15-20% use in terms of Volatile GPU Utilisation

Output of nvidia-smi below: image

Also, it would be useful to know how the threads using the GPUs. Are all threads using the same GPU? Or, is each thread assigned to a GPU?

I've different threads/processes running on all 4 different GPUs. Although the load is not ideally balanced among the 4 GPUs but it doesn't differ so much between 4 devices.

I'll try and test with equal load distribution among 4 GPUs as well & see if the problem gets resolved.