Jittor / jittor

Jittor is a high-performance deep learning framework based on JIT compiling and meta-operators.
https://cg.cs.tsinghua.edu.cn/jittor/
Apache License 2.0
3.08k stars 311 forks source link

python -m jittor.test.test_cudnn_op failed #277

Open LiUzHiAn opened 2 years ago

LiUzHiAn commented 2 years ago

Hi, thanks for your great work!

When I try to install Jittor with CUDA enabled, the following problem was raised. I'd appreciate it if you can give me some ideas to fix that.

Terminal Log

[i 1103 17:01:10.549464 16 compiler.py:941] Jittor(1.3.1.16) src: /home/liuzhian/anaconda3/envs/jittor/lib/python3.7/site-packages/jittor
[i 1103 17:01:10.551980 16 compiler.py:942] g++ at /usr/bin/g++(5.4.0)
[i 1103 17:01:10.552081 16 compiler.py:943] cache_path: /home/liuzhian/.cache/jittor/jt1.3.1/g++5.4.0/py3.7.11/Linux-4.15.0-1xca/AMDRyzen536006x1a/default
[i 1103 17:01:10.554688 16 __init__.py:372] Found nvcc(10.0.130) at /usr/local/cuda/bin/nvcc.
[i 1103 17:01:10.592039 16 __init__.py:372] Found gdb(7.11.1) at /usr/bin/gdb.
[i 1103 17:01:10.594744 16 __init__.py:372] Found addr2line(2.26.1) at /usr/bin/addr2line.
[i 1103 17:01:10.681617 16 compiler.py:993] cuda key:cu10.0.130_sm_75
[i 1103 17:01:10.836206 16 __init__.py:187] Total mem: 47.17GB, using 15 procs for compiling.
[i 1103 17:01:10.892981 16 jit_compiler.cc:27] Load cc_path: /usr/bin/g++
[i 1103 17:01:10.990998 16 init.cc:61] Found cuda archs: [75,]
[i 1103 17:01:11.056757 16 __init__.py:372] Found mpicc(1.10.2) at /usr/bin/mpicc.
[i 1103 17:01:11.134825 16 compile_extern.py:29] found /usr/local/cuda/include/cublas.h
[i 1103 17:01:11.138702 16 compile_extern.py:29] found /usr/local/cuda/lib64/libcublas.so
[i 1103 17:01:11.554852 16 compile_extern.py:29] found /usr/local/cuda/include/cudnn.h
[i 1103 17:01:11.563999 16 compile_extern.py:29] found /usr/local/cuda/lib64/libcudnn.so
[i 1103 17:01:12.492202 16 compile_extern.py:29] found /usr/local/cuda/include/curand.h
[i 1103 17:01:12.508559 16 compile_extern.py:29] found /usr/local/cuda/lib64/libcurand.so
[i 1103 17:01:12.566813 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:12.586208 16 v10 op.cc:258] Jit op key not found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32]
[i 1103 17:01:12.587537 16 v10 op.cc:265] Get jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:12.591935 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:12.606935 16 v10 op.cc:258] Jit op key not found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:acdb][WFORMAT:oihw][YFORMAT:acdb][JIT:1][JIT_cuda:1][index_t:int32]
[i 1103 17:01:12.609154 16 v10 op.cc:265] Get jit op entry: 0x7f4a07beef10
[i 1103 17:01:12.705230 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:12.705329 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:12.705372 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:12.719387 16 v100 op.cc:254] Jit op key found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:acdb][WFORMAT:oihw][YFORMAT:acdb][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a07beef10
[i 1103 17:01:12.791396 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:12.791492 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:12.791535 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:12.809329 16 v100 op.cc:254] Jit op key found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:acdb][WFORMAT:oihw][YFORMAT:acdb][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a07beef10
.[i 1103 17:01:12.880788 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:12.880968 16 v100 executor.cc:250] id: 0  type: reduce  addr: Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a701115d80)
[i 1103 17:01:12.880988 16 v100 executor.cc:259] input: 4  addr: Op(0x55a7370fa1d0:2:1:2:i2:o1:s0,binary.multiply->0x55a7365c9ff0)
[i 1103 17:01:12.880999 16 v100 executor.cc:261] 
[i 1103 17:01:12.881006 16 v100 executor.cc:250] id: 1  type: element  addr: Op(0x55a7370ead80:2:1:2:i2:o1:s0,binary.multiply->0x55a7365ceb70)
[i 1103 17:01:12.881016 16 v100 executor.cc:259] input: 5  addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73710e460)
[i 1103 17:01:12.881025 16 v100 executor.cc:259] input: 0  addr: Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a701115d80)
[i 1103 17:01:12.881033 16 v100 executor.cc:261] 
[i 1103 17:01:12.881040 16 v100 executor.cc:250] id: 2  type: reduce  addr: Op(0x55a7370f8450:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a7370f8530)
[i 1103 17:01:12.881048 16 v100 executor.cc:259] input: 6  addr: Op(0x55a7370f0950:2:1:2:i2:o1:s0,binary.multiply->0x55a7370f09e0)
[i 1103 17:01:12.881056 16 v100 executor.cc:261] 
[i 1103 17:01:12.881062 16 v100 executor.cc:250] id: 3  type: reduce  addr: Op(0x55a7370f86a0:1:1:2:i1:o1:s0,reduce.add->0x55a73734ef40)
[i 1103 17:01:12.881069 16 v100 executor.cc:259] input: 7  addr: Op(0x55a737114320:2:1:2:i2:o1:s0,binary.multiply->0x55a7371143b0)
[i 1103 17:01:12.881077 16 v100 executor.cc:261] 
[i 1103 17:01:12.881083 16 v100 executor.cc:250] id: 4  type: element  addr: Op(0x55a7370fa1d0:2:1:2:i2:o1:s0,binary.multiply->0x55a7365c9ff0)
[i 1103 17:01:12.881092 16 v100 executor.cc:259] input: 8  addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a73734fa30)
[i 1103 17:01:12.881100 16 v100 executor.cc:259] input: 9  addr: Op(0x55a7365d80d0:1:1:2:i1:o1:s0,broadcast_to->0x55a7365a2900)
[i 1103 17:01:12.881110 16 v100 executor.cc:261] 
[i 1103 17:01:12.881116 16 v100 executor.cc:250] id: 5  type: others  addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73710e460)
[i 1103 17:01:12.881125 16 v100 executor.cc:261] 
[i 1103 17:01:12.881131 16 v100 executor.cc:250] id: 6  type: element  addr: Op(0x55a7370f0950:2:1:2:i2:o1:s0,binary.multiply->0x55a7370f09e0)
[i 1103 17:01:12.881139 16 v100 executor.cc:259] input: 10  addr: Op(0x55a7370f0730:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f07e0)
[i 1103 17:01:12.881147 16 v100 executor.cc:259] input: 11  addr: Op(0x55a7370eaa40:1:1:2:i1:o1:s0,broadcast_to->0x55a73710f2f0)
[i 1103 17:01:12.881154 16 v100 executor.cc:261] 
[i 1103 17:01:12.881161 16 v100 executor.cc:250] id: 7  type: element  addr: Op(0x55a737114320:2:1:2:i2:o1:s0,binary.multiply->0x55a7371143b0)
[i 1103 17:01:12.881170 16 v100 executor.cc:259] input: 12  addr: Op(0x55a7370f8280:1:1:2:i1:o1:s0,reindex->0x55a7371141e0)
[i 1103 17:01:12.881178 16 v100 executor.cc:259] input: 13  addr: Op(0x55a7370f7fd0:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f8080)
[i 1103 17:01:12.881185 16 v100 executor.cc:261] 
[i 1103 17:01:12.881191 16 v100 executor.cc:250] id: 8  type: broadcast  addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a73734fa30)
[i 1103 17:01:12.881199 16 v100 executor.cc:259] input: 14  addr: Op(0x55a73734f880:0:1:1:i0:o1:s0,curand_random->0x55a7370f2010)
[i 1103 17:01:12.881207 16 v100 executor.cc:261] 
[i 1103 17:01:12.881214 16 v100 executor.cc:250] id: 9  type: broadcast  addr: Op(0x55a7365d80d0:1:1:2:i1:o1:s0,broadcast_to->0x55a7365a2900)
[i 1103 17:01:12.881221 16 v100 executor.cc:259] input: 15  addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7370ea850)
[i 1103 17:01:12.881230 16 v100 executor.cc:261] 
[i 1103 17:01:12.881237 16 v100 executor.cc:250] id: 10  type: broadcast  addr: Op(0x55a7370f0730:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f07e0)
[i 1103 17:01:12.881245 16 v100 executor.cc:259] input: 15  addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7370ea850)
[i 1103 17:01:12.881252 16 v100 executor.cc:261] 
[i 1103 17:01:12.881259 16 v100 executor.cc:250] id: 11  type: broadcast  addr: Op(0x55a7370eaa40:1:1:2:i1:o1:s0,broadcast_to->0x55a73710f2f0)
[i 1103 17:01:12.881267 16 v100 executor.cc:259] input: 16  addr: Op(0x55a73710e200:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430)
[i 1103 17:01:12.881275 16 v100 executor.cc:261] 
[i 1103 17:01:12.881281 16 v100 executor.cc:250] id: 12  type: broadcast  addr: Op(0x55a7370f8280:1:1:2:i1:o1:s0,reindex->0x55a7371141e0)
[i 1103 17:01:12.881289 16 v100 executor.cc:259] input: 14  addr: Op(0x55a73734f880:0:1:1:i0:o1:s0,curand_random->0x55a7370f2010)
[i 1103 17:01:12.881296 16 v100 executor.cc:261] 
[i 1103 17:01:12.881303 16 v100 executor.cc:250] id: 13  type: broadcast  addr: Op(0x55a7370f7fd0:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f8080)
[i 1103 17:01:12.881311 16 v100 executor.cc:259] input: 16  addr: Op(0x55a73710e200:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430)
[i 1103 17:01:12.881319 16 v100 executor.cc:261] 
[i 1103 17:01:12.881325 16 v100 executor.cc:250] id: 14  type: others  addr: Op(0x55a73734f880:0:1:1:i0:o1:s0,curand_random->0x55a7370f2010)
[i 1103 17:01:12.881333 16 v100 executor.cc:261] 
[i 1103 17:01:12.881339 16 v100 executor.cc:250] id: 15  type: others  addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7370ea850)
[i 1103 17:01:12.881346 16 v100 executor.cc:261] 
[i 1103 17:01:12.881353 16 v100 executor.cc:250] id: 16  type: element  addr: Op(0x55a73710e200:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430)
[i 1103 17:01:12.881361 16 v100 executor.cc:259] input: 5  addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73710e460)
[i 1103 17:01:12.881370 16 v100 executor.cc:259] input: 17  addr: Op(0x55a73710e780:0:1:1:i1:o1:s0,broadcast_to->0x55a73723af20)
[i 1103 17:01:12.881378 16 v100 executor.cc:261] 
[i 1103 17:01:12.881384 16 v100 executor.cc:250] id: 17  type: broadcast  addr: Op(0x55a73710e780:0:1:1:i1:o1:s0,broadcast_to->0x55a73723af20)
[i 1103 17:01:12.881392 16 v100 executor.cc:259] input: 18  addr: Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300)
[i 1103 17:01:12.881399 16 v100 executor.cc:261] 
[i 1103 17:01:12.881406 16 v100 executor.cc:250] id: 18  type: element  addr: Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300)
[i 1103 17:01:12.881414 16 v100 executor.cc:261] 
[i 1103 17:01:12.881427 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:12.881434 16 v1000 executor.cc:446] topsort internal [1,]
[i 1103 17:01:12.881441 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:12.881448 16 v1000 executor.cc:446] topsort internal [13,12,7,3,]
[i 1103 17:01:12.881455 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:12.881462 16 v1000 executor.cc:446] topsort internal [11,10,6,2,]
[i 1103 17:01:12.881470 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:12.881477 16 v1000 executor.cc:446] topsort internal [9,8,4,0,]
[i 1103 17:01:12.881484 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:12.881490 16 v1000 executor.cc:446] topsort internal [18,17,16,]
[i 1103 17:01:12.881497 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:12.881503 16 v1000 executor.cc:446] topsort internal [15,]
[i 1103 17:01:12.881510 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:12.881516 16 v1000 executor.cc:446] topsort internal [14,]
[i 1103 17:01:12.881523 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:12.881529 16 v1000 executor.cc:446] topsort internal [5,]
[i 1103 17:01:12.881537 16 v100 executor.cc:481] Run Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73710e460)
[i 1103 17:01:12.881547 16 v100 executor.cc:490] Run Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73710e460) inputs: [] outputs: [Var(0x55a73710e460:1:3:3:i1:o2:s0,float32,,0x7f49b9400000)[10,5,49,49,],]
[i 1103 17:01:12.881560 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0
10,5,49,49,],]
[i 1103 17:01:12.881615 16 v100 executor.cc:481] Run Op(0x55a73734f880:0:1:1:i0:o1:s0,curand_random->0x55a7370f2010)
[i 1103 17:01:12.881624 16 v100 executor.cc:490] Run Op(0x55a73734f880:0:1:1:i0:o1:s0,curand_random->0x55a7370f2010) inputs: [] outputs: [Var(0x55a7370f2010:1:3:3:i1:o2:s0,float32,,0x7f49b8000000)[10,3,100,100,],]
[i 1103 17:01:12.881635 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:12.881650 16 v100 executor.cc:551] Finished Op(curand_random 1/8) output: [Var(0x55a7370f2010:1:3:3:i1:o2:s0,float32,,0x7f49b8000000)[10,3,100,100,],]
[i 1103 17:01:12.881660 16 v100 executor.cc:481] Run Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7370ea850)
[i 1103 17:01:12.881669 16 v100 executor.cc:490] Run Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7370ea850) inputs: [] outputs: [Var(0x55a7370ea850:1:3:3:i1:o2:s0,float32,,0x7f49b9475400)[5,3,3,3,],]
[i 1103 17:01:12.881680 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:12.881695 16 v100 executor.cc:551] Finished Op(curand_random 2/8) output: [Var(0x55a7370ea850:1:3:3:i1:o2:s0,float32,,0x7f49b9475400)[5,3,3,3,],]
[i 1103 17:01:12.881706 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300),Op(0x55a73710e780:0:1:1:i1:o1:s0,broadcast_to->0x55a73723af20),Op(0x55a73710e200:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430),]
[i 1103 17:01:12.881720 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a736577430)
[i 1103 17:01:12.881729 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a736577430) inputs: [] outputs: [Var(0x55a736577430:1:2:3:i1:o2:s0,float32,,0x7f49b9475800)[10,5,49,49,],]
[i 1103 17:01:12.890439 16 v100 executor.cc:551] Finished Op(fused 3/8) output: [Var(0x55a736577430:1:2:3:i1:o2:s0,float32,,0x7f49b9475800)[10,5,49,49,],]
[i 1103 17:01:12.890475 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7365d80d0:1:1:2:i1:o1:s0,broadcast_to->0x55a7365a2900),Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a73734fa30),Op(0x55a7370fa1d0:2:1:2:i2:o1:s0,binary.multiply->0x55a7365c9ff0),Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a701115d80),]
[i 1103 17:01:12.890492 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a701115d80)
[i 1103 17:01:12.890515 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a701115d80) inputs: [] outputs: [Var(0x55a701115d80:2:2:2:i1:o1:s0,float32,,0x7f49b9500000)[10,5,49,49,],]
[i 1103 17:01:12.905278 16 v10 op.cc:258] Jit op key not found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32]
[i 1103 17:01:12.907661 16 v10 op.cc:265] Get jit op entry: 0x7f49785eff10
[i 1103 17:01:12.907803 16 v100 executor.cc:551] Finished Op(fused 4/8) output: [Var(0x55a701115d80:2:2:2:i1:o1:s0,float32,,0x7f49b9500000)[10,5,49,49,],]
[i 1103 17:01:12.907817 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7370eaa40:1:1:2:i1:o1:s0,broadcast_to->0x55a73710f2f0),Op(0x55a7370f0730:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f07e0),Op(0x55a7370f0950:2:1:2:i2:o1:s0,binary.multiply->0x55a7370f09e0),Op(0x55a7370f8450:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a7370f8530),]
[i 1103 17:01:12.907833 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370f8530)
[i 1103 17:01:12.907843 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370f8530) inputs: [] outputs: [Var(0x55a7370f8530:2:1:1:i1:o0:s0,float32,,0x7f49b8125000)[10,3,100,100,],]
[i 1103 17:01:12.923124 16 v10 op.cc:258] Jit op key not found: cudnn_conv_backward_x[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32]
[i 1103 17:01:12.925697 16 v10 op.cc:265] Get jit op entry: 0x7f49781d7100
[i 1103 17:01:12.932590 16 v100 executor.cc:551] Finished Op(fused 5/8) output: [Var(0x55a7370f8530:2:1:1:i1:o0:s0,float32,,0x7f49b8125000)[10,3,100,100,],]
[i 1103 17:01:12.932610 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7370f7fd0:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f8080),Op(0x55a7370f8280:1:1:2:i1:o1:s0,reindex->0x55a7371141e0),Op(0x55a737114320:2:1:2:i2:o1:s0,binary.multiply->0x55a7371143b0),Op(0x55a7370f86a0:1:1:2:i1:o1:s0,reduce.add->0x55a73734ef40),]
[i 1103 17:01:12.932627 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734ef40)
[i 1103 17:01:12.932636 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734ef40) inputs: [] outputs: [Var(0x55a73734ef40:2:1:1:i1:o0:s0,float32,,0x7f49b94eac00)[5,3,3,3,],]
[i 1103 17:01:12.951765 16 v10 op.cc:258] Jit op key not found: cudnn_conv_backward_w[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32]
[i 1103 17:01:12.954082 16 v10 op.cc:265] Get jit op entry: 0x7f4945bf3438
[i 1103 17:01:12.960545 16 v100 executor.cc:551] Finished Op(fused 6/8) output: [Var(0x55a73734ef40:2:1:1:i1:o0:s0,float32,,0x7f49b94eac00)[5,3,3,3,],]
[i 1103 17:01:12.960566 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7370ead80:2:1:2:i2:o1:s0,binary.multiply->0x55a7365ceb70),]
[i 1103 17:01:12.960577 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7365ceb70)
[i 1103 17:01:12.960587 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7365ceb70) inputs: [] outputs: [Var(0x55a7365ceb70:2:1:1:i1:o0:s0,float32,,0x7f49b9575400)[10,5,49,49,],]
[i 1103 17:01:12.981995 16 v100 executor.cc:551] Finished Op(fused 7/8) output: [Var(0x55a7365ceb70:2:1:1:i1:o0:s0,float32,,0x7f49b9575400)[10,5,49,49,],]
[i 1103 17:01:12.982034 16 v10 executor.cc:592] All 19 ops finished, return vars: [Var(0x55a701115d80:2:2:1:i1:o1:s1,float32,,0x7f49b9500000)[10,5,49,49,],Var(0x55a7365ceb70:2:1:0:i1:o0:s1,float32,,0x7f49b9575400)[10,5,49,49,],Var(0x55a7370f8530:2:1:0:i1:o0:s1,float32,,0x7f49b8125000)[10,3,100,100,],Var(0x55a73734ef40:2:1:0:i1:o0:s1,float32,,0x7f49b94eac00)[5,3,3,3,],]
[i 1103 17:01:12.982046 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 8 device_sync: 0
[i 1103 17:01:12.982136 16 v1 cuda_flags.cc:34] CUDA disabled.
[i 1103 17:01:12.982157 16 v10 executor.cc:592] All 0 ops finished, return vars: [Var(0x55a73734ef40:2:1:0:i1:o0:s1,float32,,0x7f49b94eac00)[5,3,3,3,],Var(0x55a7370f8530:2:1:0:i1:o0:s1,float32,,0x7f49b8125000)[10,3,100,100,],Var(0x55a7365ceb70:2:1:0:i1:o0:s1,float32,,0x7f49b9575400)[10,5,49,49,],]
[i 1103 17:01:12.982165 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 0 device_sync: 0
[i 1103 17:01:13.032396 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.032525 16 v100 executor.cc:250] id: 0  type: reduce  addr: Op(0x55a7370ead80:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0)
[i 1103 17:01:13.032538 16 v100 executor.cc:259] input: 4  addr: Op(0x55a737117180:2:1:2:i2:o1:s0,binary.multiply->0x55a737134600)
[i 1103 17:01:13.032544 16 v100 executor.cc:261] 
[i 1103 17:01:13.032549 16 v100 executor.cc:250] id: 1  type: element  addr: Op(0x55a73734f880:2:1:2:i2:o1:s0,binary.multiply->0x55a73710e780)
[i 1103 17:01:13.032555 16 v100 executor.cc:259] input: 5  addr: Op(0x55a737329990:0:1:1:i0:o1:s0,curand_random->0x55a701115d80)
[i 1103 17:01:13.032561 16 v100 executor.cc:259] input: 0  addr: Op(0x55a7370ead80:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0)
[i 1103 17:01:13.032566 16 v100 executor.cc:261] 
[i 1103 17:01:13.032571 16 v100 executor.cc:250] id: 2  type: reduce  addr: Op(0x55a737102ad0:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a7372454e0)
[i 1103 17:01:13.032577 16 v100 executor.cc:259] input: 6  addr: Op(0x55a73710e200:2:1:2:i2:o1:s0,binary.multiply->0x55a73711ae70)
[i 1103 17:01:13.032582 16 v100 executor.cc:261] 
[i 1103 17:01:13.032587 16 v100 executor.cc:250] id: 3  type: reduce  addr: Op(0x55a7365ceb70:1:1:2:i1:o1:s0,reduce.add->0x55a73734ef40)
[i 1103 17:01:13.032592 16 v100 executor.cc:259] input: 7  addr: Op(0x55a737339680:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0)
[i 1103 17:01:13.032598 16 v100 executor.cc:261] 
[i 1103 17:01:13.032602 16 v100 executor.cc:250] id: 4  type: element  addr: Op(0x55a737117180:2:1:2:i2:o1:s0,binary.multiply->0x55a737134600)
[i 1103 17:01:13.032608 16 v100 executor.cc:259] input: 8  addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a7365a2900)
[i 1103 17:01:13.032613 16 v100 executor.cc:259] input: 9  addr: Op(0x55a7365c9ff0:1:1:2:i1:o1:s0,broadcast_to->0x55a737244740)
[i 1103 17:01:13.032621 16 v100 executor.cc:261] 
[i 1103 17:01:13.032626 16 v100 executor.cc:250] id: 5  type: others  addr: Op(0x55a737329990:0:1:1:i0:o1:s0,curand_random->0x55a701115d80)
[i 1103 17:01:13.032631 16 v100 executor.cc:261] 
[i 1103 17:01:13.032636 16 v100 executor.cc:250] id: 6  type: element  addr: Op(0x55a73710e200:2:1:2:i2:o1:s0,binary.multiply->0x55a73711ae70)
[i 1103 17:01:13.032641 16 v100 executor.cc:259] input: 10  addr: Op(0x55a7370ea850:1:1:2:i1:o1:s0,broadcast_to->0x55a737244e20)
[i 1103 17:01:13.032646 16 v100 executor.cc:259] input: 11  addr: Op(0x55a7370ed2a0:1:1:2:i1:o1:s0,broadcast_to->0x55a737339d20)
[i 1103 17:01:13.032652 16 v100 executor.cc:261] 
[i 1103 17:01:13.032656 16 v100 executor.cc:250] id: 7  type: element  addr: Op(0x55a737339680:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0)
[i 1103 17:01:13.032661 16 v100 executor.cc:259] input: 12  addr: Op(0x55a73711c220:1:1:2:i1:o1:s0,reindex->0x55a7370f2010)
[i 1103 17:01:13.032667 16 v100 executor.cc:259] input: 13  addr: Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0)
[i 1103 17:01:13.032672 16 v100 executor.cc:261] 
[i 1103 17:01:13.032676 16 v100 executor.cc:250] id: 8  type: broadcast  addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a7365a2900)
[i 1103 17:01:13.032682 16 v100 executor.cc:259] input: 14  addr: Op(0x55a73712bab0:0:1:1:i0:o1:s0,curand_random->0x55a7365d80d0)
[i 1103 17:01:13.032687 16 v100 executor.cc:261] 
[i 1103 17:01:13.032692 16 v100 executor.cc:250] id: 9  type: broadcast  addr: Op(0x55a7365c9ff0:1:1:2:i1:o1:s0,broadcast_to->0x55a737244740)
[i 1103 17:01:13.032697 16 v100 executor.cc:259] input: 15  addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73734fa30)
[i 1103 17:01:13.032703 16 v100 executor.cc:261] 
[i 1103 17:01:13.032707 16 v100 executor.cc:250] id: 10  type: broadcast  addr: Op(0x55a7370ea850:1:1:2:i1:o1:s0,broadcast_to->0x55a737244e20)
[i 1103 17:01:13.032712 16 v100 executor.cc:259] input: 15  addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73734fa30)
[i 1103 17:01:13.032718 16 v100 executor.cc:261] 
[i 1103 17:01:13.032722 16 v100 executor.cc:250] id: 11  type: broadcast  addr: Op(0x55a7370ed2a0:1:1:2:i1:o1:s0,broadcast_to->0x55a737339d20)
[i 1103 17:01:13.032727 16 v100 executor.cc:259] input: 16  addr: Op(0x55a737114690:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430)
[i 1103 17:01:13.032733 16 v100 executor.cc:261] 
[i 1103 17:01:13.032737 16 v100 executor.cc:250] id: 12  type: broadcast  addr: Op(0x55a73711c220:1:1:2:i1:o1:s0,reindex->0x55a7370f2010)
[i 1103 17:01:13.032742 16 v100 executor.cc:259] input: 14  addr: Op(0x55a73712bab0:0:1:1:i0:o1:s0,curand_random->0x55a7365d80d0)
[i 1103 17:01:13.032748 16 v100 executor.cc:261] 
[i 1103 17:01:13.032752 16 v100 executor.cc:250] id: 13  type: broadcast  addr: Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0)
[i 1103 17:01:13.032757 16 v100 executor.cc:259] input: 16  addr: Op(0x55a737114690:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430)
[i 1103 17:01:13.032762 16 v100 executor.cc:261] 
[i 1103 17:01:13.032766 16 v100 executor.cc:250] id: 14  type: others  addr: Op(0x55a73712bab0:0:1:1:i0:o1:s0,curand_random->0x55a7365d80d0)
[i 1103 17:01:13.032772 16 v100 executor.cc:261] 
[i 1103 17:01:13.032776 16 v100 executor.cc:250] id: 15  type: others  addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73734fa30)
[i 1103 17:01:13.032781 16 v100 executor.cc:261] 
[i 1103 17:01:13.032786 16 v100 executor.cc:250] id: 16  type: element  addr: Op(0x55a737114690:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430)
[i 1103 17:01:13.032791 16 v100 executor.cc:259] input: 5  addr: Op(0x55a737329990:0:1:1:i0:o1:s0,curand_random->0x55a701115d80)
[i 1103 17:01:13.032797 16 v100 executor.cc:259] input: 17  addr: Op(0x55a7370eaa40:0:1:1:i1:o1:s0,broadcast_to->0x55a73710f2f0)
[i 1103 17:01:13.032801 16 v100 executor.cc:261] 
[i 1103 17:01:13.032805 16 v100 executor.cc:250] id: 17  type: broadcast  addr: Op(0x55a7370eaa40:0:1:1:i1:o1:s0,broadcast_to->0x55a73710f2f0)
[i 1103 17:01:13.032811 16 v100 executor.cc:259] input: 18  addr: Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300)
[i 1103 17:01:13.032817 16 v100 executor.cc:261] 
[i 1103 17:01:13.032821 16 v100 executor.cc:250] id: 18  type: element  addr: Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300)
[i 1103 17:01:13.032826 16 v100 executor.cc:261] 
[i 1103 17:01:13.032835 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.032840 16 v1000 executor.cc:446] topsort internal [1,]
[i 1103 17:01:13.032845 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.032850 16 v1000 executor.cc:446] topsort internal [13,12,7,3,]
[i 1103 17:01:13.032855 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.032859 16 v1000 executor.cc:446] topsort internal [11,10,6,2,]
[i 1103 17:01:13.032865 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.032870 16 v1000 executor.cc:446] topsort internal [9,8,4,0,]
[i 1103 17:01:13.032874 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.032879 16 v1000 executor.cc:446] topsort internal [18,17,16,]
[i 1103 17:01:13.032883 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.032888 16 v1000 executor.cc:446] topsort internal [15,]
[i 1103 17:01:13.032892 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.032896 16 v1000 executor.cc:446] topsort internal [14,]
[i 1103 17:01:13.032901 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.032905 16 v1000 executor.cc:446] topsort internal [5,]
[i 1103 17:01:13.032910 16 v100 executor.cc:481] Run Op(0x55a737329990:0:1:1:i0:o1:s0,curand_random->0x55a701115d80)
[i 1103 17:01:13.032917 16 v100 executor.cc:490] Run Op(0x55a737329990:0:1:1:i0:o1:s0,curand_random->0x55a701115d80) inputs: [] outputs: [Var(0x55a701115d80:1:3:3:i1:o2:s0,float32,,0x7f49b9400000)[10,5,38,48,],]
[i 1103 17:01:13.032927 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:13.032971 16 v100 executor.cc:551] Finished Op(curand_random 0/8) output: [Var(0x55a701115d80:1:3:3:i1:o2:s0,float32,,0x7f49b9400000)[10,5,38,48,],]
[i 1103 17:01:13.032979 16 v100 executor.cc:481] Run Op(0x55a73712bab0:0:1:1:i0:o1:s0,curand_random->0x55a7365d80d0)
[i 1103 17:01:13.032984 16 v100 executor.cc:490] Run Op(0x55a73712bab0:0:1:1:i0:o1:s0,curand_random->0x55a7365d80d0) inputs: [] outputs: [Var(0x55a7365d80d0:1:3:3:i1:o2:s0,float32,,0x7f49b9459200)[10,4,40,50,],]
[i 1103 17:01:13.032991 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:13.033001 16 v100 executor.cc:551] Finished Op(curand_random 1/8) output: [Var(0x55a7365d80d0:1:3:3:i1:o2:s0,float32,,0x7f49b9459200)[10,4,40,50,],]
[i 1103 17:01:13.033007 16 v100 executor.cc:481] Run Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73734fa30)
[i 1103 17:01:13.033013 16 v100 executor.cc:490] Run Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73734fa30) inputs: [] outputs: [Var(0x55a73734fa30:1:3:3:i1:o2:s0,float32,,0x7f49b94a7400)[5,4,5,5,],]
[i 1103 17:01:13.033020 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:13.033027 16 v100 executor.cc:551] Finished Op(curand_random 2/8) output: [Var(0x55a73734fa30:1:3:3:i1:o2:s0,float32,,0x7f49b94a7400)[5,4,5,5,],]
[i 1103 17:01:13.033034 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300),Op(0x55a7370eaa40:0:1:1:i1:o1:s0,broadcast_to->0x55a73710f2f0),Op(0x55a737114690:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430),]
[i 1103 17:01:13.033044 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a736577430)
[i 1103 17:01:13.033050 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a736577430) inputs: [] outputs: [Var(0x55a736577430:1:2:3:i1:o2:s0,float32,,0x7f49b9500000)[10,5,38,48,],]
[i 1103 17:01:13.033074 16 v100 executor.cc:551] Finished Op(fused 3/8) output: [Var(0x55a736577430:1:2:3:i1:o2:s0,float32,,0x7f49b9500000)[10,5,38,48,],]
[i 1103 17:01:13.033081 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7365c9ff0:1:1:2:i1:o1:s0,broadcast_to->0x55a737244740),Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a7365a2900),Op(0x55a737117180:2:1:2:i2:o1:s0,binary.multiply->0x55a737134600),Op(0x55a7370ead80:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0),]
[i 1103 17:01:13.033092 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7373379c0)
[i 1103 17:01:13.033098 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7373379c0) inputs: [] outputs: [Var(0x55a7373379c0:2:2:2:i1:o1:s0,float32,,0x7f49b9559200)[10,5,38,48,],]
[i 1103 17:01:13.046859 16 v100 op.cc:254] Jit op key found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f49785eff10
[i 1103 17:01:13.046962 16 v100 executor.cc:551] Finished Op(fused 4/8) output: [Var(0x55a7373379c0:2:2:2:i1:o1:s0,float32,,0x7f49b9559200)[10,5,38,48,],]
[i 1103 17:01:13.046971 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7370ed2a0:1:1:2:i1:o1:s0,broadcast_to->0x55a737339d20),Op(0x55a7370ea850:1:1:2:i1:o1:s0,broadcast_to->0x55a737244e20),Op(0x55a73710e200:2:1:2:i2:o1:s0,binary.multiply->0x55a73711ae70),Op(0x55a737102ad0:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a7372454e0),]
[i 1103 17:01:13.046983 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7372454e0)
[i 1103 17:01:13.046991 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7372454e0) inputs: [] outputs: [Var(0x55a7372454e0:2:1:1:i1:o0:s0,float32,,0x7f49b94a7c00)[10,4,40,50,],]
[i 1103 17:01:13.060729 16 v100 op.cc:254] Jit op key found: cudnn_conv_backward_x[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f49781d7100
[i 1103 17:01:13.074692 16 v100 executor.cc:551] Finished Op(fused 5/8) output: [Var(0x55a7372454e0:2:1:1:i1:o0:s0,float32,,0x7f49b94a7c00)[10,4,40,50,],]
[i 1103 17:01:13.074734 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0),Op(0x55a73711c220:1:1:2:i1:o1:s0,reindex->0x55a7370f2010),Op(0x55a737339680:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0),Op(0x55a7365ceb70:1:1:2:i1:o1:s0,reduce.add->0x55a73734ef40),]
[i 1103 17:01:13.074758 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734ef40)
[i 1103 17:01:13.074771 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734ef40) inputs: [] outputs: [Var(0x55a73734ef40:2:1:1:i1:o0:s0,float32,,0x7f49b94f5e00)[5,4,5,5,],]
[i 1103 17:01:13.094181 16 v100 op.cc:254] Jit op key found: cudnn_conv_backward_w[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4945bf3438
[i 1103 17:01:13.109275 16 v100 executor.cc:551] Finished Op(fused 6/8) output: [Var(0x55a73734ef40:2:1:1:i1:o0:s0,float32,,0x7f49b94f5e00)[5,4,5,5,],]
[i 1103 17:01:13.109301 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a73734f880:2:1:2:i2:o1:s0,binary.multiply->0x55a73710e780),]
[i 1103 17:01:13.109316 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73710e780)
[i 1103 17:01:13.110649 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73710e780) inputs: [] outputs: [Var(0x55a73710e780:2:1:1:i1:o0:s0,float32,,0x7f4946800000)[10,5,38,48,],]
[i 1103 17:01:13.110696 16 v100 executor.cc:551] Finished Op(fused 7/8) output: [Var(0x55a73710e780:2:1:1:i1:o0:s0,float32,,0x7f4946800000)[10,5,38,48,],]
[i 1103 17:01:13.110711 16 v10 executor.cc:592] All 19 ops finished, return vars: [Var(0x55a7373379c0:2:2:1:i1:o1:s1,float32,,0x7f49b9559200)[10,5,38,48,],Var(0x55a73710e780:2:1:0:i1:o0:s1,float32,,0x7f4946800000)[10,5,38,48,],Var(0x55a7372454e0:2:1:0:i1:o0:s1,float32,,0x7f49b94a7c00)[10,4,40,50,],Var(0x55a73734ef40:2:1:0:i1:o0:s1,float32,,0x7f49b94f5e00)[5,4,5,5,],]
[i 1103 17:01:13.110732 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 8 device_sync: 0
[i 1103 17:01:13.110776 16 v1 cuda_flags.cc:34] CUDA disabled.
[i 1103 17:01:13.110796 16 v10 executor.cc:592] All 0 ops finished, return vars: [Var(0x55a73734ef40:2:1:0:i1:o0:s1,float32,,0x7f49b94f5e00)[5,4,5,5,],Var(0x55a7372454e0:2:1:0:i1:o0:s1,float32,,0x7f49b94a7c00)[10,4,40,50,],Var(0x55a73710e780:2:1:0:i1:o0:s1,float32,,0x7f4946800000)[10,5,38,48,],]
[i 1103 17:01:13.110808 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 0 device_sync: 0
[i 1103 17:01:13.184799 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.184934 16 v100 executor.cc:250] id: 0  type: reduce  addr: Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0)
[i 1103 17:01:13.184949 16 v100 executor.cc:259] input: 4  addr: Op(0x55a737110490:2:1:2:i2:o1:s0,binary.multiply->0x55a737253bf0)
[i 1103 17:01:13.184956 16 v100 executor.cc:261] 
[i 1103 17:01:13.184961 16 v100 executor.cc:250] id: 1  type: element  addr: Op(0x55a73710e200:2:1:2:i2:o1:s0,binary.multiply->0x55a7370fe300)
[i 1103 17:01:13.184967 16 v100 executor.cc:259] input: 5  addr: Op(0x55a737114690:0:1:1:i0:o1:s0,curand_random->0x55a7370eaa40)
[i 1103 17:01:13.184973 16 v100 executor.cc:259] input: 0  addr: Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0)
[i 1103 17:01:13.184979 16 v100 executor.cc:261] 
[i 1103 17:01:13.184983 16 v100 executor.cc:250] id: 2  type: reduce  addr: Op(0x55a737331780:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a73734fa30)
[i 1103 17:01:13.184989 16 v100 executor.cc:259] input: 6  addr: Op(0x55a7365ceb70:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0)
[i 1103 17:01:13.184995 16 v100 executor.cc:261] 
[i 1103 17:01:13.184999 16 v100 executor.cc:250] id: 3  type: reduce  addr: Op(0x55a737117180:1:1:2:i1:o1:s0,reduce.add->0x55a7370f86a0)
[i 1103 17:01:13.185005 16 v100 executor.cc:259] input: 7  addr: Op(0x55a737353850:2:1:2:i2:o1:s0,binary.multiply->0x55a737113b30)
[i 1103 17:01:13.185010 16 v100 executor.cc:261] 
[i 1103 17:01:13.185014 16 v100 executor.cc:250] id: 4  type: element  addr: Op(0x55a737110490:2:1:2:i2:o1:s0,binary.multiply->0x55a737253bf0)
[i 1103 17:01:13.185020 16 v100 executor.cc:259] input: 8  addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a737244740)
[i 1103 17:01:13.185026 16 v100 executor.cc:259] input: 9  addr: Op(0x55a737134600:1:1:2:i1:o1:s0,broadcast_to->0x55a73710e460)
[i 1103 17:01:13.185033 16 v100 executor.cc:261] 
[i 1103 17:01:13.185037 16 v100 executor.cc:250] id: 5  type: others  addr: Op(0x55a737114690:0:1:1:i0:o1:s0,curand_random->0x55a7370eaa40)
[i 1103 17:01:13.185043 16 v100 executor.cc:261] 
[i 1103 17:01:13.185047 16 v100 executor.cc:250] id: 6  type: element  addr: Op(0x55a7365ceb70:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0)
[i 1103 17:01:13.185053 16 v100 executor.cc:259] input: 10  addr: Op(0x55a7370f2010:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0)
[i 1103 17:01:13.185059 16 v100 executor.cc:259] input: 11  addr: Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a736577430)
[i 1103 17:01:13.185065 16 v100 executor.cc:261] 
[i 1103 17:01:13.185069 16 v100 executor.cc:250] id: 7  type: element  addr: Op(0x55a737353850:2:1:2:i2:o1:s0,binary.multiply->0x55a737113b30)
[i 1103 17:01:13.185074 16 v100 executor.cc:259] input: 12  addr: Op(0x55a737328490:1:1:2:i1:o1:s0,reindex->0x55a73734f0d0)
[i 1103 17:01:13.185080 16 v100 executor.cc:259] input: 13  addr: Op(0x55a737103240:1:1:2:i1:o1:s0,broadcast_to->0x55a7370ebe60)
[i 1103 17:01:13.185085 16 v100 executor.cc:261] 
[i 1103 17:01:13.185089 16 v100 executor.cc:250] id: 8  type: broadcast  addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a737244740)
[i 1103 17:01:13.185094 16 v100 executor.cc:259] input: 14  addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7365c9ff0)
[i 1103 17:01:13.185099 16 v100 executor.cc:261] 
[i 1103 17:01:13.185104 16 v100 executor.cc:250] id: 9  type: broadcast  addr: Op(0x55a737134600:1:1:2:i1:o1:s0,broadcast_to->0x55a73710e460)
[i 1103 17:01:13.185110 16 v100 executor.cc:259] input: 15  addr: Op(0x55a7370ead80:0:1:1:i0:o1:s0,curand_random->0x55a7365a2900)
[i 1103 17:01:13.185116 16 v100 executor.cc:261] 
[i 1103 17:01:13.185120 16 v100 executor.cc:250] id: 10  type: broadcast  addr: Op(0x55a7370f2010:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0)
[i 1103 17:01:13.185126 16 v100 executor.cc:259] input: 15  addr: Op(0x55a7370ead80:0:1:1:i0:o1:s0,curand_random->0x55a7365a2900)
[i 1103 17:01:13.185131 16 v100 executor.cc:261] 
[i 1103 17:01:13.185135 16 v100 executor.cc:250] id: 11  type: broadcast  addr: Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a736577430)
[i 1103 17:01:13.185140 16 v100 executor.cc:259] input: 16  addr: Op(0x55a737339680:1:1:2:i2:o1:s0,binary.multiply->0x55a737244e20)
[i 1103 17:01:13.185145 16 v100 executor.cc:261] 
[i 1103 17:01:13.185149 16 v100 executor.cc:250] id: 12  type: broadcast  addr: Op(0x55a737328490:1:1:2:i1:o1:s0,reindex->0x55a73734f0d0)
[i 1103 17:01:13.185155 16 v100 executor.cc:259] input: 14  addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7365c9ff0)
[i 1103 17:01:13.185160 16 v100 executor.cc:261] 
[i 1103 17:01:13.185165 16 v100 executor.cc:250] id: 13  type: broadcast  addr: Op(0x55a737103240:1:1:2:i1:o1:s0,broadcast_to->0x55a7370ebe60)
[i 1103 17:01:13.218807 16 v100 executor.cc:259] input: 16  addr: Op(0x55a737339680:1:1:2:i2:o1:s0,binary.multiply->0x55a737244e20)
[i 1103 17:01:13.218854 16 v100 executor.cc:261] 
[i 1103 17:01:13.218867 16 v100 executor.cc:250] id: 14  type: others  addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7365c9ff0)
[i 1103 17:01:13.218884 16 v100 executor.cc:261] 
[i 1103 17:01:13.218896 16 v100 executor.cc:250] id: 15  type: others  addr: Op(0x55a7370ead80:0:1:1:i0:o1:s0,curand_random->0x55a7365a2900)
[i 1103 17:01:13.218910 16 v100 executor.cc:261] 
[i 1103 17:01:13.219168 16 v100 executor.cc:250] id: 16  type: element  addr: Op(0x55a737339680:1:1:2:i2:o1:s0,binary.multiply->0x55a737244e20)
[i 1103 17:01:13.219189 16 v100 executor.cc:259] input: 5  addr: Op(0x55a737114690:0:1:1:i0:o1:s0,curand_random->0x55a7370eaa40)
[i 1103 17:01:13.219203 16 v100 executor.cc:259] input: 17  addr: Op(0x55a7370ea850:0:1:1:i1:o1:s0,broadcast_to->0x55a7370ed2a0)
[i 1103 17:01:13.219218 16 v100 executor.cc:261] 
[i 1103 17:01:13.219229 16 v100 executor.cc:250] id: 17  type: broadcast  addr: Op(0x55a7370ea850:0:1:1:i1:o1:s0,broadcast_to->0x55a7370ed2a0)
[i 1103 17:01:13.219246 16 v100 executor.cc:259] input: 18  addr: Op(0x55a737276ba0:0:1:1:i0:o1:s0,array->0x55a73710f2f0)
[i 1103 17:01:13.219261 16 v100 executor.cc:261] 
[i 1103 17:01:13.219273 16 v100 executor.cc:250] id: 18  type: element  addr: Op(0x55a737276ba0:0:1:1:i0:o1:s0,array->0x55a73710f2f0)
[i 1103 17:01:13.219288 16 v100 executor.cc:261] 
[i 1103 17:01:13.219323 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.219337 16 v1000 executor.cc:446] topsort internal [1,]
[i 1103 17:01:13.219349 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.219360 16 v1000 executor.cc:446] topsort internal [13,12,7,3,]
[i 1103 17:01:13.219372 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.219383 16 v1000 executor.cc:446] topsort internal [11,10,6,2,]
[i 1103 17:01:13.219401 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.219412 16 v1000 executor.cc:446] topsort internal [9,8,4,0,]
[i 1103 17:01:13.219425 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.219436 16 v1000 executor.cc:446] topsort internal [18,17,16,]
[i 1103 17:01:13.219447 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.219458 16 v1000 executor.cc:446] topsort internal [15,]
[i 1103 17:01:13.219469 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.219480 16 v1000 executor.cc:446] topsort internal [14,]
[i 1103 17:01:13.219490 16 v1000 executor.cc:426] sharegraph_q []
[i 1103 17:01:13.219501 16 v1000 executor.cc:446] topsort internal [5,]
[i 1103 17:01:13.219516 16 v100 executor.cc:481] Run Op(0x55a737114690:0:1:1:i0:o1:s0,curand_random->0x55a7370eaa40)
[i 1103 17:01:13.219537 16 v100 executor.cc:490] Run Op(0x55a737114690:0:1:1:i0:o1:s0,curand_random->0x55a7370eaa40) inputs: [] outputs: [Var(0x55a7370eaa40:1:3:3:i1:o2:s0,float32,,0x7f49b9400000)[10,5,13,17,],]
[i 1103 17:01:13.219562 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:13.219627 16 v100 executor.cc:551] Finished Op(curand_random 0/8) output: [Var(0x55a7370eaa40:1:3:3:i1:o2:s0,float32,,0x7f49b9400000)[10,5,13,17,],]
[i 1103 17:01:13.219663 16 v100 executor.cc:481] Run Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7365c9ff0)
[i 1103 17:01:13.219679 16 v100 executor.cc:490] Run Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7365c9ff0) inputs: [] outputs: [Var(0x55a7365c9ff0:1:3:3:i1:o2:s0,float32,,0x7f49b940ae00)[10,4,40,50,],]
[i 1103 17:01:13.219698 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:13.219723 16 v100 executor.cc:551] Finished Op(curand_random 1/8) output: [Var(0x55a7365c9ff0:1:3:3:i1:o2:s0,float32,,0x7f49b940ae00)[10,4,40,50,],]
[i 1103 17:01:13.219741 16 v100 executor.cc:481] Run Op(0x55a7370ead80:0:1:1:i0:o1:s0,curand_random->0x55a7365a2900)
[i 1103 17:01:13.219754 16 v100 executor.cc:490] Run Op(0x55a7370ead80:0:1:1:i0:o1:s0,curand_random->0x55a7365a2900) inputs: [] outputs: [Var(0x55a7365a2900:1:3:3:i1:o2:s0,float32,,0x7f49b9459000)[5,4,4,4,],]
[i 1103 17:01:13.219773 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be
[i 1103 17:01:13.219795 16 v100 executor.cc:551] Finished Op(curand_random 2/8) output: [Var(0x55a7365a2900:1:3:3:i1:o2:s0,float32,,0x7f49b9459000)[5,4,4,4,],]
[i 1103 17:01:13.219813 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a737276ba0:0:1:1:i0:o1:s0,array->0x55a73710f2f0),Op(0x55a7370ea850:0:1:1:i1:o1:s0,broadcast_to->0x55a7370ed2a0),Op(0x55a737339680:1:1:2:i2:o1:s0,binary.multiply->0x55a737244e20),]
[i 1103 17:01:13.219837 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a737244e20)
[i 1103 17:01:13.219852 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a737244e20) inputs: [] outputs: [Var(0x55a737244e20:1:2:3:i1:o2:s0,float32,,0x7f49b9459600)[10,5,13,17,],]
[i 1103 17:01:13.219896 16 v100 executor.cc:551] Finished Op(fused 3/8) output: [Var(0x55a737244e20:1:2:3:i1:o2:s0,float32,,0x7f49b9459600)[10,5,13,17,],]
[i 1103 17:01:13.219915 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a737134600:1:1:2:i1:o1:s0,broadcast_to->0x55a73710e460),Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a737244740),Op(0x55a737110490:2:1:2:i2:o1:s0,binary.multiply->0x55a737253bf0),Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0),]
[i 1103 17:01:13.219940 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7373379c0)
[i 1103 17:01:13.219955 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7373379c0) inputs: [] outputs: [Var(0x55a7373379c0:2:2:2:i1:o1:s0,float32,,0x7f49b9464400)[10,5,13,17,],]
[i 1103 17:01:13.243395 16 v100 op.cc:254] Jit op key found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f49785eff10
[i 1103 17:01:13.243493 16 v100 executor.cc:551] Finished Op(fused 4/8) output: [Var(0x55a7373379c0:2:2:2:i1:o1:s0,float32,,0x7f49b9464400)[10,5,13,17,],]
[i 1103 17:01:13.243503 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a736577430),Op(0x55a7370f2010:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0),Op(0x55a7365ceb70:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0),Op(0x55a737331780:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a73734fa30),]
[i 1103 17:01:13.243515 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734fa30)
[i 1103 17:01:13.243523 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734fa30) inputs: [] outputs: [Var(0x55a73734fa30:2:1:1:i1:o0:s0,float32,,0x7f49b946f200)[10,4,40,50,],]
[i 1103 17:01:13.257515 16 v100 op.cc:254] Jit op key found: cudnn_conv_backward_x[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f49781d7100
[i 1103 17:01:13.261385 16 v100 executor.cc:551] Finished Op(fused 5/8) output: [Var(0x55a73734fa30:2:1:1:i1:o0:s0,float32,,0x7f49b946f200)[10,4,40,50,],]
[i 1103 17:01:13.261412 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a737103240:1:1:2:i1:o1:s0,broadcast_to->0x55a7370ebe60),Op(0x55a737328490:1:1:2:i1:o1:s0,reindex->0x55a73734f0d0),Op(0x55a737353850:2:1:2:i2:o1:s0,binary.multiply->0x55a737113b30),Op(0x55a737117180:1:1:2:i1:o1:s0,reduce.add->0x55a7370f86a0),]
[i 1103 17:01:13.261431 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370f86a0)
[i 1103 17:01:13.261442 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370f86a0) inputs: [] outputs: [Var(0x55a7370f86a0:2:1:1:i1:o0:s0,float32,,0x7f49b94bd400)[5,4,4,4,],]
[i 1103 17:01:13.279361 16 v100 op.cc:254] Jit op key found: cudnn_conv_backward_w[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4945bf3438
[i 1103 17:01:13.287683 16 v100 executor.cc:551] Finished Op(fused 6/8) output: [Var(0x55a7370f86a0:2:1:1:i1:o0:s0,float32,,0x7f49b94bd400)[5,4,4,4,],]
[i 1103 17:01:13.287716 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a73710e200:2:1:2:i2:o1:s0,binary.multiply->0x55a7370fe300),]
[i 1103 17:01:13.287730 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370fe300)
[i 1103 17:01:13.287742 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370fe300) inputs: [] outputs: [Var(0x55a7370fe300:2:1:1:i1:o0:s0,float32,,0x7f49b94bda00)[10,5,13,17,],]
[i 1103 17:01:13.291295 16 v100 executor.cc:551] Finished Op(fused 7/8) output: [Var(0x55a7370fe300:2:1:1:i1:o0:s0,float32,,0x7f49b94bda00)[10,5,13,17,],]
[i 1103 17:01:13.291313 16 v10 executor.cc:592] All 19 ops finished, return vars: [Var(0x55a7373379c0:2:2:1:i1:o1:s1,float32,,0x7f49b9464400)[10,5,13,17,],Var(0x55a7370fe300:2:1:0:i1:o0:s1,float32,,0x7f49b94bda00)[10,5,13,17,],Var(0x55a73734fa30:2:1:0:i1:o0:s1,float32,,0x7f49b946f200)[10,4,40,50,],Var(0x55a7370f86a0:2:1:0:i1:o0:s1,float32,,0x7f49b94bd400)[5,4,4,4,],]
[i 1103 17:01:13.291327 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 8 device_sync: 0
[i 1103 17:01:13.291421 16 v1 cuda_flags.cc:34] CUDA disabled.
[i 1103 17:01:13.291444 16 v10 executor.cc:592] All 0 ops finished, return vars: [Var(0x55a7370f86a0:2:1:0:i1:o0:s1,float32,,0x7f49b94bd400)[5,4,4,4,],Var(0x55a73734fa30:2:1:0:i1:o0:s1,float32,,0x7f49b946f200)[10,4,40,50,],Var(0x55a7370fe300:2:1:0:i1:o0:s1,float32,,0x7f49b94bda00)[10,5,13,17,],]
[i 1103 17:01:13.291455 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 0 device_sync: 0
..[i 1103 17:01:13.319210 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.425845 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.493165 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.561802 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.620734 16 cuda_flags.cc:32] CUDA enabled.
F[i 1103 17:01:13.677055 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.688857 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.718698 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.723490 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.779703 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.783386 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.842737 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.845882 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.903293 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.909218 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.947171 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.953387 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:13.991226 16 cuda_flags.cc:32] CUDA enabled.
[i 1103 17:01:14.005891 16 cuda_flags.cc:32] CUDA enabled.
.
======================================================================
FAIL: test_conv3d (__main__.TestCudnnConvOp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/liuzhian/anaconda3/envs/jittor/lib/python3.7/site-packages/jittor/test/test_cudnn_op.py", line 153, in test_conv3d
    check((2,4,10,10,10), (5,4,3,4,5), (1,1,1), (1,1,1))
  File "/home/liuzhian/anaconda3/envs/jittor/lib/python3.7/site-packages/jittor/test/test_cudnn_op.py", line 145, in check
    np.testing.assert_allclose(y.data, y2.data)
  File "/home/liuzhian/anaconda3/envs/jittor/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 1531, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/home/liuzhian/anaconda3/envs/jittor/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 4623 / 7200 (64.2%)
Max absolute difference: 5.340576e-05
Max relative difference: 8.419145e-07
 x: array([[[[[26.199913, 32.35394 , 31.798569, ..., 27.34977 , 27.382778,
           20.727432],
          [34.28533 , 41.313995, 40.744705, ..., 36.90609 , 39.26516 ,...
 y: array([[[[[26.199905, 32.35394 , 31.79857 , ..., 27.349777, 27.38277 ,
           20.72743 ],
          [34.28532 , 41.314003, 40.744694, ..., 36.906097, 39.265167,...

----------------------------------------------------------------------
Ran 5 tests in 1.481s

FAILED (failures=1)

I can pass the test process via python -m jittor.test.test_example

My Envs

Jittor commented 2 years ago

This log is fine, the error can be ignore

------------------ 原始邮件 ------------------ 发件人: "Jittor/jittor" @.>; 发送时间: 2021年11月3日(星期三) 下午5:07 @.>; @.***>; 主题: [Jittor/jittor] python -m jittor.test.test_cudnn_op failed (Issue #277)

Hi, thanks for your great work!

When I try to install Jittor with CUDA enabled, the following problem was raised. I'd appreciate it if you can give me some ideas to fix that.

Terminal Log [i 1103 17:01:10.549464 16 compiler.py:941] Jittor(1.3.1.16) src: /home/liuzhian/anaconda3/envs/jittor/lib/python3.7/site-packages/jittor [i 1103 17:01:10.551980 16 compiler.py:942] g++ at /usr/bin/g++(5.4.0) [i 1103 17:01:10.552081 16 compiler.py:943] cache_path: /home/liuzhian/.cache/jittor/jt1.3.1/g++5.4.0/py3.7.11/Linux-4.15.0-1xca/AMDRyzen536006x1a/default [i 1103 17:01:10.554688 16 init.py:372] Found nvcc(10.0.130) at /usr/local/cuda/bin/nvcc. [i 1103 17:01:10.592039 16 init.py:372] Found gdb(7.11.1) at /usr/bin/gdb. [i 1103 17:01:10.594744 16 init.py:372] Found addr2line(2.26.1) at /usr/bin/addr2line. [i 1103 17:01:10.681617 16 compiler.py:993] cuda key:cu10.0.130_sm_75 [i 1103 17:01:10.836206 16 init.py:187] Total mem: 47.17GB, using 15 procs for compiling. [i 1103 17:01:10.892981 16 jit_compiler.cc:27] Load cc_path: /usr/bin/g++ [i 1103 17:01:10.990998 16 init.cc:61] Found cuda archs: [75,] [i 1103 17:01:11.056757 16 init.py:372] Found mpicc(1.10.2) at /usr/bin/mpicc. [i 1103 17:01:11.134825 16 compile_extern.py:29] found /usr/local/cuda/include/cublas.h [i 1103 17:01:11.138702 16 compile_extern.py:29] found /usr/local/cuda/lib64/libcublas.so [i 1103 17:01:11.554852 16 compile_extern.py:29] found /usr/local/cuda/include/cudnn.h [i 1103 17:01:11.563999 16 compile_extern.py:29] found /usr/local/cuda/lib64/libcudnn.so [i 1103 17:01:12.492202 16 compile_extern.py:29] found /usr/local/cuda/include/curand.h [i 1103 17:01:12.508559 16 compile_extern.py:29] found /usr/local/cuda/lib64/libcurand.so [i 1103 17:01:12.566813 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:12.586208 16 v10 op.cc:258] Jit op key not found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] [i 1103 17:01:12.587537 16 v10 op.cc:265] Get jit op entry: 0x7f4a0a1e50be [i 1103 17:01:12.591935 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:12.606935 16 v10 op.cc:258] Jit op key not found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:acdb][WFORMAT:oihw][YFORMAT:acdb][JIT:1][JIT_cuda:1][index_t:int32] [i 1103 17:01:12.609154 16 v10 op.cc:265] Get jit op entry: 0x7f4a07beef10 [i 1103 17:01:12.705230 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:12.705329 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:12.705372 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:12.719387 16 v100 op.cc:254] Jit op key found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:acdb][WFORMAT:oihw][YFORMAT:acdb][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a07beef10 [i 1103 17:01:12.791396 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:12.791492 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:12.791535 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:12.809329 16 v100 op.cc:254] Jit op key found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:acdb][WFORMAT:oihw][YFORMAT:acdb][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a07beef10 .[i 1103 17:01:12.880788 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:12.880968 16 v100 executor.cc:250] id: 0 type: reduce addr: Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a701115d80) [i 1103 17:01:12.880988 16 v100 executor.cc:259] input: 4 addr: Op(0x55a7370fa1d0:2:1:2:i2:o1:s0,binary.multiply->0x55a7365c9ff0) [i 1103 17:01:12.880999 16 v100 executor.cc:261] [i 1103 17:01:12.881006 16 v100 executor.cc:250] id: 1 type: element addr: Op(0x55a7370ead80:2:1:2:i2:o1:s0,binary.multiply->0x55a7365ceb70) [i 1103 17:01:12.881016 16 v100 executor.cc:259] input: 5 addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73710e460) [i 1103 17:01:12.881025 16 v100 executor.cc:259] input: 0 addr: Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a701115d80) [i 1103 17:01:12.881033 16 v100 executor.cc:261] [i 1103 17:01:12.881040 16 v100 executor.cc:250] id: 2 type: reduce addr: Op(0x55a7370f8450:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a7370f8530) [i 1103 17:01:12.881048 16 v100 executor.cc:259] input: 6 addr: Op(0x55a7370f0950:2:1:2:i2:o1:s0,binary.multiply->0x55a7370f09e0) [i 1103 17:01:12.881056 16 v100 executor.cc:261] [i 1103 17:01:12.881062 16 v100 executor.cc:250] id: 3 type: reduce addr: Op(0x55a7370f86a0:1:1:2:i1:o1:s0,reduce.add->0x55a73734ef40) [i 1103 17:01:12.881069 16 v100 executor.cc:259] input: 7 addr: Op(0x55a737114320:2:1:2:i2:o1:s0,binary.multiply->0x55a7371143b0) [i 1103 17:01:12.881077 16 v100 executor.cc:261] [i 1103 17:01:12.881083 16 v100 executor.cc:250] id: 4 type: element addr: Op(0x55a7370fa1d0:2:1:2:i2:o1:s0,binary.multiply->0x55a7365c9ff0) [i 1103 17:01:12.881092 16 v100 executor.cc:259] input: 8 addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a73734fa30) [i 1103 17:01:12.881100 16 v100 executor.cc:259] input: 9 addr: Op(0x55a7365d80d0:1:1:2:i1:o1:s0,broadcast_to->0x55a7365a2900) [i 1103 17:01:12.881110 16 v100 executor.cc:261] [i 1103 17:01:12.881116 16 v100 executor.cc:250] id: 5 type: others addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73710e460) [i 1103 17:01:12.881125 16 v100 executor.cc:261] [i 1103 17:01:12.881131 16 v100 executor.cc:250] id: 6 type: element addr: Op(0x55a7370f0950:2:1:2:i2:o1:s0,binary.multiply->0x55a7370f09e0) [i 1103 17:01:12.881139 16 v100 executor.cc:259] input: 10 addr: Op(0x55a7370f0730:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f07e0) [i 1103 17:01:12.881147 16 v100 executor.cc:259] input: 11 addr: Op(0x55a7370eaa40:1:1:2:i1:o1:s0,broadcast_to->0x55a73710f2f0) [i 1103 17:01:12.881154 16 v100 executor.cc:261] [i 1103 17:01:12.881161 16 v100 executor.cc:250] id: 7 type: element addr: Op(0x55a737114320:2:1:2:i2:o1:s0,binary.multiply->0x55a7371143b0) [i 1103 17:01:12.881170 16 v100 executor.cc:259] input: 12 addr: Op(0x55a7370f8280:1:1:2:i1:o1:s0,reindex->0x55a7371141e0) [i 1103 17:01:12.881178 16 v100 executor.cc:259] input: 13 addr: Op(0x55a7370f7fd0:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f8080) [i 1103 17:01:12.881185 16 v100 executor.cc:261] [i 1103 17:01:12.881191 16 v100 executor.cc:250] id: 8 type: broadcast addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a73734fa30) [i 1103 17:01:12.881199 16 v100 executor.cc:259] input: 14 addr: Op(0x55a73734f880:0:1:1:i0:o1:s0,curand_random->0x55a7370f2010) [i 1103 17:01:12.881207 16 v100 executor.cc:261] [i 1103 17:01:12.881214 16 v100 executor.cc:250] id: 9 type: broadcast addr: Op(0x55a7365d80d0:1:1:2:i1:o1:s0,broadcast_to->0x55a7365a2900) [i 1103 17:01:12.881221 16 v100 executor.cc:259] input: 15 addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7370ea850) [i 1103 17:01:12.881230 16 v100 executor.cc:261] [i 1103 17:01:12.881237 16 v100 executor.cc:250] id: 10 type: broadcast addr: Op(0x55a7370f0730:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f07e0) [i 1103 17:01:12.881245 16 v100 executor.cc:259] input: 15 addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7370ea850) [i 1103 17:01:12.881252 16 v100 executor.cc:261] [i 1103 17:01:12.881259 16 v100 executor.cc:250] id: 11 type: broadcast addr: Op(0x55a7370eaa40:1:1:2:i1:o1:s0,broadcast_to->0x55a73710f2f0) [i 1103 17:01:12.881267 16 v100 executor.cc:259] input: 16 addr: Op(0x55a73710e200:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430) [i 1103 17:01:12.881275 16 v100 executor.cc:261] [i 1103 17:01:12.881281 16 v100 executor.cc:250] id: 12 type: broadcast addr: Op(0x55a7370f8280:1:1:2:i1:o1:s0,reindex->0x55a7371141e0) [i 1103 17:01:12.881289 16 v100 executor.cc:259] input: 14 addr: Op(0x55a73734f880:0:1:1:i0:o1:s0,curand_random->0x55a7370f2010) [i 1103 17:01:12.881296 16 v100 executor.cc:261] [i 1103 17:01:12.881303 16 v100 executor.cc:250] id: 13 type: broadcast addr: Op(0x55a7370f7fd0:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f8080) [i 1103 17:01:12.881311 16 v100 executor.cc:259] input: 16 addr: Op(0x55a73710e200:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430) [i 1103 17:01:12.881319 16 v100 executor.cc:261] [i 1103 17:01:12.881325 16 v100 executor.cc:250] id: 14 type: others addr: Op(0x55a73734f880:0:1:1:i0:o1:s0,curand_random->0x55a7370f2010) [i 1103 17:01:12.881333 16 v100 executor.cc:261] [i 1103 17:01:12.881339 16 v100 executor.cc:250] id: 15 type: others addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7370ea850) [i 1103 17:01:12.881346 16 v100 executor.cc:261] [i 1103 17:01:12.881353 16 v100 executor.cc:250] id: 16 type: element addr: Op(0x55a73710e200:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430) [i 1103 17:01:12.881361 16 v100 executor.cc:259] input: 5 addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73710e460) [i 1103 17:01:12.881370 16 v100 executor.cc:259] input: 17 addr: Op(0x55a73710e780:0:1:1:i1:o1:s0,broadcast_to->0x55a73723af20) [i 1103 17:01:12.881378 16 v100 executor.cc:261] [i 1103 17:01:12.881384 16 v100 executor.cc:250] id: 17 type: broadcast addr: Op(0x55a73710e780:0:1:1:i1:o1:s0,broadcast_to->0x55a73723af20) [i 1103 17:01:12.881392 16 v100 executor.cc:259] input: 18 addr: Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300) [i 1103 17:01:12.881399 16 v100 executor.cc:261] [i 1103 17:01:12.881406 16 v100 executor.cc:250] id: 18 type: element addr: Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300) [i 1103 17:01:12.881414 16 v100 executor.cc:261] [i 1103 17:01:12.881427 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:12.881434 16 v1000 executor.cc:446] topsort internal [1,] [i 1103 17:01:12.881441 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:12.881448 16 v1000 executor.cc:446] topsort internal [13,12,7,3,] [i 1103 17:01:12.881455 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:12.881462 16 v1000 executor.cc:446] topsort internal [11,10,6,2,] [i 1103 17:01:12.881470 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:12.881477 16 v1000 executor.cc:446] topsort internal [9,8,4,0,] [i 1103 17:01:12.881484 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:12.881490 16 v1000 executor.cc:446] topsort internal [18,17,16,] [i 1103 17:01:12.881497 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:12.881503 16 v1000 executor.cc:446] topsort internal [15,] [i 1103 17:01:12.881510 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:12.881516 16 v1000 executor.cc:446] topsort internal [14,] [i 1103 17:01:12.881523 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:12.881529 16 v1000 executor.cc:446] topsort internal [5,] [i 1103 17:01:12.881537 16 v100 executor.cc:481] Run Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73710e460) [i 1103 17:01:12.881547 16 v100 executor.cc:490] Run Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73710e460) inputs: [] outputs: [Var(0x55a73710e460:1:3:3:i1:o2:s0,float32,,0x7f49b9400000)[10,5,49,49,],] [i 1103 17:01:12.881560 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0 10,5,49,49,],] [i 1103 17:01:12.881615 16 v100 executor.cc:481] Run Op(0x55a73734f880:0:1:1:i0:o1:s0,curand_random->0x55a7370f2010) [i 1103 17:01:12.881624 16 v100 executor.cc:490] Run Op(0x55a73734f880:0:1:1:i0:o1:s0,curand_random->0x55a7370f2010) inputs: [] outputs: [Var(0x55a7370f2010:1:3:3:i1:o2:s0,float32,,0x7f49b8000000)[10,3,100,100,],] [i 1103 17:01:12.881635 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:12.881650 16 v100 executor.cc:551] Finished Op(curand_random 1/8) output: [Var(0x55a7370f2010:1:3:3:i1:o2:s0,float32,,0x7f49b8000000)[10,3,100,100,],] [i 1103 17:01:12.881660 16 v100 executor.cc:481] Run Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7370ea850) [i 1103 17:01:12.881669 16 v100 executor.cc:490] Run Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7370ea850) inputs: [] outputs: [Var(0x55a7370ea850:1:3:3:i1:o2:s0,float32,,0x7f49b9475400)[5,3,3,3,],] [i 1103 17:01:12.881680 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:12.881695 16 v100 executor.cc:551] Finished Op(curand_random 2/8) output: [Var(0x55a7370ea850:1:3:3:i1:o2:s0,float32,,0x7f49b9475400)[5,3,3,3,],] [i 1103 17:01:12.881706 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300),Op(0x55a73710e780:0:1:1:i1:o1:s0,broadcast_to->0x55a73723af20),Op(0x55a73710e200:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430),] [i 1103 17:01:12.881720 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a736577430) [i 1103 17:01:12.881729 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a736577430) inputs: [] outputs: [Var(0x55a736577430:1:2:3:i1:o2:s0,float32,,0x7f49b9475800)[10,5,49,49,],] [i 1103 17:01:12.890439 16 v100 executor.cc:551] Finished Op(fused 3/8) output: [Var(0x55a736577430:1:2:3:i1:o2:s0,float32,,0x7f49b9475800)[10,5,49,49,],] [i 1103 17:01:12.890475 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7365d80d0:1:1:2:i1:o1:s0,broadcast_to->0x55a7365a2900),Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a73734fa30),Op(0x55a7370fa1d0:2:1:2:i2:o1:s0,binary.multiply->0x55a7365c9ff0),Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a701115d80),] [i 1103 17:01:12.890492 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a701115d80) [i 1103 17:01:12.890515 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a701115d80) inputs: [] outputs: [Var(0x55a701115d80:2:2:2:i1:o1:s0,float32,,0x7f49b9500000)[10,5,49,49,],] [i 1103 17:01:12.905278 16 v10 op.cc:258] Jit op key not found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] [i 1103 17:01:12.907661 16 v10 op.cc:265] Get jit op entry: 0x7f49785eff10 [i 1103 17:01:12.907803 16 v100 executor.cc:551] Finished Op(fused 4/8) output: [Var(0x55a701115d80:2:2:2:i1:o1:s0,float32,,0x7f49b9500000)[10,5,49,49,],] [i 1103 17:01:12.907817 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7370eaa40:1:1:2:i1:o1:s0,broadcast_to->0x55a73710f2f0),Op(0x55a7370f0730:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f07e0),Op(0x55a7370f0950:2:1:2:i2:o1:s0,binary.multiply->0x55a7370f09e0),Op(0x55a7370f8450:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a7370f8530),] [i 1103 17:01:12.907833 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370f8530) [i 1103 17:01:12.907843 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370f8530) inputs: [] outputs: [Var(0x55a7370f8530:2:1:1:i1:o0:s0,float32,,0x7f49b8125000)[10,3,100,100,],] [i 1103 17:01:12.923124 16 v10 op.cc:258] Jit op key not found: cudnn_conv_backward_x[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] [i 1103 17:01:12.925697 16 v10 op.cc:265] Get jit op entry: 0x7f49781d7100 [i 1103 17:01:12.932590 16 v100 executor.cc:551] Finished Op(fused 5/8) output: [Var(0x55a7370f8530:2:1:1:i1:o0:s0,float32,,0x7f49b8125000)[10,3,100,100,],] [i 1103 17:01:12.932610 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7370f7fd0:1:1:2:i1:o1:s0,broadcast_to->0x55a7370f8080),Op(0x55a7370f8280:1:1:2:i1:o1:s0,reindex->0x55a7371141e0),Op(0x55a737114320:2:1:2:i2:o1:s0,binary.multiply->0x55a7371143b0),Op(0x55a7370f86a0:1:1:2:i1:o1:s0,reduce.add->0x55a73734ef40),] [i 1103 17:01:12.932627 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734ef40) [i 1103 17:01:12.932636 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734ef40) inputs: [] outputs: [Var(0x55a73734ef40:2:1:1:i1:o0:s0,float32,,0x7f49b94eac00)[5,3,3,3,],] [i 1103 17:01:12.951765 16 v10 op.cc:258] Jit op key not found: cudnn_conv_backward_w[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] [i 1103 17:01:12.954082 16 v10 op.cc:265] Get jit op entry: 0x7f4945bf3438 [i 1103 17:01:12.960545 16 v100 executor.cc:551] Finished Op(fused 6/8) output: [Var(0x55a73734ef40:2:1:1:i1:o0:s0,float32,,0x7f49b94eac00)[5,3,3,3,],] [i 1103 17:01:12.960566 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7370ead80:2:1:2:i2:o1:s0,binary.multiply->0x55a7365ceb70),] [i 1103 17:01:12.960577 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7365ceb70) [i 1103 17:01:12.960587 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7365ceb70) inputs: [] outputs: [Var(0x55a7365ceb70:2:1:1:i1:o0:s0,float32,,0x7f49b9575400)[10,5,49,49,],] [i 1103 17:01:12.981995 16 v100 executor.cc:551] Finished Op(fused 7/8) output: [Var(0x55a7365ceb70:2:1:1:i1:o0:s0,float32,,0x7f49b9575400)[10,5,49,49,],] [i 1103 17:01:12.982034 16 v10 executor.cc:592] All 19 ops finished, return vars: [Var(0x55a701115d80:2:2:1:i1:o1:s1,float32,,0x7f49b9500000)[10,5,49,49,],Var(0x55a7365ceb70:2:1:0:i1:o0:s1,float32,,0x7f49b9575400)[10,5,49,49,],Var(0x55a7370f8530:2:1:0:i1:o0:s1,float32,,0x7f49b8125000)[10,3,100,100,],Var(0x55a73734ef40:2:1:0:i1:o0:s1,float32,,0x7f49b94eac00)[5,3,3,3,],] [i 1103 17:01:12.982046 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 8 device_sync: 0 [i 1103 17:01:12.982136 16 v1 cuda_flags.cc:34] CUDA disabled. [i 1103 17:01:12.982157 16 v10 executor.cc:592] All 0 ops finished, return vars: [Var(0x55a73734ef40:2:1:0:i1:o0:s1,float32,,0x7f49b94eac00)[5,3,3,3,],Var(0x55a7370f8530:2:1:0:i1:o0:s1,float32,,0x7f49b8125000)[10,3,100,100,],Var(0x55a7365ceb70:2:1:0:i1:o0:s1,float32,,0x7f49b9575400)[10,5,49,49,],] [i 1103 17:01:12.982165 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 0 device_sync: 0 [i 1103 17:01:13.032396 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.032525 16 v100 executor.cc:250] id: 0 type: reduce addr: Op(0x55a7370ead80:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0) [i 1103 17:01:13.032538 16 v100 executor.cc:259] input: 4 addr: Op(0x55a737117180:2:1:2:i2:o1:s0,binary.multiply->0x55a737134600) [i 1103 17:01:13.032544 16 v100 executor.cc:261] [i 1103 17:01:13.032549 16 v100 executor.cc:250] id: 1 type: element addr: Op(0x55a73734f880:2:1:2:i2:o1:s0,binary.multiply->0x55a73710e780) [i 1103 17:01:13.032555 16 v100 executor.cc:259] input: 5 addr: Op(0x55a737329990:0:1:1:i0:o1:s0,curand_random->0x55a701115d80) [i 1103 17:01:13.032561 16 v100 executor.cc:259] input: 0 addr: Op(0x55a7370ead80:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0) [i 1103 17:01:13.032566 16 v100 executor.cc:261] [i 1103 17:01:13.032571 16 v100 executor.cc:250] id: 2 type: reduce addr: Op(0x55a737102ad0:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a7372454e0) [i 1103 17:01:13.032577 16 v100 executor.cc:259] input: 6 addr: Op(0x55a73710e200:2:1:2:i2:o1:s0,binary.multiply->0x55a73711ae70) [i 1103 17:01:13.032582 16 v100 executor.cc:261] [i 1103 17:01:13.032587 16 v100 executor.cc:250] id: 3 type: reduce addr: Op(0x55a7365ceb70:1:1:2:i1:o1:s0,reduce.add->0x55a73734ef40) [i 1103 17:01:13.032592 16 v100 executor.cc:259] input: 7 addr: Op(0x55a737339680:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0) [i 1103 17:01:13.032598 16 v100 executor.cc:261] [i 1103 17:01:13.032602 16 v100 executor.cc:250] id: 4 type: element addr: Op(0x55a737117180:2:1:2:i2:o1:s0,binary.multiply->0x55a737134600) [i 1103 17:01:13.032608 16 v100 executor.cc:259] input: 8 addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a7365a2900) [i 1103 17:01:13.032613 16 v100 executor.cc:259] input: 9 addr: Op(0x55a7365c9ff0:1:1:2:i1:o1:s0,broadcast_to->0x55a737244740) [i 1103 17:01:13.032621 16 v100 executor.cc:261] [i 1103 17:01:13.032626 16 v100 executor.cc:250] id: 5 type: others addr: Op(0x55a737329990:0:1:1:i0:o1:s0,curand_random->0x55a701115d80) [i 1103 17:01:13.032631 16 v100 executor.cc:261] [i 1103 17:01:13.032636 16 v100 executor.cc:250] id: 6 type: element addr: Op(0x55a73710e200:2:1:2:i2:o1:s0,binary.multiply->0x55a73711ae70) [i 1103 17:01:13.032641 16 v100 executor.cc:259] input: 10 addr: Op(0x55a7370ea850:1:1:2:i1:o1:s0,broadcast_to->0x55a737244e20) [i 1103 17:01:13.032646 16 v100 executor.cc:259] input: 11 addr: Op(0x55a7370ed2a0:1:1:2:i1:o1:s0,broadcast_to->0x55a737339d20) [i 1103 17:01:13.032652 16 v100 executor.cc:261] [i 1103 17:01:13.032656 16 v100 executor.cc:250] id: 7 type: element addr: Op(0x55a737339680:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0) [i 1103 17:01:13.032661 16 v100 executor.cc:259] input: 12 addr: Op(0x55a73711c220:1:1:2:i1:o1:s0,reindex->0x55a7370f2010) [i 1103 17:01:13.032667 16 v100 executor.cc:259] input: 13 addr: Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0) [i 1103 17:01:13.032672 16 v100 executor.cc:261] [i 1103 17:01:13.032676 16 v100 executor.cc:250] id: 8 type: broadcast addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a7365a2900) [i 1103 17:01:13.032682 16 v100 executor.cc:259] input: 14 addr: Op(0x55a73712bab0:0:1:1:i0:o1:s0,curand_random->0x55a7365d80d0) [i 1103 17:01:13.032687 16 v100 executor.cc:261] [i 1103 17:01:13.032692 16 v100 executor.cc:250] id: 9 type: broadcast addr: Op(0x55a7365c9ff0:1:1:2:i1:o1:s0,broadcast_to->0x55a737244740) [i 1103 17:01:13.032697 16 v100 executor.cc:259] input: 15 addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73734fa30) [i 1103 17:01:13.032703 16 v100 executor.cc:261] [i 1103 17:01:13.032707 16 v100 executor.cc:250] id: 10 type: broadcast addr: Op(0x55a7370ea850:1:1:2:i1:o1:s0,broadcast_to->0x55a737244e20) [i 1103 17:01:13.032712 16 v100 executor.cc:259] input: 15 addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73734fa30) [i 1103 17:01:13.032718 16 v100 executor.cc:261] [i 1103 17:01:13.032722 16 v100 executor.cc:250] id: 11 type: broadcast addr: Op(0x55a7370ed2a0:1:1:2:i1:o1:s0,broadcast_to->0x55a737339d20) [i 1103 17:01:13.032727 16 v100 executor.cc:259] input: 16 addr: Op(0x55a737114690:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430) [i 1103 17:01:13.032733 16 v100 executor.cc:261] [i 1103 17:01:13.032737 16 v100 executor.cc:250] id: 12 type: broadcast addr: Op(0x55a73711c220:1:1:2:i1:o1:s0,reindex->0x55a7370f2010) [i 1103 17:01:13.032742 16 v100 executor.cc:259] input: 14 addr: Op(0x55a73712bab0:0:1:1:i0:o1:s0,curand_random->0x55a7365d80d0) [i 1103 17:01:13.032748 16 v100 executor.cc:261] [i 1103 17:01:13.032752 16 v100 executor.cc:250] id: 13 type: broadcast addr: Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0) [i 1103 17:01:13.032757 16 v100 executor.cc:259] input: 16 addr: Op(0x55a737114690:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430) [i 1103 17:01:13.032762 16 v100 executor.cc:261] [i 1103 17:01:13.032766 16 v100 executor.cc:250] id: 14 type: others addr: Op(0x55a73712bab0:0:1:1:i0:o1:s0,curand_random->0x55a7365d80d0) [i 1103 17:01:13.032772 16 v100 executor.cc:261] [i 1103 17:01:13.032776 16 v100 executor.cc:250] id: 15 type: others addr: Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73734fa30) [i 1103 17:01:13.032781 16 v100 executor.cc:261] [i 1103 17:01:13.032786 16 v100 executor.cc:250] id: 16 type: element addr: Op(0x55a737114690:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430) [i 1103 17:01:13.032791 16 v100 executor.cc:259] input: 5 addr: Op(0x55a737329990:0:1:1:i0:o1:s0,curand_random->0x55a701115d80) [i 1103 17:01:13.032797 16 v100 executor.cc:259] input: 17 addr: Op(0x55a7370eaa40:0:1:1:i1:o1:s0,broadcast_to->0x55a73710f2f0) [i 1103 17:01:13.032801 16 v100 executor.cc:261] [i 1103 17:01:13.032805 16 v100 executor.cc:250] id: 17 type: broadcast addr: Op(0x55a7370eaa40:0:1:1:i1:o1:s0,broadcast_to->0x55a73710f2f0) [i 1103 17:01:13.032811 16 v100 executor.cc:259] input: 18 addr: Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300) [i 1103 17:01:13.032817 16 v100 executor.cc:261] [i 1103 17:01:13.032821 16 v100 executor.cc:250] id: 18 type: element addr: Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300) [i 1103 17:01:13.032826 16 v100 executor.cc:261] [i 1103 17:01:13.032835 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.032840 16 v1000 executor.cc:446] topsort internal [1,] [i 1103 17:01:13.032845 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.032850 16 v1000 executor.cc:446] topsort internal [13,12,7,3,] [i 1103 17:01:13.032855 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.032859 16 v1000 executor.cc:446] topsort internal [11,10,6,2,] [i 1103 17:01:13.032865 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.032870 16 v1000 executor.cc:446] topsort internal [9,8,4,0,] [i 1103 17:01:13.032874 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.032879 16 v1000 executor.cc:446] topsort internal [18,17,16,] [i 1103 17:01:13.032883 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.032888 16 v1000 executor.cc:446] topsort internal [15,] [i 1103 17:01:13.032892 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.032896 16 v1000 executor.cc:446] topsort internal [14,] [i 1103 17:01:13.032901 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.032905 16 v1000 executor.cc:446] topsort internal [5,] [i 1103 17:01:13.032910 16 v100 executor.cc:481] Run Op(0x55a737329990:0:1:1:i0:o1:s0,curand_random->0x55a701115d80) [i 1103 17:01:13.032917 16 v100 executor.cc:490] Run Op(0x55a737329990:0:1:1:i0:o1:s0,curand_random->0x55a701115d80) inputs: [] outputs: [Var(0x55a701115d80:1:3:3:i1:o2:s0,float32,,0x7f49b9400000)[10,5,38,48,],] [i 1103 17:01:13.032927 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:13.032971 16 v100 executor.cc:551] Finished Op(curand_random 0/8) output: [Var(0x55a701115d80:1:3:3:i1:o2:s0,float32,,0x7f49b9400000)[10,5,38,48,],] [i 1103 17:01:13.032979 16 v100 executor.cc:481] Run Op(0x55a73712bab0:0:1:1:i0:o1:s0,curand_random->0x55a7365d80d0) [i 1103 17:01:13.032984 16 v100 executor.cc:490] Run Op(0x55a73712bab0:0:1:1:i0:o1:s0,curand_random->0x55a7365d80d0) inputs: [] outputs: [Var(0x55a7365d80d0:1:3:3:i1:o2:s0,float32,,0x7f49b9459200)[10,4,40,50,],] [i 1103 17:01:13.032991 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:13.033001 16 v100 executor.cc:551] Finished Op(curand_random 1/8) output: [Var(0x55a7365d80d0:1:3:3:i1:o2:s0,float32,,0x7f49b9459200)[10,4,40,50,],] [i 1103 17:01:13.033007 16 v100 executor.cc:481] Run Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73734fa30) [i 1103 17:01:13.033013 16 v100 executor.cc:490] Run Op(0x55a737110490:0:1:1:i0:o1:s0,curand_random->0x55a73734fa30) inputs: [] outputs: [Var(0x55a73734fa30:1:3:3:i1:o2:s0,float32,,0x7f49b94a7400)[5,4,5,5,],] [i 1103 17:01:13.033020 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:13.033027 16 v100 executor.cc:551] Finished Op(curand_random 2/8) output: [Var(0x55a73734fa30:1:3:3:i1:o2:s0,float32,,0x7f49b94a7400)[5,4,5,5,],] [i 1103 17:01:13.033034 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a73658eb70:0:1:1:i0:o1:s0,array->0x55a7370fe300),Op(0x55a7370eaa40:0:1:1:i1:o1:s0,broadcast_to->0x55a73710f2f0),Op(0x55a737114690:1:1:2:i2:o1:s0,binary.multiply->0x55a736577430),] [i 1103 17:01:13.033044 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a736577430) [i 1103 17:01:13.033050 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a736577430) inputs: [] outputs: [Var(0x55a736577430:1:2:3:i1:o2:s0,float32,,0x7f49b9500000)[10,5,38,48,],] [i 1103 17:01:13.033074 16 v100 executor.cc:551] Finished Op(fused 3/8) output: [Var(0x55a736577430:1:2:3:i1:o2:s0,float32,,0x7f49b9500000)[10,5,38,48,],] [i 1103 17:01:13.033081 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7365c9ff0:1:1:2:i1:o1:s0,broadcast_to->0x55a737244740),Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a7365a2900),Op(0x55a737117180:2:1:2:i2:o1:s0,binary.multiply->0x55a737134600),Op(0x55a7370ead80:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0),] [i 1103 17:01:13.033092 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7373379c0) [i 1103 17:01:13.033098 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7373379c0) inputs: [] outputs: [Var(0x55a7373379c0:2:2:2:i1:o1:s0,float32,,0x7f49b9559200)[10,5,38,48,],] [i 1103 17:01:13.046859 16 v100 op.cc:254] Jit op key found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f49785eff10 [i 1103 17:01:13.046962 16 v100 executor.cc:551] Finished Op(fused 4/8) output: [Var(0x55a7373379c0:2:2:2:i1:o1:s0,float32,,0x7f49b9559200)[10,5,38,48,],] [i 1103 17:01:13.046971 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a7370ed2a0:1:1:2:i1:o1:s0,broadcast_to->0x55a737339d20),Op(0x55a7370ea850:1:1:2:i1:o1:s0,broadcast_to->0x55a737244e20),Op(0x55a73710e200:2:1:2:i2:o1:s0,binary.multiply->0x55a73711ae70),Op(0x55a737102ad0:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a7372454e0),] [i 1103 17:01:13.046983 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7372454e0) [i 1103 17:01:13.046991 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7372454e0) inputs: [] outputs: [Var(0x55a7372454e0:2:1:1:i1:o0:s0,float32,,0x7f49b94a7c00)[10,4,40,50,],] [i 1103 17:01:13.060729 16 v100 op.cc:254] Jit op key found: cudnn_conv_backward_x[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f49781d7100 [i 1103 17:01:13.074692 16 v100 executor.cc:551] Finished Op(fused 5/8) output: [Var(0x55a7372454e0:2:1:1:i1:o0:s0,float32,,0x7f49b94a7c00)[10,4,40,50,],] [i 1103 17:01:13.074734 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0),Op(0x55a73711c220:1:1:2:i1:o1:s0,reindex->0x55a7370f2010),Op(0x55a737339680:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0),Op(0x55a7365ceb70:1:1:2:i1:o1:s0,reduce.add->0x55a73734ef40),] [i 1103 17:01:13.074758 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734ef40) [i 1103 17:01:13.074771 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734ef40) inputs: [] outputs: [Var(0x55a73734ef40:2:1:1:i1:o0:s0,float32,,0x7f49b94f5e00)[5,4,5,5,],] [i 1103 17:01:13.094181 16 v100 op.cc:254] Jit op key found: cudnn_conv_backward_w[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4945bf3438 [i 1103 17:01:13.109275 16 v100 executor.cc:551] Finished Op(fused 6/8) output: [Var(0x55a73734ef40:2:1:1:i1:o0:s0,float32,,0x7f49b94f5e00)[5,4,5,5,],] [i 1103 17:01:13.109301 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a73734f880:2:1:2:i2:o1:s0,binary.multiply->0x55a73710e780),] [i 1103 17:01:13.109316 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73710e780) [i 1103 17:01:13.110649 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73710e780) inputs: [] outputs: [Var(0x55a73710e780:2:1:1:i1:o0:s0,float32,,0x7f4946800000)[10,5,38,48,],] [i 1103 17:01:13.110696 16 v100 executor.cc:551] Finished Op(fused 7/8) output: [Var(0x55a73710e780:2:1:1:i1:o0:s0,float32,,0x7f4946800000)[10,5,38,48,],] [i 1103 17:01:13.110711 16 v10 executor.cc:592] All 19 ops finished, return vars: [Var(0x55a7373379c0:2:2:1:i1:o1:s1,float32,,0x7f49b9559200)[10,5,38,48,],Var(0x55a73710e780:2:1:0:i1:o0:s1,float32,,0x7f4946800000)[10,5,38,48,],Var(0x55a7372454e0:2:1:0:i1:o0:s1,float32,,0x7f49b94a7c00)[10,4,40,50,],Var(0x55a73734ef40:2:1:0:i1:o0:s1,float32,,0x7f49b94f5e00)[5,4,5,5,],] [i 1103 17:01:13.110732 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 8 device_sync: 0 [i 1103 17:01:13.110776 16 v1 cuda_flags.cc:34] CUDA disabled. [i 1103 17:01:13.110796 16 v10 executor.cc:592] All 0 ops finished, return vars: [Var(0x55a73734ef40:2:1:0:i1:o0:s1,float32,,0x7f49b94f5e00)[5,4,5,5,],Var(0x55a7372454e0:2:1:0:i1:o0:s1,float32,,0x7f49b94a7c00)[10,4,40,50,],Var(0x55a73710e780:2:1:0:i1:o0:s1,float32,,0x7f4946800000)[10,5,38,48,],] [i 1103 17:01:13.110808 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 0 device_sync: 0 [i 1103 17:01:13.184799 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.184934 16 v100 executor.cc:250] id: 0 type: reduce addr: Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0) [i 1103 17:01:13.184949 16 v100 executor.cc:259] input: 4 addr: Op(0x55a737110490:2:1:2:i2:o1:s0,binary.multiply->0x55a737253bf0) [i 1103 17:01:13.184956 16 v100 executor.cc:261] [i 1103 17:01:13.184961 16 v100 executor.cc:250] id: 1 type: element addr: Op(0x55a73710e200:2:1:2:i2:o1:s0,binary.multiply->0x55a7370fe300) [i 1103 17:01:13.184967 16 v100 executor.cc:259] input: 5 addr: Op(0x55a737114690:0:1:1:i0:o1:s0,curand_random->0x55a7370eaa40) [i 1103 17:01:13.184973 16 v100 executor.cc:259] input: 0 addr: Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0) [i 1103 17:01:13.184979 16 v100 executor.cc:261] [i 1103 17:01:13.184983 16 v100 executor.cc:250] id: 2 type: reduce addr: Op(0x55a737331780:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a73734fa30) [i 1103 17:01:13.184989 16 v100 executor.cc:259] input: 6 addr: Op(0x55a7365ceb70:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0) [i 1103 17:01:13.184995 16 v100 executor.cc:261] [i 1103 17:01:13.184999 16 v100 executor.cc:250] id: 3 type: reduce addr: Op(0x55a737117180:1:1:2:i1:o1:s0,reduce.add->0x55a7370f86a0) [i 1103 17:01:13.185005 16 v100 executor.cc:259] input: 7 addr: Op(0x55a737353850:2:1:2:i2:o1:s0,binary.multiply->0x55a737113b30) [i 1103 17:01:13.185010 16 v100 executor.cc:261] [i 1103 17:01:13.185014 16 v100 executor.cc:250] id: 4 type: element addr: Op(0x55a737110490:2:1:2:i2:o1:s0,binary.multiply->0x55a737253bf0) [i 1103 17:01:13.185020 16 v100 executor.cc:259] input: 8 addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a737244740) [i 1103 17:01:13.185026 16 v100 executor.cc:259] input: 9 addr: Op(0x55a737134600:1:1:2:i1:o1:s0,broadcast_to->0x55a73710e460) [i 1103 17:01:13.185033 16 v100 executor.cc:261] [i 1103 17:01:13.185037 16 v100 executor.cc:250] id: 5 type: others addr: Op(0x55a737114690:0:1:1:i0:o1:s0,curand_random->0x55a7370eaa40) [i 1103 17:01:13.185043 16 v100 executor.cc:261] [i 1103 17:01:13.185047 16 v100 executor.cc:250] id: 6 type: element addr: Op(0x55a7365ceb70:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0) [i 1103 17:01:13.185053 16 v100 executor.cc:259] input: 10 addr: Op(0x55a7370f2010:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0) [i 1103 17:01:13.185059 16 v100 executor.cc:259] input: 11 addr: Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a736577430) [i 1103 17:01:13.185065 16 v100 executor.cc:261] [i 1103 17:01:13.185069 16 v100 executor.cc:250] id: 7 type: element addr: Op(0x55a737353850:2:1:2:i2:o1:s0,binary.multiply->0x55a737113b30) [i 1103 17:01:13.185074 16 v100 executor.cc:259] input: 12 addr: Op(0x55a737328490:1:1:2:i1:o1:s0,reindex->0x55a73734f0d0) [i 1103 17:01:13.185080 16 v100 executor.cc:259] input: 13 addr: Op(0x55a737103240:1:1:2:i1:o1:s0,broadcast_to->0x55a7370ebe60) [i 1103 17:01:13.185085 16 v100 executor.cc:261] [i 1103 17:01:13.185089 16 v100 executor.cc:250] id: 8 type: broadcast addr: Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a737244740) [i 1103 17:01:13.185094 16 v100 executor.cc:259] input: 14 addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7365c9ff0) [i 1103 17:01:13.185099 16 v100 executor.cc:261] [i 1103 17:01:13.185104 16 v100 executor.cc:250] id: 9 type: broadcast addr: Op(0x55a737134600:1:1:2:i1:o1:s0,broadcast_to->0x55a73710e460) [i 1103 17:01:13.185110 16 v100 executor.cc:259] input: 15 addr: Op(0x55a7370ead80:0:1:1:i0:o1:s0,curand_random->0x55a7365a2900) [i 1103 17:01:13.185116 16 v100 executor.cc:261] [i 1103 17:01:13.185120 16 v100 executor.cc:250] id: 10 type: broadcast addr: Op(0x55a7370f2010:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0) [i 1103 17:01:13.185126 16 v100 executor.cc:259] input: 15 addr: Op(0x55a7370ead80:0:1:1:i0:o1:s0,curand_random->0x55a7365a2900) [i 1103 17:01:13.185131 16 v100 executor.cc:261] [i 1103 17:01:13.185135 16 v100 executor.cc:250] id: 11 type: broadcast addr: Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a736577430) [i 1103 17:01:13.185140 16 v100 executor.cc:259] input: 16 addr: Op(0x55a737339680:1:1:2:i2:o1:s0,binary.multiply->0x55a737244e20) [i 1103 17:01:13.185145 16 v100 executor.cc:261] [i 1103 17:01:13.185149 16 v100 executor.cc:250] id: 12 type: broadcast addr: Op(0x55a737328490:1:1:2:i1:o1:s0,reindex->0x55a73734f0d0) [i 1103 17:01:13.185155 16 v100 executor.cc:259] input: 14 addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7365c9ff0) [i 1103 17:01:13.185160 16 v100 executor.cc:261] [i 1103 17:01:13.185165 16 v100 executor.cc:250] id: 13 type: broadcast addr: Op(0x55a737103240:1:1:2:i1:o1:s0,broadcast_to->0x55a7370ebe60) [i 1103 17:01:13.218807 16 v100 executor.cc:259] input: 16 addr: Op(0x55a737339680:1:1:2:i2:o1:s0,binary.multiply->0x55a737244e20) [i 1103 17:01:13.218854 16 v100 executor.cc:261] [i 1103 17:01:13.218867 16 v100 executor.cc:250] id: 14 type: others addr: Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7365c9ff0) [i 1103 17:01:13.218884 16 v100 executor.cc:261] [i 1103 17:01:13.218896 16 v100 executor.cc:250] id: 15 type: others addr: Op(0x55a7370ead80:0:1:1:i0:o1:s0,curand_random->0x55a7365a2900) [i 1103 17:01:13.218910 16 v100 executor.cc:261] [i 1103 17:01:13.219168 16 v100 executor.cc:250] id: 16 type: element addr: Op(0x55a737339680:1:1:2:i2:o1:s0,binary.multiply->0x55a737244e20) [i 1103 17:01:13.219189 16 v100 executor.cc:259] input: 5 addr: Op(0x55a737114690:0:1:1:i0:o1:s0,curand_random->0x55a7370eaa40) [i 1103 17:01:13.219203 16 v100 executor.cc:259] input: 17 addr: Op(0x55a7370ea850:0:1:1:i1:o1:s0,broadcast_to->0x55a7370ed2a0) [i 1103 17:01:13.219218 16 v100 executor.cc:261] [i 1103 17:01:13.219229 16 v100 executor.cc:250] id: 17 type: broadcast addr: Op(0x55a7370ea850:0:1:1:i1:o1:s0,broadcast_to->0x55a7370ed2a0) [i 1103 17:01:13.219246 16 v100 executor.cc:259] input: 18 addr: Op(0x55a737276ba0:0:1:1:i0:o1:s0,array->0x55a73710f2f0) [i 1103 17:01:13.219261 16 v100 executor.cc:261] [i 1103 17:01:13.219273 16 v100 executor.cc:250] id: 18 type: element addr: Op(0x55a737276ba0:0:1:1:i0:o1:s0,array->0x55a73710f2f0) [i 1103 17:01:13.219288 16 v100 executor.cc:261] [i 1103 17:01:13.219323 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.219337 16 v1000 executor.cc:446] topsort internal [1,] [i 1103 17:01:13.219349 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.219360 16 v1000 executor.cc:446] topsort internal [13,12,7,3,] [i 1103 17:01:13.219372 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.219383 16 v1000 executor.cc:446] topsort internal [11,10,6,2,] [i 1103 17:01:13.219401 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.219412 16 v1000 executor.cc:446] topsort internal [9,8,4,0,] [i 1103 17:01:13.219425 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.219436 16 v1000 executor.cc:446] topsort internal [18,17,16,] [i 1103 17:01:13.219447 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.219458 16 v1000 executor.cc:446] topsort internal [15,] [i 1103 17:01:13.219469 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.219480 16 v1000 executor.cc:446] topsort internal [14,] [i 1103 17:01:13.219490 16 v1000 executor.cc:426] sharegraph_q [] [i 1103 17:01:13.219501 16 v1000 executor.cc:446] topsort internal [5,] [i 1103 17:01:13.219516 16 v100 executor.cc:481] Run Op(0x55a737114690:0:1:1:i0:o1:s0,curand_random->0x55a7370eaa40) [i 1103 17:01:13.219537 16 v100 executor.cc:490] Run Op(0x55a737114690:0:1:1:i0:o1:s0,curand_random->0x55a7370eaa40) inputs: [] outputs: [Var(0x55a7370eaa40:1:3:3:i1:o2:s0,float32,,0x7f49b9400000)[10,5,13,17,],] [i 1103 17:01:13.219562 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:13.219627 16 v100 executor.cc:551] Finished Op(curand_random 0/8) output: [Var(0x55a7370eaa40:1:3:3:i1:o2:s0,float32,,0x7f49b9400000)[10,5,13,17,],] [i 1103 17:01:13.219663 16 v100 executor.cc:481] Run Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7365c9ff0) [i 1103 17:01:13.219679 16 v100 executor.cc:490] Run Op(0x55a7370ea960:0:1:1:i0:o1:s0,curand_random->0x55a7365c9ff0) inputs: [] outputs: [Var(0x55a7365c9ff0:1:3:3:i1:o2:s0,float32,,0x7f49b940ae00)[10,4,40,50,],] [i 1103 17:01:13.219698 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:13.219723 16 v100 executor.cc:551] Finished Op(curand_random 1/8) output: [Var(0x55a7365c9ff0:1:3:3:i1:o2:s0,float32,,0x7f49b940ae00)[10,4,40,50,],] [i 1103 17:01:13.219741 16 v100 executor.cc:481] Run Op(0x55a7370ead80:0:1:1:i0:o1:s0,curand_random->0x55a7365a2900) [i 1103 17:01:13.219754 16 v100 executor.cc:490] Run Op(0x55a7370ead80:0:1:1:i0:o1:s0,curand_random->0x55a7365a2900) inputs: [] outputs: [Var(0x55a7365a2900:1:3:3:i1:o2:s0,float32,,0x7f49b9459000)[5,4,4,4,],] [i 1103 17:01:13.219773 16 v100 op.cc:254] Jit op key found: curand_random[T:float32][R:uniform][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4a0a1e50be [i 1103 17:01:13.219795 16 v100 executor.cc:551] Finished Op(curand_random 2/8) output: [Var(0x55a7365a2900:1:3:3:i1:o2:s0,float32,,0x7f49b9459000)[5,4,4,4,],] [i 1103 17:01:13.219813 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a737276ba0:0:1:1:i0:o1:s0,array->0x55a73710f2f0),Op(0x55a7370ea850:0:1:1:i1:o1:s0,broadcast_to->0x55a7370ed2a0),Op(0x55a737339680:1:1:2:i2:o1:s0,binary.multiply->0x55a737244e20),] [i 1103 17:01:13.219837 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a737244e20) [i 1103 17:01:13.219852 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a737244e20) inputs: [] outputs: [Var(0x55a737244e20:1:2:3:i1:o2:s0,float32,,0x7f49b9459600)[10,5,13,17,],] [i 1103 17:01:13.219896 16 v100 executor.cc:551] Finished Op(fused 3/8) output: [Var(0x55a737244e20:1:2:3:i1:o2:s0,float32,,0x7f49b9459600)[10,5,13,17,],] [i 1103 17:01:13.219915 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a737134600:1:1:2:i1:o1:s0,broadcast_to->0x55a73710e460),Op(0x55a73732b060:1:1:2:i1:o1:s0,reindex->0x55a737244740),Op(0x55a737110490:2:1:2:i2:o1:s0,binary.multiply->0x55a737253bf0),Op(0x55a73712bab0:1:1:2:i1:o1:s0,reduce.add->0x55a7373379c0),] [i 1103 17:01:13.219940 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7373379c0) [i 1103 17:01:13.219955 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7373379c0) inputs: [] outputs: [Var(0x55a7373379c0:2:2:2:i1:o1:s0,float32,,0x7f49b9464400)[10,5,13,17,],] [i 1103 17:01:13.243395 16 v100 op.cc:254] Jit op key found: cudnn_conv[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f49785eff10 [i 1103 17:01:13.243493 16 v100 executor.cc:551] Finished Op(fused 4/8) output: [Var(0x55a7373379c0:2:2:2:i1:o1:s0,float32,,0x7f49b9464400)[10,5,13,17,],] [i 1103 17:01:13.243503 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a737253540:1:1:2:i1:o1:s0,broadcast_to->0x55a736577430),Op(0x55a7370f2010:1:1:2:i1:o1:s0,broadcast_to->0x55a73711a0a0),Op(0x55a7365ceb70:2:1:2:i2:o1:s0,binary.multiply->0x55a7371162f0),Op(0x55a737331780:1:1:2:i1:o1:s0,reindex_reduce.add->0x55a73734fa30),] [i 1103 17:01:13.243515 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734fa30) [i 1103 17:01:13.243523 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a73734fa30) inputs: [] outputs: [Var(0x55a73734fa30:2:1:1:i1:o0:s0,float32,,0x7f49b946f200)[10,4,40,50,],] [i 1103 17:01:13.257515 16 v100 op.cc:254] Jit op key found: cudnn_conv_backward_x[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f49781d7100 [i 1103 17:01:13.261385 16 v100 executor.cc:551] Finished Op(fused 5/8) output: [Var(0x55a73734fa30:2:1:1:i1:o0:s0,float32,,0x7f49b946f200)[10,4,40,50,],] [i 1103 17:01:13.261412 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a737103240:1:1:2:i1:o1:s0,broadcast_to->0x55a7370ebe60),Op(0x55a737328490:1:1:2:i1:o1:s0,reindex->0x55a73734f0d0),Op(0x55a737353850:2:1:2:i2:o1:s0,binary.multiply->0x55a737113b30),Op(0x55a737117180:1:1:2:i1:o1:s0,reduce.add->0x55a7370f86a0),] [i 1103 17:01:13.261431 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370f86a0) [i 1103 17:01:13.261442 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370f86a0) inputs: [] outputs: [Var(0x55a7370f86a0:2:1:1:i1:o0:s0,float32,,0x7f49b94bd400)[5,4,4,4,],] [i 1103 17:01:13.279361 16 v100 op.cc:254] Jit op key found: cudnn_conv_backward_w[Tx:float32][Ty:float32][Tw:float32][XFORMAT:abcd][WFORMAT:oihw][YFORMAT:abcd][JIT:1][JIT_cuda:1][index_t:int32] jit op entry: 0x7f4945bf3438 [i 1103 17:01:13.287683 16 v100 executor.cc:551] Finished Op(fused 6/8) output: [Var(0x55a7370f86a0:2:1:1:i1:o0:s0,float32,,0x7f49b94bd400)[5,4,4,4,],] [i 1103 17:01:13.287716 16 v100 executor.cc:60] Prepare fused_op [Op(0x55a73710e200:2:1:2:i2:o1:s0,binary.multiply->0x55a7370fe300),] [i 1103 17:01:13.287730 16 v100 executor.cc:481] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370fe300) [i 1103 17:01:13.287742 16 v100 executor.cc:490] Run Op(0x7ffe7aef09b0:0:0:0:i0:o1:s0,fused->0x55a7370fe300) inputs: [] outputs: [Var(0x55a7370fe300:2:1:1:i1:o0:s0,float32,,0x7f49b94bda00)[10,5,13,17,],] [i 1103 17:01:13.291295 16 v100 executor.cc:551] Finished Op(fused 7/8) output: [Var(0x55a7370fe300:2:1:1:i1:o0:s0,float32,,0x7f49b94bda00)[10,5,13,17,],] [i 1103 17:01:13.291313 16 v10 executor.cc:592] All 19 ops finished, return vars: [Var(0x55a7373379c0:2:2:1:i1:o1:s1,float32,,0x7f49b9464400)[10,5,13,17,],Var(0x55a7370fe300:2:1:0:i1:o0:s1,float32,,0x7f49b94bda00)[10,5,13,17,],Var(0x55a73734fa30:2:1:0:i1:o0:s1,float32,,0x7f49b946f200)[10,4,40,50,],Var(0x55a7370f86a0:2:1:0:i1:o0:s1,float32,,0x7f49b94bd400)[5,4,4,4,],] [i 1103 17:01:13.291327 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 8 device_sync: 0 [i 1103 17:01:13.291421 16 v1 cuda_flags.cc:34] CUDA disabled. [i 1103 17:01:13.291444 16 v10 executor.cc:592] All 0 ops finished, return vars: [Var(0x55a7370f86a0:2:1:0:i1:o0:s1,float32,,0x7f49b94bd400)[5,4,4,4,],Var(0x55a73734fa30:2:1:0:i1:o0:s1,float32,,0x7f49b946f200)[10,4,40,50,],Var(0x55a7370fe300:2:1:0:i1:o0:s1,float32,,0x7f49b94bda00)[10,5,13,17,],] [i 1103 17:01:13.291455 16 v10 executor.cc:606] cudaDeviceSynchronize times: 0 / 0 device_sync: 0 ..[i 1103 17:01:13.319210 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.425845 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.493165 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.561802 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.620734 16 cuda_flags.cc:32] CUDA enabled. F[i 1103 17:01:13.677055 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.688857 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.718698 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.723490 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.779703 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.783386 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.842737 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.845882 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.903293 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.909218 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.947171 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.953387 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:13.991226 16 cuda_flags.cc:32] CUDA enabled. [i 1103 17:01:14.005891 16 cuda_flags.cc:32] CUDA enabled. .. ====================================================================== FAIL: test_conv3d (main.TestCudnnConvOp) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/liuzhian/anaconda3/envs/jittor/lib/python3.7/site-packages/jittor/test/test_cudnn_op.py", line 153, in test_conv3d check((2,4,10,10,10), (5,4,3,4,5), (1,1,1), (1,1,1)) File "/home/liuzhian/anaconda3/envs/jittor/lib/python3.7/site-packages/jittor/test/test_cudnn_op.py", line 145, in check np.testing.assert_allclose(y.data, y2.data) File "/home/liuzhian/anaconda3/envs/jittor/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 1531, in assert_allclose verbose=verbose, header=header, equal_nan=equal_nan) File "/home/liuzhian/anaconda3/envs/jittor/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 844, in assert_array_compare raise AssertionError(msg) AssertionError: Not equal to tolerance rtol=1e-07, atol=0 Mismatched elements: 4623 / 7200 (64.2%) Max absolute difference: 5.340576e-05 Max relative difference: 8.419145e-07 x: array([[[[[26.199913, 32.35394 , 31.798569, ..., 27.34977 , 27.382778, 20.727432], [34.28533 , 41.313995, 40.744705, ..., 36.90609 , 39.26516 ,... y: array([[[[[26.199905, 32.35394 , 31.79857 , ..., 27.349777, 27.38277 , 20.72743 ], [34.28532 , 41.314003, 40.744694, ..., 36.906097, 39.265167,... ---------------------------------------------------------------------- Ran 5 tests in 1.481s FAILED (failures=1)
I can pass the test process via python -m jittor.test.test_example

My Envs

Ubuntu 16.04

g++ 5.4.0

cuda 10.0

cudnn 7.6.5

python 3.7 under conda

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

LiUzHiAn commented 2 years ago

So, just ignore this failure case and use Jottor as normal?

ssmem commented 2 years ago

I have the same problem and my environment is right.

fabro66 commented 2 years ago

Hi~ I also met the same problem. Could you please tell me how to solve it.

te/utils.py", line 844, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=0.001

Mismatched elements: 80 / 540 (14.8%)
Max absolute difference: 0.0144043
Max relative difference: 3.3597564e-05
 x: array([[[[[359.2016 , 414.49707, 351.72098],
          [403.03696, 447.25607, 400.26517],
          [354.52756, 411.22162, 354.2814 ]],...
 y: array([[[[[359.20465, 414.499  , 351.72192],
          [403.03748, 447.25623, 400.26544],
          [354.53018, 411.22043, 354.28287]],...

----------------------------------------------------------------------
Ran 5 tests in 2.270s

FAILED (failures=1)

My Envs