search

DetectionTeamUCAS / Cascade_FPN_Tensorflow

Cascade R-CNN: Delving into High Quality Object Detection.
MIT License
35 stars 10 forks source link

Training automatically closes #10

Open g18tka17 opened 4 years ago

g18tka17 commented 4 years ago

hello,author. i meet a question when i run your code(GTX 1080 Ti), sometime report this error 2019-12-12 14:41:58.024277: E tensorflow/stream_executor/cuda/cuda_dnn.cc:81] 2019-12-12 14:41:58.024505: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a617f8d40 2019-12-12 14:41:58.024537: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a3bffdd40 2019-12-12 14:41:58.024574: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a3bffdd40 2019-12-12 14:41:58.024542: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a617f8d40 2019-12-12 14:41:58.027290: I tensorflow/stream_executor/stream.cc:1936] stream 0x560dcbf2bd50 did not enqueue 'start timer': 0x7f2a61ff9d30 2019-12-12 14:41:58.027373: I tensorflow/stream_executor/stream.cc:1947] stream 0x560dcbf2bd50 did not enqueue 'stop timer': 0x7f2a61ff9d30 2019-12-12 14:41:58.027400: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.027426: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.027456: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a3b7fcd40 2019-12-12 14:41:58.027499: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a617f8d40 2019-12-12 14:41:58.027884: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.027907: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.027926: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.027944: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.027962: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.028478: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.028501: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.028519: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.028537: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.028555: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.028829: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.028898: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a61ff9d40 2019-12-12 14:41:58.029039: I tensorflow/stream_executor/stream.cc:4818] stream 0x560dcbf2bd50 did not memzero GPU location; source: 0x7f2a617f8d40 2019-12-12 14:41:58.029051: I tensorflow/stream_executor/stream.cc:302] did not allocate timer: 0x7f2a617f8d30 2019-12-12 14:41:58.029058: I tensorflow/stream_executor/stream.cc:1936] stream 0x560dcbf2bd50 did not enqueue 'start timer': 0x7f2a617f8d30 2019-12-12 14:41:58.029069: I tensorflow/stream_executor/stream.cc:1947] stream 0x560dcbf2bd50 did not enqueue 'stop timer': 0x7f2a617f8d30 2019-12-12 14:41:58.029076: F tensorflow/stream_executor/cuda/cuda_timer.cc:65] Check failed: startevent != nullptr && stopevent != nullptr Aborted (core dumped)

sometimes reports this error: InternalError (see above for traceback): cuDNN Backward Filter function launch failure : input shape([1,256,150,200]) filter shape([3,3,256,512]) [[Node: tower_0/gradients/tower_0/build_rpn/rpn_conv/3x3/Conv2D_grad/Conv2DBackpropFilter = Conv2DBackpropFilter[T=DT_FLOAT, _class=["loc:@tower_0/gradients/AddN_20"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](tower_0/build_pyramid/fuse_P2/BiasAdd, ConstantFolding/tower_0/gradients/tower_0/build_rpn/rpn_conv/3x3/Conv2D_grad/ShapeN-matshapes-1, tower_0/gradients/tower_0/build_rpn/rpn_conv/3x3/Relu_grad/ReluGrad, ^tower_0/gradients/tower_0/build_rpn/rpn_conv/3x3/BiasAdd_grad/BiasAddGrad)]]

I can't find the answer from Google，could you help me? thank you very much.

g18tka17 commented 4 years ago

by the way, i use pascal voc2007 training dataset.