matterport / Mask_RCNN

Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow
Other
24.7k stars 11.71k forks source link

change save_weight_only to save_best_only caused problem #530

Open xiaolisarah opened 6 years ago

xiaolisarah commented 6 years ago

/home/jgq/anaconda3/envs/python34/bin/python /media/jgq/GXL/project/2018/DDIM-OD/train_ddim.py train --dataset=data_process --weight=coco Using TensorFlow backend. Weights: coco Dataset: data_process Logs: /media/jgq/GXL/project/2018/DDIM-OD/logs

Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[256 256] [128 128] [ 64 64] [ 32 32] [ 16 16]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.9 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 1024 IMAGE_MIN_DIM 800 IMAGE_PADDING True IMAGE_SHAPE [1024 1024 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME bioisland NUM_CLASSES 5 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (32, 64, 128, 256, 512) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 100 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 5 WEIGHT_DECAY 0.0001

Loading weights /media/jgq/GXL/project/2018/DDIM-OD/mask_rcnn_coco.h5 2018-05-08 09:34:54.784447: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2018-05-08 09:34:54.784468: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2018-05-08 09:34:54.784486: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2018-05-08 09:34:54.784489: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2018-05-08 09:34:54.784506: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. 2018-05-08 09:34:54.940052: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2018-05-08 09:34:54.940281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate (GHz) 1.835 pciBusID 0000:01:00.0 Total memory: 7.92GiB Free memory: 7.41GiB 2018-05-08 09:34:54.940291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 2018-05-08 09:34:54.940295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0: Y 2018-05-08 09:34:54.940300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0) starting prepare train data starting prepare val data Training network heads

Starting at epoch 0. LR=0.001

Checkpoint Path: /media/jgq/GXL/project/2018/DDIM-OD/logs/bioisland20180508T0934/mask_rcnnbioisland{epoch:04d}.h5 Selecting layers to train fpn_c5p5 (Conv2D) fpn_c4p4 (Conv2D) fpn_c3p3 (Conv2D) fpn_c2p2 (Conv2D) fpn_p5 (Conv2D) fpn_p2 (Conv2D) fpn_p3 (Conv2D) fpn_p4 (Conv2D) In model: rpn_model rpn_conv_shared (Conv2D) rpn_class_raw (Conv2D) rpn_bbox_pred (Conv2D) mrcnn_mask_conv1 (TimeDistributed) mrcnn_mask_bn1 (TimeDistributed) mrcnn_mask_conv2 (TimeDistributed) mrcnn_mask_bn2 (TimeDistributed) mrcnn_class_conv1 (TimeDistributed) mrcnn_class_bn1 (TimeDistributed) mrcnn_mask_conv3 (TimeDistributed) mrcnn_mask_bn3 (TimeDistributed) mrcnn_class_conv2 (TimeDistributed) mrcnn_class_bn2 (TimeDistributed) mrcnn_mask_conv4 (TimeDistributed) mrcnn_mask_bn4 (TimeDistributed) mrcnn_bbox_fc (TimeDistributed) mrcnn_mask_deconv (TimeDistributed) mrcnn_class_logits (TimeDistributed) mrcnn_mask (TimeDistributed) /home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/tensorflow/python/ops/gradients_impl.py:95: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " /home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/keras/engine/training.py:1987: UserWarning: Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using thekeras.utils.Sequence class. UserWarning('Using a generator withuse_multiprocessing=True`' Epoch 1/10 /home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/scipy/ndimage/interpolation.py:616: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed. "the returned array has changed.", UserWarning) /home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/scipy/ndimage/interpolation.py:616: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed. "the returned array has changed.", UserWarning) 1/100 [..............................] - ETA: 410s - loss: 4.8978 - rpn_class_loss: 0.0816 - rpn_bbox_loss: 3.0647 - mrcnn_class_loss: 1.7514 - mrcnn_bbox_loss: 0.0000e+00 - mrcnn_mask_loss: 0.0000e+00 2/100 [..............................] - ETA: 227s - loss: 4.6023 - rpn_class_loss: 0.0419 - rpn_bbox_loss: 1.9728 - mrcnn_class_loss: 1.4380 - mrcnn_bbox_loss: 0.6925 - mrcnn_mask_loss: 0.4571
3/100 [..............................] - ETA: 167s - loss: 3.8355 - rpn_class_loss: 0.0296 - rpn_bbox_loss: 1.4527 - mrcnn_class_loss: 1.1266 - mrcnn_bbox_loss: 0.7389 - mrcnn_mask_loss: 0.4877 4/100 [>.............................] - ETA: 136s - loss: 3.3419 - rpn_class_loss: 0.0360 - rpn_bbox_loss: 1.1600 - mrcnn_class_loss: 0.9058 - mrcnn_bbox_loss: 0.7312 - mrcnn_mask_loss: 0.5088 5/100 [>.............................] - ETA: 117s - loss: 3.5668 - rpn_class_loss: 0.0351 - rpn_bbox_loss: 1.8142 - mrcnn_class_loss: 0.7255 - mrcnn_bbox_loss: 0.5849 - mrcnn_mask_loss: 0.4071 6/100 [>.............................] - ETA: 105s - loss: 3.8032 - rpn_class_loss: 0.0336 - rpn_bbox_loss: 2.3381 - mrcnn_class_loss: 0.6048 - mrcnn_bbox_loss: 0.4874 - mrcnn_mask_loss: 0.3392 7/100 [=>............................] - ETA: 95s - loss: 3.6564 - rpn_class_loss: 0.0297 - rpn_bbox_loss: 2.1107 - mrcnn_class_loss: 0.5572 - mrcnn_bbox_loss: 0.5503 - mrcnn_mask_loss: 0.4085 8/100 [=>............................] - ETA: 89s - loss: 3.8016 - rpn_class_loss: 0.0435 - rpn_bbox_loss: 2.4317 - mrcnn_class_loss: 0.4876 - mrcnn_bbox_loss: 0.4815 - mrcnn_mask_loss: 0.3574 9/100 [=>............................] - ETA: 83s - loss: 3.6490 - rpn_class_loss: 0.0400 - rpn_bbox_loss: 2.2238 - mrcnn_class_loss: 0.4707 - mrcnn_bbox_loss: 0.5142 - mrcnn_mask_loss: 0.4003 10/100 [==>...........................] - ETA: 78s - loss: 3.8509 - rpn_class_loss: 0.0417 - rpn_bbox_loss: 2.3887 - mrcnn_class_loss: 0.4257 - mrcnn_bbox_loss: 0.5512 - mrcnn_mask_loss: 0.4436 11/100 [==>...........................] - ETA: 75s - loss: 3.7439 - rpn_class_loss: 0.0435 - rpn_bbox_loss: 2.2545 - mrcnn_class_loss: 0.4008 - mrcnn_bbox_loss: 0.5820 - mrcnn_mask_loss: 0.4631 12/100 [==>...........................] - ETA: 71s - loss: 3.6049 - rpn_class_loss: 0.0412 - rpn_bbox_loss: 2.0746 - mrcnn_class_loss: 0.3779 - mrcnn_bbox_loss: 0.6244 - mrcnn_mask_loss: 0.4869 13/100 [==>...........................] - ETA: 68s - loss: 3.5330 - rpn_class_loss: 0.0392 - rpn_bbox_loss: 1.9521 - mrcnn_class_loss: 0.3714 - mrcnn_bbox_loss: 0.6446 - mrcnn_mask_loss: 0.5257 14/100 [===>..........................] - ETA: 66s - loss: 3.4339 - rpn_class_loss: 0.0376 - rpn_bbox_loss: 1.8234 - mrcnn_class_loss: 0.3529 - mrcnn_bbox_loss: 0.6849 - mrcnn_mask_loss: 0.5351 15/100 [===>..........................] - ETA: 64s - loss: 3.2090 - rpn_class_loss: 0.0370 - rpn_bbox_loss: 1.7039 - mrcnn_class_loss: 0.3294 - mrcnn_bbox_loss: 0.6392 - mrcnn_mask_loss: 0.4994 16/100 [===>..........................] - ETA: 62s - loss: 3.1388 - rpn_class_loss: 0.0368 - rpn_bbox_loss: 1.6199 - mrcnn_class_loss: 0.3278 - mrcnn_bbox_loss: 0.6515 - mrcnn_mask_loss: 0.5028 17/100 [====>.........................] - ETA: 60s - loss: 3.0607 - rpn_class_loss: 0.0352 - rpn_bbox_loss: 1.5324 - mrcnn_class_loss: 0.3310 - mrcnn_bbox_loss: 0.6547 - mrcnn_mask_loss: 0.5075 18/100 [====>.........................] - ETA: 59s - loss: 3.0089 - rpn_class_loss: 0.0356 - rpn_bbox_loss: 1.4572 - mrcnn_class_loss: 0.3307 - mrcnn_bbox_loss: 0.6677 - mrcnn_mask_loss: 0.5178 19/100 [====>.........................] - ETA: 57s - loss: 2.9409 - rpn_class_loss: 0.0343 - rpn_bbox_loss: 1.3839 - mrcnn_class_loss: 0.3221 - mrcnn_bbox_loss: 0.6661 - mrcnn_mask_loss: 0.5346 20/100 [=====>........................] - ETA: 56s - loss: 2.9232 - rpn_class_loss: 0.0366 - rpn_bbox_loss: 1.3771 - mrcnn_class_loss: 0.3072 - mrcnn_bbox_loss: 0.6509 - mrcnn_mask_loss: 0.5513 21/100 [=====>........................] - ETA: 54s - loss: 3.0412 - rpn_class_loss: 0.0366 - rpn_bbox_loss: 1.5670 - mrcnn_class_loss: 0.2925 - mrcnn_bbox_loss: 0.6199 - mrcnn_mask_loss: 0.5251 22/100 [=====>........................] - ETA: 53s - loss: 2.9590 - rpn_class_loss: 0.0355 - rpn_bbox_loss: 1.5090 - mrcnn_class_loss: 0.2831 - mrcnn_bbox_loss: 0.6127 - mrcnn_mask_loss: 0.5186 23/100 [=====>........................] - ETA: 52s - loss: 2.9014 - rpn_class_loss: 0.0344 - rpn_bbox_loss: 1.4518 - mrcnn_class_loss: 0.2767 - mrcnn_bbox_loss: 0.6211 - mrcnn_mask_loss: 0.5174 24/100 [======>.......................] - ETA: 51s - loss: 2.9078 - rpn_class_loss: 0.0335 - rpn_bbox_loss: 1.4113 - mrcnn_class_loss: 0.2771 - mrcnn_bbox_loss: 0.6342 - mrcnn_mask_loss: 0.5518 25/100 [======>.......................] - ETA: 50s - loss: 2.8614 - rpn_class_loss: 0.0322 - rpn_bbox_loss: 1.3599 - mrcnn_class_loss: 0.2834 - mrcnn_bbox_loss: 0.6357 - mrcnn_mask_loss: 0.5503 26/100 [======>.......................] - ETA: 48s - loss: 2.9103 - rpn_class_loss: 0.0318 - rpn_bbox_loss: 1.3994 - mrcnn_class_loss: 0.2798 - mrcnn_bbox_loss: 0.6401 - mrcnn_mask_loss: 0.5592 27/100 [=======>......................] - ETA: 47s - loss: 2.9387 - rpn_class_loss: 0.0311 - rpn_bbox_loss: 1.3674 - mrcnn_class_loss: 0.2707 - mrcnn_bbox_loss: 0.7049 - mrcnn_mask_loss: 0.5647 28/100 [=======>......................] - ETA: 46s - loss: 2.9125 - rpn_class_loss: 0.0338 - rpn_bbox_loss: 1.3246 - mrcnn_class_loss: 0.2724 - mrcnn_bbox_loss: 0.7119 - mrcnn_mask_loss: 0.5698 29/100 [=======>......................] - ETA: 45s - loss: 2.8980 - rpn_class_loss: 0.0332 - rpn_bbox_loss: 1.3070 - mrcnn_class_loss: 0.2673 - mrcnn_bbox_loss: 0.7040 - mrcnn_mask_loss: 0.5865 30/100 [========>.....................] - ETA: 44s - loss: 2.8521 - rpn_class_loss: 0.0332 - rpn_bbox_loss: 1.2662 - mrcnn_class_loss: 0.2593 - mrcnn_bbox_loss: 0.7140 - mrcnn_mask_loss: 0.5794 31/100 [========>.....................] - ETA: 44s - loss: 2.8123 - rpn_class_loss: 0.0322 - rpn_bbox_loss: 1.2309 - mrcnn_class_loss: 0.2539 - mrcnn_bbox_loss: 0.7100 - mrcnn_mask_loss: 0.5854 32/100 [========>.....................] - ETA: 43s - loss: 2.7963 - rpn_class_loss: 0.0315 - rpn_bbox_loss: 1.2160 - mrcnn_class_loss: 0.2501 - mrcnn_bbox_loss: 0.7122 - mrcnn_mask_loss: 0.5864 33/100 [========>.....................] - ETA: 42s - loss: 2.7763 - rpn_class_loss: 0.0307 - rpn_bbox_loss: 1.1974 - mrcnn_class_loss: 0.2475 - mrcnn_bbox_loss: 0.7117 - mrcnn_mask_loss: 0.5891 34/100 [=========>....................] - ETA: 41s - loss: 2.7392 - rpn_class_loss: 0.0298 - rpn_bbox_loss: 1.1679 - mrcnn_class_loss: 0.2477 - mrcnn_bbox_loss: 0.7080 - mrcnn_mask_loss: 0.5859 35/100 [=========>....................] - ETA: 40s - loss: 2.6995 - rpn_class_loss: 0.0290 - rpn_bbox_loss: 1.1398 - mrcnn_class_loss: 0.2453 - mrcnn_bbox_loss: 0.7040 - mrcnn_mask_loss: 0.5814 36/100 [=========>....................] - ETA: 39s - loss: 2.6515 - rpn_class_loss: 0.0285 - rpn_bbox_loss: 1.1112 - mrcnn_class_loss: 0.2396 - mrcnn_bbox_loss: 0.6964 - mrcnn_mask_loss: 0.5758 37/100 [==========>...................] - ETA: 38s - loss: 2.6511 - rpn_class_loss: 0.0279 - rpn_bbox_loss: 1.0910 - mrcnn_class_loss: 0.2358 - mrcnn_bbox_loss: 0.7180 - mrcnn_mask_loss: 0.5784 38/100 [==========>...................] - ETA: 38s - loss: 2.6420 - rpn_class_loss: 0.0288 - rpn_bbox_loss: 1.0662 - mrcnn_class_loss: 0.2348 - mrcnn_bbox_loss: 0.7308 - mrcnn_mask_loss: 0.5814 39/100 [==========>...................] - ETA: 37s - loss: 2.6040 - rpn_class_loss: 0.0281 - rpn_bbox_loss: 1.0419 - mrcnn_class_loss: 0.2337 - mrcnn_bbox_loss: 0.7211 - mrcnn_mask_loss: 0.5791 40/100 [===========>..................] - ETA: 36s - loss: 2.5944 - rpn_class_loss: 0.0285 - rpn_bbox_loss: 1.0267 - mrcnn_class_loss: 0.2283 - mrcnn_bbox_loss: 0.7277 - mrcnn_mask_loss: 0.5831 41/100 [===========>..................] - ETA: 35s - loss: 2.5873 - rpn_class_loss: 0.0282 - rpn_bbox_loss: 1.0149 - mrcnn_class_loss: 0.2246 - mrcnn_bbox_loss: 0.7337 - mrcnn_mask_loss: 0.5858 42/100 [===========>..................] - ETA: 35s - loss: 2.5803 - rpn_class_loss: 0.0277 - rpn_bbox_loss: 1.0036 - mrcnn_class_loss: 0.2217 - mrcnn_bbox_loss: 0.7391 - mrcnn_mask_loss: 0.5882 43/100 [===========>..................] - ETA: 34s - loss: 2.5557 - rpn_class_loss: 0.0283 - rpn_bbox_loss: 0.9826 - mrcnn_class_loss: 0.2192 - mrcnn_bbox_loss: 0.7358 - mrcnn_mask_loss: 0.5897 44/100 [============>.................] - ETA: 33s - loss: 2.5287 - rpn_class_loss: 0.0284 - rpn_bbox_loss: 0.9633 - mrcnn_class_loss: 0.2175 - mrcnn_bbox_loss: 0.7276 - mrcnn_mask_loss: 0.5919 45/100 [============>.................] - ETA: 33s - loss: 2.5293 - rpn_class_loss: 0.0278 - rpn_bbox_loss: 0.9624 - mrcnn_class_loss: 0.2154 - mrcnn_bbox_loss: 0.7320 - mrcnn_mask_loss: 0.5915 46/100 [============>.................] - ETA: 32s - loss: 2.5068 - rpn_class_loss: 0.0273 - rpn_bbox_loss: 0.9450 - mrcnn_class_loss: 0.2142 - mrcnn_bbox_loss: 0.7298 - mrcnn_mask_loss: 0.5906 47/100 [=============>................] - ETA: 31s - loss: 2.5437 - rpn_class_loss: 0.0270 - rpn_bbox_loss: 0.9406 - mrcnn_class_loss: 0.2123 - mrcnn_bbox_loss: 0.7358 - mrcnn_mask_loss: 0.6279 48/100 [=============>................] - ETA: 30s - loss: 2.5540 - rpn_class_loss: 0.0267 - rpn_bbox_loss: 0.9279 - mrcnn_class_loss: 0.2088 - mrcnn_bbox_loss: 0.7574 - mrcnn_mask_loss: 0.6333 49/100 [=============>................] - ETA: 30s - loss: 2.5458 - rpn_class_loss: 0.0266 - rpn_bbox_loss: 0.9095 - mrcnn_class_loss: 0.2067 - mrcnn_bbox_loss: 0.7616 - mrcnn_mask_loss: 0.6414 50/100 [==============>...............] - ETA: 29s - loss: 2.5201 - rpn_class_loss: 0.0265 - rpn_bbox_loss: 0.8928 - mrcnn_class_loss: 0.2037 - mrcnn_bbox_loss: 0.7615 - mrcnn_mask_loss: 0.6357 51/100 [==============>...............] - ETA: 28s - loss: 2.5022 - rpn_class_loss: 0.0260 - rpn_bbox_loss: 0.8822 - mrcnn_class_loss: 0.2015 - mrcnn_bbox_loss: 0.7587 - mrcnn_mask_loss: 0.6337 52/100 [==============>...............] - ETA: 28s - loss: 2.4968 - rpn_class_loss: 0.0275 - rpn_bbox_loss: 0.8696 - mrcnn_class_loss: 0.1997 - mrcnn_bbox_loss: 0.7650 - mrcnn_mask_loss: 0.6350 53/100 [==============>...............] - ETA: 27s - loss: 2.4723 - rpn_class_loss: 0.0271 - rpn_bbox_loss: 0.8541 - mrcnn_class_loss: 0.1970 - mrcnn_bbox_loss: 0.7623 - mrcnn_mask_loss: 0.6317 54/100 [===============>..............] - ETA: 27s - loss: 2.4641 - rpn_class_loss: 0.0266 - rpn_bbox_loss: 0.8463 - mrcnn_class_loss: 0.2004 - mrcnn_bbox_loss: 0.7622 - mrcnn_mask_loss: 0.6286 55/100 [===============>..............] - ETA: 26s - loss: 2.4351 - rpn_class_loss: 0.0262 - rpn_bbox_loss: 0.8310 - mrcnn_class_loss: 0.1980 - mrcnn_bbox_loss: 0.7547 - mrcnn_mask_loss: 0.6253 56/100 [===============>..............] - ETA: 25s - loss: 2.4102 - rpn_class_loss: 0.0260 - rpn_bbox_loss: 0.8179 - mrcnn_class_loss: 0.1966 - mrcnn_bbox_loss: 0.7481 - mrcnn_mask_loss: 0.6217 57/100 [================>.............] - ETA: 25s - loss: 2.4039 - rpn_class_loss: 0.0256 - rpn_bbox_loss: 0.8141 - mrcnn_class_loss: 0.1937 - mrcnn_bbox_loss: 0.7492 - mrcnn_mask_loss: 0.6213 58/100 [================>.............] - ETA: 24s - loss: 2.3975 - rpn_class_loss: 0.0255 - rpn_bbox_loss: 0.8030 - mrcnn_class_loss: 0.1923 - mrcnn_bbox_loss: 0.7560 - mrcnn_mask_loss: 0.6208 59/100 [================>.............] - ETA: 23s - loss: 2.3806 - rpn_class_loss: 0.0258 - rpn_bbox_loss: 0.7909 - mrcnn_class_loss: 0.1910 - mrcnn_bbox_loss: 0.7516 - mrcnn_mask_loss: 0.6213 60/100 [=================>............] - ETA: 23s - loss: 2.3913 - rpn_class_loss: 0.0266 - rpn_bbox_loss: 0.8070 - mrcnn_class_loss: 0.1888 - mrcnn_bbox_loss: 0.7484 - mrcnn_mask_loss: 0.6204 61/100 [=================>............] - ETA: 22s - loss: 2.3715 - rpn_class_loss: 0.0264 - rpn_bbox_loss: 0.7956 - mrcnn_class_loss: 0.1860 - mrcnn_bbox_loss: 0.7439 - mrcnn_mask_loss: 0.6195 62/100 [=================>............] - ETA: 21s - loss: 2.3536 - rpn_class_loss: 0.0260 - rpn_bbox_loss: 0.7846 - mrcnn_class_loss: 0.1848 - mrcnn_bbox_loss: 0.7423 - mrcnn_mask_loss: 0.6159 63/100 [=================>............] - ETA: 21s - loss: 2.3325 - rpn_class_loss: 0.0262 - rpn_bbox_loss: 0.7737 - mrcnn_class_loss: 0.1826 - mrcnn_bbox_loss: 0.7372 - mrcnn_mask_loss: 0.6128 64/100 [==================>...........] - ETA: 20s - loss: 2.3176 - rpn_class_loss: 0.0260 - rpn_bbox_loss: 0.7637 - mrcnn_class_loss: 0.1821 - mrcnn_bbox_loss: 0.7365 - mrcnn_mask_loss: 0.6094 65/100 [==================>...........] - ETA: 20s - loss: 2.3006 - rpn_class_loss: 0.0256 - rpn_bbox_loss: 0.7541 - mrcnn_class_loss: 0.1805 - mrcnn_bbox_loss: 0.7343 - mrcnn_mask_loss: 0.6061 66/100 [==================>...........] - ETA: 19s - loss: 2.2798 - rpn_class_loss: 0.0254 - rpn_bbox_loss: 0.7430 - mrcnn_class_loss: 0.1788 - mrcnn_bbox_loss: 0.7285 - mrcnn_mask_loss: 0.6041 67/100 [===================>..........] - ETA: 18s - loss: 2.2792 - rpn_class_loss: 0.0253 - rpn_bbox_loss: 0.7339 - mrcnn_class_loss: 0.1774 - mrcnn_bbox_loss: 0.7392 - mrcnn_mask_loss: 0.6033 68/100 [===================>..........] - ETA: 18s - loss: 2.2696 - rpn_class_loss: 0.0253 - rpn_bbox_loss: 0.7277 - mrcnn_class_loss: 0.1765 - mrcnn_bbox_loss: 0.7393 - mrcnn_mask_loss: 0.6009 69/100 [===================>..........] - ETA: 17s - loss: 2.2470 - rpn_class_loss: 0.0252 - rpn_bbox_loss: 0.7173 - mrcnn_class_loss: 0.1743 - mrcnn_bbox_loss: 0.7320 - mrcnn_mask_loss: 0.5982 70/100 [====================>.........] - ETA: 17s - loss: 2.2425 - rpn_class_loss: 0.0250 - rpn_bbox_loss: 0.7183 - mrcnn_class_loss: 0.1723 - mrcnn_bbox_loss: 0.7306 - mrcnn_mask_loss: 0.5963 71/100 [====================>.........] - ETA: 16s - loss: 2.2329 - rpn_class_loss: 0.0252 - rpn_bbox_loss: 0.7120 - mrcnn_class_loss: 0.1704 - mrcnn_bbox_loss: 0.7306 - mrcnn_mask_loss: 0.5948 72/100 [====================>.........] - ETA: 15s - loss: 2.2095 - rpn_class_loss: 0.0250 - rpn_bbox_loss: 0.7027 - mrcnn_class_loss: 0.1684 - mrcnn_bbox_loss: 0.7212 - mrcnn_mask_loss: 0.5923 73/100 [====================>.........] - ETA: 15s - loss: 2.1931 - rpn_class_loss: 0.0250 - rpn_bbox_loss: 0.6946 - mrcnn_class_loss: 0.1664 - mrcnn_bbox_loss: 0.7163 - mrcnn_mask_loss: 0.5908 74/100 [=====================>........] - ETA: 14s - loss: 2.1853 - rpn_class_loss: 0.0247 - rpn_bbox_loss: 0.6875 - mrcnn_class_loss: 0.1650 - mrcnn_bbox_loss: 0.7178 - mrcnn_mask_loss: 0.5904 75/100 [=====================>........] - ETA: 14s - loss: 2.1696 - rpn_class_loss: 0.0245 - rpn_bbox_loss: 0.6787 - mrcnn_class_loss: 0.1646 - mrcnn_bbox_loss: 0.7162 - mrcnn_mask_loss: 0.5855 76/100 [=====================>........] - ETA: 13s - loss: 2.1727 - rpn_class_loss: 0.0246 - rpn_bbox_loss: 0.6703 - mrcnn_class_loss: 0.1633 - mrcnn_bbox_loss: 0.7286 - mrcnn_mask_loss: 0.5860 77/100 [======================>.......] - ETA: 13s - loss: 2.1647 - rpn_class_loss: 0.0244 - rpn_bbox_loss: 0.6659 - mrcnn_class_loss: 0.1633 - mrcnn_bbox_loss: 0.7272 - mrcnn_mask_loss: 0.5841 78/100 [======================>.......] - ETA: 12s - loss: 2.1554 - rpn_class_loss: 0.0241 - rpn_bbox_loss: 0.6583 - mrcnn_class_loss: 0.1618 - mrcnn_bbox_loss: 0.7292 - mrcnn_mask_loss: 0.5821 79/100 [======================>.......] - ETA: 11s - loss: 2.1430 - rpn_class_loss: 0.0238 - rpn_bbox_loss: 0.6537 - mrcnn_class_loss: 0.1608 - mrcnn_bbox_loss: 0.7239 - mrcnn_mask_loss: 0.5807 80/100 [=======================>......] - ETA: 11s - loss: 2.1310 - rpn_class_loss: 0.0236 - rpn_bbox_loss: 0.6490 - mrcnn_class_loss: 0.1600 - mrcnn_bbox_loss: 0.7192 - mrcnn_mask_loss: 0.5793 81/100 [=======================>......] - ETA: 10s - loss: 2.1212 - rpn_class_loss: 0.0235 - rpn_bbox_loss: 0.6438 - mrcnn_class_loss: 0.1587 - mrcnn_bbox_loss: 0.7182 - mrcnn_mask_loss: 0.5771 82/100 [=======================>......] - ETA: 10s - loss: 2.1098 - rpn_class_loss: 0.0233 - rpn_bbox_loss: 0.6370 - mrcnn_class_loss: 0.1572 - mrcnn_bbox_loss: 0.7177 - mrcnn_mask_loss: 0.5746 83/100 [=======================>......] - ETA: 9s - loss: 2.1073 - rpn_class_loss: 0.0233 - rpn_bbox_loss: 0.6336 - mrcnn_class_loss: 0.1566 - mrcnn_bbox_loss: 0.7121 - mrcnn_mask_loss: 0.5816 84/100 [========================>.....] - ETA: 9s - loss: 2.0990 - rpn_class_loss: 0.0230 - rpn_bbox_loss: 0.6278 - mrcnn_class_loss: 0.1558 - mrcnn_bbox_loss: 0.7110 - mrcnn_mask_loss: 0.5813 85/100 [========================>.....] - ETA: 8s - loss: 2.0850 - rpn_class_loss: 0.0229 - rpn_bbox_loss: 0.6213 - mrcnn_class_loss: 0.1548 - mrcnn_bbox_loss: 0.7068 - mrcnn_mask_loss: 0.5791 86/100 [========================>.....] - ETA: 7s - loss: 2.0820 - rpn_class_loss: 0.0229 - rpn_bbox_loss: 0.6153 - mrcnn_class_loss: 0.1557 - mrcnn_bbox_loss: 0.7089 - mrcnn_mask_loss: 0.5792 87/100 [=========================>....] - ETA: 7s - loss: 2.0718 - rpn_class_loss: 0.0229 - rpn_bbox_loss: 0.6095 - mrcnn_class_loss: 0.1549 - mrcnn_bbox_loss: 0.7080 - mrcnn_mask_loss: 0.5765 88/100 [=========================>....] - ETA: 6s - loss: 2.0709 - rpn_class_loss: 0.0226 - rpn_bbox_loss: 0.6080 - mrcnn_class_loss: 0.1566 - mrcnn_bbox_loss: 0.7071 - mrcnn_mask_loss: 0.5765 89/100 [=========================>....] - ETA: 6s - loss: 2.0713 - rpn_class_loss: 0.0224 - rpn_bbox_loss: 0.6134 - mrcnn_class_loss: 0.1550 - mrcnn_bbox_loss: 0.7070 - mrcnn_mask_loss: 0.5735 90/100 [==========================>...] - ETA: 5s - loss: 2.0624 - rpn_class_loss: 0.0225 - rpn_bbox_loss: 0.6204 - mrcnn_class_loss: 0.1532 - mrcnn_bbox_loss: 0.6992 - mrcnn_mask_loss: 0.5672 91/100 [==========================>...] - ETA: 5s - loss: 2.0626 - rpn_class_loss: 0.0224 - rpn_bbox_loss: 0.6145 - mrcnn_class_loss: 0.1550 - mrcnn_bbox_loss: 0.7043 - mrcnn_mask_loss: 0.5664 92/100 [==========================>...] - ETA: 4s - loss: 2.0539 - rpn_class_loss: 0.0222 - rpn_bbox_loss: 0.6097 - mrcnn_class_loss: 0.1544 - mrcnn_bbox_loss: 0.7012 - mrcnn_mask_loss: 0.5663 93/100 [==========================>...] - ETA: 3s - loss: 2.0421 - rpn_class_loss: 0.0220 - rpn_bbox_loss: 0.6048 - mrcnn_class_loss: 0.1533 - mrcnn_bbox_loss: 0.6980 - mrcnn_mask_loss: 0.5640 94/100 [===========================>..] - ETA: 3s - loss: 2.0283 - rpn_class_loss: 0.0220 - rpn_bbox_loss: 0.5988 - mrcnn_class_loss: 0.1522 - mrcnn_bbox_loss: 0.6936 - mrcnn_mask_loss: 0.5616 95/100 [===========================>..] - ETA: 2s - loss: 2.0198 - rpn_class_loss: 0.0220 - rpn_bbox_loss: 0.5942 - mrcnn_class_loss: 0.1509 - mrcnn_bbox_loss: 0.6867 - mrcnn_mask_loss: 0.5659 96/100 [===========================>..] - ETA: 2s - loss: 2.0161 - rpn_class_loss: 0.0218 - rpn_bbox_loss: 0.5911 - mrcnn_class_loss: 0.1504 - mrcnn_bbox_loss: 0.6898 - mrcnn_mask_loss: 0.5630 97/100 [============================>.] - ETA: 1s - loss: 2.0079 - rpn_class_loss: 0.0219 - rpn_bbox_loss: 0.5857 - mrcnn_class_loss: 0.1496 - mrcnn_bbox_loss: 0.6871 - mrcnn_mask_loss: 0.5636 98/100 [============================>.] - ETA: 1s - loss: 2.0016 - rpn_class_loss: 0.0218 - rpn_bbox_loss: 0.5823 - mrcnn_class_loss: 0.1491 - mrcnn_bbox_loss: 0.6878 - mrcnn_mask_loss: 0.5607 99/100 [============================>.] - ETA: 0s - loss: 2.0148 - rpn_class_loss: 0.0216 - rpn_bbox_loss: 0.5795 - mrcnn_class_loss: 0.1480 - mrcnn_bbox_loss: 0.6850 - mrcnn_mask_loss: 0.5807Epoch 00000: val_loss improved from inf to 1.17010, saving model to /media/jgq/GXL/project/2018/DDIM-OD/logs/bioisland20180508T0934/mask_rcnn_bioisland_0000.h5 Traceback (most recent call last): File "/media/jgq/GXL/project/2018/DDIM-OD/train_ddim.py", line 329, in train(model) File "/media/jgq/GXL/project/2018/DDIM-OD/train_ddim.py", line 213, in train layers='heads') File "/media/jgq/GXL/project/2018/DDIM-OD/model.py", line 2252, in train use_multiprocessing=True, File "/home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/keras/legacy/interfaces.py", line 87, in wrapper return func(*args, **kwargs) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/keras/engine/training.py", line 2082, in fit_generator callbacks.on_epoch_end(epoch, epoch_logs) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/keras/callbacks.py", line 77, in on_epoch_end callback.on_epoch_end(epoch, logs) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/keras/callbacks.py", line 417, in on_epoch_end self.model.save(filepath, overwrite=True) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/keras/engine/topology.py", line 2553, in save save_model(self, filepath, overwrite, include_optimizer) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/keras/models.py", line 107, in save_model 'config': model.get_config() File "/home/jgq/anaconda3/envs/python34/lib/python3.4/site-packages/keras/engine/topology.py", line 2394, in get_config return copy.deepcopy(config) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 226, in _deepcopy_tuple y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 226, in _deepcopy_tuple y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 219, in _deepcopy_list y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 226, in _deepcopy_tuple y.append(deepcopy(a, memo)) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 300, in _reconstruct state = deepcopy(state, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 155, in deepcopy y = copier(x, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 246, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 182, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/home/jgq/anaconda3/envs/python34/lib/python3.4/copy.py", line 309, in _reconstruct y.dict.update(state) AttributeError: 'NoneType' object has no attribute 'update'

Process finished with exit code 1

xiaolisarah commented 6 years ago

save_weight_only is too large to save,i just want save_best_only,here are my code callbacks = [ keras.callbacks.TensorBoard(log_dir=self.log_dir, histogram_freq=0, write_graph=True, write_images=False),

keras.callbacks.ModelCheckpoint(self.checkpoint_path,

        #                                 verbose=0, save_weights_only=True),
        keras.callbacks.ModelCheckpoint(self.checkpoint_path,monitor='val_loss',
                                        verbose=1, save_best_only=True, mode='auto'),     
    ]
xiaolisarah commented 6 years ago

@waleedka@rymalia

bahmed11 commented 6 years ago

Did you manage fixing this issue?

TiongSun commented 6 years ago

https://rdrr.io/github/dfalbel/keras/man/callback_model_checkpoint.html you can try: keras.callbacks.ModelCheckpoint(self.checkpoint_path,monitor='val_loss', verbose=1, save_best_only=True, save_weights_only = True, mode='min'),]

"set_weight_only = True" saves model's weight, rather than the whole model. It should be set as True to reduce the size (as this is your concern). You should keep it there rather than deleting it.

"save_best_only = True" will save all epoch that improve according to the mode ('min' in the above example). It will not save only one weight, but rather, all weight that improve from last epoch.

To reduce size, i recommend you set the period. Say period = 5, so that weights are saved every 5 epochs. This can also serve as a back up in case training got interrupted.

Eg. keras.callbacks.ModelCheckpoint(self.checkpoint_path,monitor='val_loss', verbose=1, save_best_only=True, save_weights_only = True, mode='min', period=5),]

bahmed11 commented 6 years ago

I know this can fix the issue but i wanted to save the whole model rather than the weights only so i don't need to build the model every time i need to test the model. This way i won't need to remember what was the model architecture i used during the training. It seems that the lambda layer is causing the problem but couldn't find a solution yet.

xiaolisarah commented 6 years ago

@bahmed11 nope

xiaolisarah commented 6 years ago

@TiongSun thanks ,i will try this

xiaolisarah commented 6 years ago

@bahmed11 from keras.models import load_model

model.save('my_model.h5') # creates a HDF5 file 'my_model.h5' del model # deletes the existing model

returns a compiled model

identical to the previous one

model = load_model('my_model.h5')

bahmed11 commented 6 years ago

@xiaolisarah If you tried to use model.save it will return the same error again from deepcopy. The only option now is to save the weights only and save the model architecture separately and combine them together during testing. A possible solution is instead of using the loss functions as lambda layers to use them as custom functions instead and pass them to the compiler. Still didn't test that yet.

angelbaowei commented 6 years ago

@bahmed11 Have you solved it? I tried to use model.save and then get the error: TypeError: can't pickle _thread.RLock objects

bahmed11 commented 6 years ago

@angelbaowei The only solution right now is to set save_weights_only=True in the ModelCheckPoint callback. This way, you will save the weights and then when testing you have to build the model and load the weights separately. Hope that help

ashnair1 commented 5 years ago

@bahmed11 How can you save just the model? The weights are already saved at every epoch so if I can save the model as well, then during test time I could load the weights and model separately and do inference. Is this possible?

bahmed11 commented 5 years ago

@ash1995 In keras there are two functions: 1) model.save ---> this will save the model and weights together which won't work with Mask RCNN. 2) mode.save_weights ---> this will save only the weights. Yes, you can save the model architecture only by using:

json_model = model.to_json()

and you can then load the model by using:

from keras.models import model_from_json model = model_from_json(json_model)

NorthLatitudeOne commented 5 years ago

Balloon sample code, I want to modify the model.py to save the model all not only weights because I need convert mask rcnn model to tensorflow model for C++ calling keras.callbacks.ModelCheckpoint(self.checkpoint_path, verbose=0, save_weights_only=True) to keras.callbacks.ModelCheckpoint(self.checkpoint_path, verbose=0, save_weights_only=False) after running , I got the error message same as others. @bahmed11 as per your talking, model.save ---> this will save the model and weights together which won't work with Mask RCNN. The only way is save the model architecture first: json_model = model.to_json() then load model by using: from keras.models import model_from_json model = model_from_json(json_model) next save model again by using: from keras.models import load_model model.save('my_model.h5') is that make sense ?

bahmed11 commented 5 years ago

@waynezhang72 Actually I didn't try resave the whole model with the weights after loading the weights on the saved model architecture. You can try to do that.