facebookresearch / VMZ

VMZ: Model Zoo for Video Modeling
Apache License 2.0
1.04k stars 155 forks source link

segmentation fault while running the finetune UCF101 #10

Closed KamranAlipour closed 6 years ago

KamranAlipour commented 6 years ago

finetuning on hmdb51 works fine for me, but once I tried ucf the model gives me a segmentation fault at the very beginning of training process right after model loading. I've downloaded the dataset and extracted it to the correct path, then used scripts/create_ucf101_lmdb.sh to create the lmdbs, and then I ran scrips/finetune_ucf101 and I get this:

Ignoring @/caffe2/caffe2/contrib/nccl:nccl_ops as it is not a valid file. Ignoring @/caffe2/caffe2/contrib/gloo:gloo_ops as it is not a valid file. Ignoring @/caffe2/caffe2/contrib/gloo:gloo_ops_gpu as it is not a valid file. E0610 17:44:44.454202 260 init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. E0610 17:44:44.455108 260 init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. E0610 17:44:44.455152 260 init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU. INFO:train_net:Namespace(base_learning_rate=0.0001, batch_size=1, clip_length_of=8, clip_length_rgb=32, crop_size=112, cudnn_workspace_limit_mb=64, db_type='pickle', display_iter=10, do_flow_aggregation=0, epoch_size=50000, file_store_path='.', flow_data_type=0, frame_gap_of=2, gamma=0.1, get_video_id=0, gpus='0,1,2,3', input_type=0, is_checkpoint=0, model_depth=34, model_name='r2plus1d', num_channels=3, num_decode_threads=4, num_epochs=16, num_gpus=1, num_labels=101, pred_layer_name=None, pretrained_model='/mnt/homedir/trandu/video_models/kinetics/l32/r2.5d_d34_l32.pkl', profiling=0, sampling_rate_of=2, sampling_rate_rgb=1, scale_h=128, scale_w=171, step_epoch=2, test_data='/data/users/trandu/datasets/ucf101_test01', train_data='/data/users/trandu/datasets/ucf101_train01', use_cudnn=1, use_dropout=0, use_local_file=0, weight_decay=0.005) INFO:model_builder:Validated: r2plus1d with 34 layers INFO:model_builder:with input 32x112x112 INFO:train_net:Running on GPUs: [0, 1, 2, 3] INFO:train_net:Using epoch size: 50000 WARNING:root:[====DEPRECATE WARNING====]: you are creating an object from CNNModelHelper class which will be deprecated soon. Please use ModelHelper object with brew module. For more information, please refer to caffe2.ai and python/brew.py, python/brew_test.py for more information. INFO:train_net:train set has 0 examples INFO:data_parallel_model:Parallelizing model for devices: [0, 1, 2, 3] INFO:data_parallel_model:Create input and model training operators INFO:data_parallel_model:Model for GPU : 0 INFO:model_helper:outputing rgb data INFO:model_builder:creating r2plus1d, depth=34... INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 230 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 460 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 921 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:data_parallel_model:Model for GPU : 1 INFO:model_helper:outputing rgb data INFO:model_builder:creating r2plus1d, depth=34... INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 230 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 460 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 921 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:data_parallel_model:Model for GPU : 2 INFO:model_helper:outputing rgb data INFO:model_builder:creating r2plus1d, depth=34... INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 230 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 460 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 921 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:data_parallel_model:Model for GPU : 3 INFO:model_helper:outputing rgb data INFO:model_builder:creating r2plus1d, depth=34... INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 230 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 460 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 921 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:data_parallel_model:Adding gradient operators INFO:data_parallel_model:Add gradient all-reduces for SyncSGD INFO:data_parallel_model:Post-iteration operators for updating params INFO:data_parallel_model:Add initial parameter sync WARNING:data_parallel_model:------- DEPRECATED API, please use data_parallel_model.OptimizeGradientMemory() ----- WARNING:memonger:NOTE: Executing memonger to optimize gradient memory INFO:memonger:Memonger memory optimization took 0.120590925217 secs WARNING:memonger:NOTE: Executing memonger to optimize gradient memory INFO:memonger:Memonger memory optimization took 0.118865966797 secs WARNING:memonger:NOTE: Executing memonger to optimize gradient memory INFO:memonger:Memonger memory optimization took 0.114645004272 secs WARNING:memonger:NOTE: Executing memonger to optimize gradient memory INFO:memonger:Memonger memory optimization took 0.114379882812 secs INFO:train_net:----- Create test net ---- WARNING:root:[====DEPRECATE WARNING====]: you are creating an object from CNNModelHelper class which will be deprecated soon. Please use ModelHelper object with brew module. For more information, please refer to caffe2.ai and python/brew.py, python/brew_test.py for more information. INFO:train_net:test set has 3783 examples INFO:data_parallel_model:Parallelizing model for devices: [0, 1, 2, 3] INFO:data_parallel_model:Create input and model training operators WARNING:data_parallel_model: WARNING:data_parallel_model:############# WARNING ############# WARNING:data_parallel_model:Model r2plus1d_test/<caffe2.python.cnn.CNNModelHelper object at 0x7fb7426d3a90> is used for testing/validation but WARNING:data_parallel_model:has init_params=True! WARNING:data_parallel_model:This can conflict with model training. WARNING:data_parallel_model:Please ensure model = ModelHelper(init_params=False) WARNING:data_parallel_model:#################################### WARNING:data_parallel_model: INFO:data_parallel_model:Model for GPU : 0 INFO:model_helper:outputing rgb data INFO:model_builder:creating r2plus1d, depth=34... INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 230 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 460 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 921 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:data_parallel_model:Model for GPU : 1 INFO:model_helper:outputing rgb data INFO:model_builder:creating r2plus1d, depth=34... INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 230 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 460 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 921 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:data_parallel_model:Model for GPU : 2 INFO:model_helper:outputing rgb data INFO:model_builder:creating r2plus1d, depth=34... INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 230 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 460 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 921 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:data_parallel_model:Model for GPU : 3 INFO:model_helper:outputing rgb data INFO:model_builder:creating r2plus1d, depth=34... INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 144 INFO:video_model:Number of middle filters: 230 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 288 INFO:video_model:Number of middle filters: 460 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 576 INFO:video_model:Number of middle filters: 921 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:video_model:Number of middle filters: 1152 INFO:data_parallel_model:Parameter update function not defined --> only forward INFO:model_loader:copying conv1_middle_w to gpu_0/conv1_middle_w INFO:model_loader:copying conv1_middle_spatbn_relu_s to gpu_0/conv1_middle_spatbn_relu_s INFO:model_loader:copying conv1_middle_spatbn_relu_b to gpu_0/conv1_middle_spatbn_relu_b INFO:model_loader:copying conv1_w to gpu_0/conv1_w INFO:model_loader:copying conv1_spatbn_relu_s to gpu_0/conv1_spatbn_relu_s INFO:model_loader:copying conv1_spatbn_relu_b to gpu_0/conv1_spatbn_relu_b INFO:model_loader:copying comp_0_conv_1_middle_w to gpu_0/comp_0_conv_1_middle_w INFO:model_loader:copying comp_0_spatbn_1_middle_s to gpu_0/comp_0_spatbn_1_middle_s INFO:model_loader:copying comp_0_spatbn_1_middle_b to gpu_0/comp_0_spatbn_1_middle_b INFO:model_loader:copying comp_0_conv_1_w to gpu_0/comp_0_conv_1_w INFO:model_loader:copying comp_0_spatbn_1_s to gpu_0/comp_0_spatbn_1_s INFO:model_loader:copying comp_0_spatbn_1_b to gpu_0/comp_0_spatbn_1_b INFO:model_loader:copying comp_0_conv_2_middle_w to gpu_0/comp_0_conv_2_middle_w INFO:model_loader:copying comp_0_spatbn_2_middle_s to gpu_0/comp_0_spatbn_2_middle_s INFO:model_loader:copying comp_0_spatbn_2_middle_b to gpu_0/comp_0_spatbn_2_middle_b INFO:model_loader:copying comp_0_conv_2_w to gpu_0/comp_0_conv_2_w INFO:model_loader:copying comp_0_spatbn_2_s to gpu_0/comp_0_spatbn_2_s INFO:model_loader:copying comp_0_spatbn_2_b to gpu_0/comp_0_spatbn_2_b INFO:model_loader:copying comp_1_conv_1_middle_w to gpu_0/comp_1_conv_1_middle_w INFO:model_loader:copying comp_1_spatbn_1_middle_s to gpu_0/comp_1_spatbn_1_middle_s INFO:model_loader:copying comp_1_spatbn_1_middle_b to gpu_0/comp_1_spatbn_1_middle_b INFO:model_loader:copying comp_1_conv_1_w to gpu_0/comp_1_conv_1_w INFO:model_loader:copying comp_1_spatbn_1_s to gpu_0/comp_1_spatbn_1_s INFO:model_loader:copying comp_1_spatbn_1_b to gpu_0/comp_1_spatbn_1_b INFO:model_loader:copying comp_1_conv_2_middle_w to gpu_0/comp_1_conv_2_middle_w INFO:model_loader:copying comp_1_spatbn_2_middle_s to gpu_0/comp_1_spatbn_2_middle_s INFO:model_loader:copying comp_1_spatbn_2_middle_b to gpu_0/comp_1_spatbn_2_middle_b INFO:model_loader:copying comp_1_conv_2_w to gpu_0/comp_1_conv_2_w INFO:model_loader:copying comp_1_spatbn_2_s to gpu_0/comp_1_spatbn_2_s INFO:model_loader:copying comp_1_spatbn_2_b to gpu_0/comp_1_spatbn_2_b INFO:model_loader:copying comp_2_conv_1_middle_w to gpu_0/comp_2_conv_1_middle_w INFO:model_loader:copying comp_2_spatbn_1_middle_s to gpu_0/comp_2_spatbn_1_middle_s INFO:model_loader:copying comp_2_spatbn_1_middle_b to gpu_0/comp_2_spatbn_1_middle_b INFO:model_loader:copying comp_2_conv_1_w to gpu_0/comp_2_conv_1_w INFO:model_loader:copying comp_2_spatbn_1_s to gpu_0/comp_2_spatbn_1_s INFO:model_loader:copying comp_2_spatbn_1_b to gpu_0/comp_2_spatbn_1_b INFO:model_loader:copying comp_2_conv_2_middle_w to gpu_0/comp_2_conv_2_middle_w INFO:model_loader:copying comp_2_spatbn_2_middle_s to gpu_0/comp_2_spatbn_2_middle_s INFO:model_loader:copying comp_2_spatbn_2_middle_b to gpu_0/comp_2_spatbn_2_middle_b INFO:model_loader:copying comp_2_conv_2_w to gpu_0/comp_2_conv_2_w INFO:model_loader:copying comp_2_spatbn_2_s to gpu_0/comp_2_spatbn_2_s INFO:model_loader:copying comp_2_spatbn_2_b to gpu_0/comp_2_spatbn_2_b INFO:model_loader:copying comp_3_conv_1_middle_w to gpu_0/comp_3_conv_1_middle_w INFO:model_loader:copying comp_3_spatbn_1_middle_s to gpu_0/comp_3_spatbn_1_middle_s INFO:model_loader:copying comp_3_spatbn_1_middle_b to gpu_0/comp_3_spatbn_1_middle_b INFO:model_loader:copying comp_3_conv_1_w to gpu_0/comp_3_conv_1_w INFO:model_loader:copying comp_3_spatbn_1_s to gpu_0/comp_3_spatbn_1_s INFO:model_loader:copying comp_3_spatbn_1_b to gpu_0/comp_3_spatbn_1_b INFO:model_loader:copying comp_3_conv_2_middle_w to gpu_0/comp_3_conv_2_middle_w INFO:model_loader:copying comp_3_spatbn_2_middle_s to gpu_0/comp_3_spatbn_2_middle_s INFO:model_loader:copying comp_3_spatbn_2_middle_b to gpu_0/comp_3_spatbn_2_middle_b INFO:model_loader:copying comp_3_conv_2_w to gpu_0/comp_3_conv_2_w INFO:model_loader:copying comp_3_spatbn_2_s to gpu_0/comp_3_spatbn_2_s INFO:model_loader:copying comp_3_spatbn_2_b to gpu_0/comp_3_spatbn_2_b INFO:model_loader:copying shortcut_projection_3_w to gpu_0/shortcut_projection_3_w INFO:model_loader:copying shortcut_projection_3_spatbn_s to gpu_0/shortcut_projection_3_spatbn_s INFO:model_loader:copying shortcut_projection_3_spatbn_b to gpu_0/shortcut_projection_3_spatbn_b INFO:model_loader:copying comp_4_conv_1_middle_w to gpu_0/comp_4_conv_1_middle_w INFO:model_loader:copying comp_4_spatbn_1_middle_s to gpu_0/comp_4_spatbn_1_middle_s INFO:model_loader:copying comp_4_spatbn_1_middle_b to gpu_0/comp_4_spatbn_1_middle_b INFO:model_loader:copying comp_4_conv_1_w to gpu_0/comp_4_conv_1_w INFO:model_loader:copying comp_4_spatbn_1_s to gpu_0/comp_4_spatbn_1_s INFO:model_loader:copying comp_4_spatbn_1_b to gpu_0/comp_4_spatbn_1_b INFO:model_loader:copying comp_4_conv_2_middle_w to gpu_0/comp_4_conv_2_middle_w INFO:model_loader:copying comp_4_spatbn_2_middle_s to gpu_0/comp_4_spatbn_2_middle_s INFO:model_loader:copying comp_4_spatbn_2_middle_b to gpu_0/comp_4_spatbn_2_middle_b INFO:model_loader:copying comp_4_conv_2_w to gpu_0/comp_4_conv_2_w INFO:model_loader:copying comp_4_spatbn_2_s to gpu_0/comp_4_spatbn_2_s INFO:model_loader:copying comp_4_spatbn_2_b to gpu_0/comp_4_spatbn_2_b INFO:model_loader:copying comp_5_conv_1_middle_w to gpu_0/comp_5_conv_1_middle_w INFO:model_loader:copying comp_5_spatbn_1_middle_s to gpu_0/comp_5_spatbn_1_middle_s INFO:model_loader:copying comp_5_spatbn_1_middle_b to gpu_0/comp_5_spatbn_1_middle_b INFO:model_loader:copying comp_5_conv_1_w to gpu_0/comp_5_conv_1_w INFO:model_loader:copying comp_5_spatbn_1_s to gpu_0/comp_5_spatbn_1_s INFO:model_loader:copying comp_5_spatbn_1_b to gpu_0/comp_5_spatbn_1_b INFO:model_loader:copying comp_5_conv_2_middle_w to gpu_0/comp_5_conv_2_middle_w INFO:model_loader:copying comp_5_spatbn_2_middle_s to gpu_0/comp_5_spatbn_2_middle_s INFO:model_loader:copying comp_5_spatbn_2_middle_b to gpu_0/comp_5_spatbn_2_middle_b INFO:model_loader:copying comp_5_conv_2_w to gpu_0/comp_5_conv_2_w INFO:model_loader:copying comp_5_spatbn_2_s to gpu_0/comp_5_spatbn_2_s INFO:model_loader:copying comp_5_spatbn_2_b to gpu_0/comp_5_spatbn_2_b INFO:model_loader:copying comp_6_conv_1_middle_w to gpu_0/comp_6_conv_1_middle_w INFO:model_loader:copying comp_6_spatbn_1_middle_s to gpu_0/comp_6_spatbn_1_middle_s INFO:model_loader:copying comp_6_spatbn_1_middle_b to gpu_0/comp_6_spatbn_1_middle_b INFO:model_loader:copying comp_6_conv_1_w to gpu_0/comp_6_conv_1_w INFO:model_loader:copying comp_6_spatbn_1_s to gpu_0/comp_6_spatbn_1_s INFO:model_loader:copying comp_6_spatbn_1_b to gpu_0/comp_6_spatbn_1_b INFO:model_loader:copying comp_6_conv_2_middle_w to gpu_0/comp_6_conv_2_middle_w INFO:model_loader:copying comp_6_spatbn_2_middle_s to gpu_0/comp_6_spatbn_2_middle_s INFO:model_loader:copying comp_6_spatbn_2_middle_b to gpu_0/comp_6_spatbn_2_middle_b INFO:model_loader:copying comp_6_conv_2_w to gpu_0/comp_6_conv_2_w INFO:model_loader:copying comp_6_spatbn_2_s to gpu_0/comp_6_spatbn_2_s INFO:model_loader:copying comp_6_spatbn_2_b to gpu_0/comp_6_spatbn_2_b INFO:model_loader:copying comp_7_conv_1_middle_w to gpu_0/comp_7_conv_1_middle_w INFO:model_loader:copying comp_7_spatbn_1_middle_s to gpu_0/comp_7_spatbn_1_middle_s INFO:model_loader:copying comp_7_spatbn_1_middle_b to gpu_0/comp_7_spatbn_1_middle_b INFO:model_loader:copying comp_7_conv_1_w to gpu_0/comp_7_conv_1_w INFO:model_loader:copying comp_7_spatbn_1_s to gpu_0/comp_7_spatbn_1_s INFO:model_loader:copying comp_7_spatbn_1_b to gpu_0/comp_7_spatbn_1_b INFO:model_loader:copying comp_7_conv_2_middle_w to gpu_0/comp_7_conv_2_middle_w INFO:model_loader:copying comp_7_spatbn_2_middle_s to gpu_0/comp_7_spatbn_2_middle_s INFO:model_loader:copying comp_7_spatbn_2_middle_b to gpu_0/comp_7_spatbn_2_middle_b INFO:model_loader:copying comp_7_conv_2_w to gpu_0/comp_7_conv_2_w INFO:model_loader:copying comp_7_spatbn_2_s to gpu_0/comp_7_spatbn_2_s INFO:model_loader:copying comp_7_spatbn_2_b to gpu_0/comp_7_spatbn_2_b INFO:model_loader:copying shortcut_projection_7_w to gpu_0/shortcut_projection_7_w INFO:model_loader:copying shortcut_projection_7_spatbn_s to gpu_0/shortcut_projection_7_spatbn_s INFO:model_loader:copying shortcut_projection_7_spatbn_b to gpu_0/shortcut_projection_7_spatbn_b INFO:model_loader:copying comp_8_conv_1_middle_w to gpu_0/comp_8_conv_1_middle_w INFO:model_loader:copying comp_8_spatbn_1_middle_s to gpu_0/comp_8_spatbn_1_middle_s INFO:model_loader:copying comp_8_spatbn_1_middle_b to gpu_0/comp_8_spatbn_1_middle_b INFO:model_loader:copying comp_8_conv_1_w to gpu_0/comp_8_conv_1_w INFO:model_loader:copying comp_8_spatbn_1_s to gpu_0/comp_8_spatbn_1_s INFO:model_loader:copying comp_8_spatbn_1_b to gpu_0/comp_8_spatbn_1_b INFO:model_loader:copying comp_8_conv_2_middle_w to gpu_0/comp_8_conv_2_middle_w INFO:model_loader:copying comp_8_spatbn_2_middle_s to gpu_0/comp_8_spatbn_2_middle_s INFO:model_loader:copying comp_8_spatbn_2_middle_b to gpu_0/comp_8_spatbn_2_middle_b INFO:model_loader:copying comp_8_conv_2_w to gpu_0/comp_8_conv_2_w INFO:model_loader:copying comp_8_spatbn_2_s to gpu_0/comp_8_spatbn_2_s INFO:model_loader:copying comp_8_spatbn_2_b to gpu_0/comp_8_spatbn_2_b INFO:model_loader:copying comp_9_conv_1_middle_w to gpu_0/comp_9_conv_1_middle_w INFO:model_loader:copying comp_9_spatbn_1_middle_s to gpu_0/comp_9_spatbn_1_middle_s INFO:model_loader:copying comp_9_spatbn_1_middle_b to gpu_0/comp_9_spatbn_1_middle_b INFO:model_loader:copying comp_9_conv_1_w to gpu_0/comp_9_conv_1_w INFO:model_loader:copying comp_9_spatbn_1_s to gpu_0/comp_9_spatbn_1_s INFO:model_loader:copying comp_9_spatbn_1_b to gpu_0/comp_9_spatbn_1_b INFO:model_loader:copying comp_9_conv_2_middle_w to gpu_0/comp_9_conv_2_middle_w INFO:model_loader:copying comp_9_spatbn_2_middle_s to gpu_0/comp_9_spatbn_2_middle_s INFO:model_loader:copying comp_9_spatbn_2_middle_b to gpu_0/comp_9_spatbn_2_middle_b INFO:model_loader:copying comp_9_conv_2_w to gpu_0/comp_9_conv_2_w INFO:model_loader:copying comp_9_spatbn_2_s to gpu_0/comp_9_spatbn_2_s INFO:model_loader:copying comp_9_spatbn_2_b to gpu_0/comp_9_spatbn_2_b INFO:model_loader:copying comp_10_conv_1_middle_w to gpu_0/comp_10_conv_1_middle_w INFO:model_loader:copying comp_10_spatbn_1_middle_s to gpu_0/comp_10_spatbn_1_middle_s INFO:model_loader:copying comp_10_spatbn_1_middle_b to gpu_0/comp_10_spatbn_1_middle_b INFO:model_loader:copying comp_10_conv_1_w to gpu_0/comp_10_conv_1_w INFO:model_loader:copying comp_10_spatbn_1_s to gpu_0/comp_10_spatbn_1_s INFO:model_loader:copying comp_10_spatbn_1_b to gpu_0/comp_10_spatbn_1_b INFO:model_loader:copying comp_10_conv_2_middle_w to gpu_0/comp_10_conv_2_middle_w INFO:model_loader:copying comp_10_spatbn_2_middle_s to gpu_0/comp_10_spatbn_2_middle_s INFO:model_loader:copying comp_10_spatbn_2_middle_b to gpu_0/comp_10_spatbn_2_middle_b INFO:model_loader:copying comp_10_conv_2_w to gpu_0/comp_10_conv_2_w INFO:model_loader:copying comp_10_spatbn_2_s to gpu_0/comp_10_spatbn_2_s INFO:model_loader:copying comp_10_spatbn_2_b to gpu_0/comp_10_spatbn_2_b INFO:model_loader:copying comp_11_conv_1_middle_w to gpu_0/comp_11_conv_1_middle_w INFO:model_loader:copying comp_11_spatbn_1_middle_s to gpu_0/comp_11_spatbn_1_middle_s INFO:model_loader:copying comp_11_spatbn_1_middle_b to gpu_0/comp_11_spatbn_1_middle_b INFO:model_loader:copying comp_11_conv_1_w to gpu_0/comp_11_conv_1_w INFO:model_loader:copying comp_11_spatbn_1_s to gpu_0/comp_11_spatbn_1_s INFO:model_loader:copying comp_11_spatbn_1_b to gpu_0/comp_11_spatbn_1_b INFO:model_loader:copying comp_11_conv_2_middle_w to gpu_0/comp_11_conv_2_middle_w INFO:model_loader:copying comp_11_spatbn_2_middle_s to gpu_0/comp_11_spatbn_2_middle_s INFO:model_loader:copying comp_11_spatbn_2_middle_b to gpu_0/comp_11_spatbn_2_middle_b INFO:model_loader:copying comp_11_conv_2_w to gpu_0/comp_11_conv_2_w INFO:model_loader:copying comp_11_spatbn_2_s to gpu_0/comp_11_spatbn_2_s INFO:model_loader:copying comp_11_spatbn_2_b to gpu_0/comp_11_spatbn_2_b INFO:model_loader:copying comp_12_conv_1_middle_w to gpu_0/comp_12_conv_1_middle_w INFO:model_loader:copying comp_12_spatbn_1_middle_s to gpu_0/comp_12_spatbn_1_middle_s INFO:model_loader:copying comp_12_spatbn_1_middle_b to gpu_0/comp_12_spatbn_1_middle_b INFO:model_loader:copying comp_12_conv_1_w to gpu_0/comp_12_conv_1_w INFO:model_loader:copying comp_12_spatbn_1_s to gpu_0/comp_12_spatbn_1_s INFO:model_loader:copying comp_12_spatbn_1_b to gpu_0/comp_12_spatbn_1_b INFO:model_loader:copying comp_12_conv_2_middle_w to gpu_0/comp_12_conv_2_middle_w INFO:model_loader:copying comp_12_spatbn_2_middle_s to gpu_0/comp_12_spatbn_2_middle_s INFO:model_loader:copying comp_12_spatbn_2_middle_b to gpu_0/comp_12_spatbn_2_middle_b INFO:model_loader:copying comp_12_conv_2_w to gpu_0/comp_12_conv_2_w INFO:model_loader:copying comp_12_spatbn_2_s to gpu_0/comp_12_spatbn_2_s INFO:model_loader:copying comp_12_spatbn_2_b to gpu_0/comp_12_spatbn_2_b INFO:model_loader:copying comp_13_conv_1_middle_w to gpu_0/comp_13_conv_1_middle_w INFO:model_loader:copying comp_13_spatbn_1_middle_s to gpu_0/comp_13_spatbn_1_middle_s INFO:model_loader:copying comp_13_spatbn_1_middle_b to gpu_0/comp_13_spatbn_1_middle_b INFO:model_loader:copying comp_13_conv_1_w to gpu_0/comp_13_conv_1_w INFO:model_loader:copying comp_13_spatbn_1_s to gpu_0/comp_13_spatbn_1_s INFO:model_loader:copying comp_13_spatbn_1_b to gpu_0/comp_13_spatbn_1_b INFO:model_loader:copying comp_13_conv_2_middle_w to gpu_0/comp_13_conv_2_middle_w INFO:model_loader:copying comp_13_spatbn_2_middle_s to gpu_0/comp_13_spatbn_2_middle_s INFO:model_loader:copying comp_13_spatbn_2_middle_b to gpu_0/comp_13_spatbn_2_middle_b INFO:model_loader:copying comp_13_conv_2_w to gpu_0/comp_13_conv_2_w INFO:model_loader:copying comp_13_spatbn_2_s to gpu_0/comp_13_spatbn_2_s INFO:model_loader:copying comp_13_spatbn_2_b to gpu_0/comp_13_spatbn_2_b INFO:model_loader:copying shortcut_projection_13_w to gpu_0/shortcut_projection_13_w INFO:model_loader:copying shortcut_projection_13_spatbn_s to gpu_0/shortcut_projection_13_spatbn_s INFO:model_loader:copying shortcut_projection_13_spatbn_b to gpu_0/shortcut_projection_13_spatbn_b INFO:model_loader:copying comp_14_conv_1_middle_w to gpu_0/comp_14_conv_1_middle_w INFO:model_loader:copying comp_14_spatbn_1_middle_s to gpu_0/comp_14_spatbn_1_middle_s INFO:model_loader:copying comp_14_spatbn_1_middle_b to gpu_0/comp_14_spatbn_1_middle_b INFO:model_loader:copying comp_14_conv_1_w to gpu_0/comp_14_conv_1_w INFO:model_loader:copying comp_14_spatbn_1_s to gpu_0/comp_14_spatbn_1_s INFO:model_loader:copying comp_14_spatbn_1_b to gpu_0/comp_14_spatbn_1_b INFO:model_loader:copying comp_14_conv_2_middle_w to gpu_0/comp_14_conv_2_middle_w INFO:model_loader:copying comp_14_spatbn_2_middle_s to gpu_0/comp_14_spatbn_2_middle_s INFO:model_loader:copying comp_14_spatbn_2_middle_b to gpu_0/comp_14_spatbn_2_middle_b INFO:model_loader:copying comp_14_conv_2_w to gpu_0/comp_14_conv_2_w INFO:model_loader:copying comp_14_spatbn_2_s to gpu_0/comp_14_spatbn_2_s INFO:model_loader:copying comp_14_spatbn_2_b to gpu_0/comp_14_spatbn_2_b INFO:model_loader:copying comp_15_conv_1_middle_w to gpu_0/comp_15_conv_1_middle_w INFO:model_loader:copying comp_15_spatbn_1_middle_s to gpu_0/comp_15_spatbn_1_middle_s INFO:model_loader:copying comp_15_spatbn_1_middle_b to gpu_0/comp_15_spatbn_1_middle_b INFO:model_loader:copying comp_15_conv_1_w to gpu_0/comp_15_conv_1_w INFO:model_loader:copying comp_15_spatbn_1_s to gpu_0/comp_15_spatbn_1_s INFO:model_loader:copying comp_15_spatbn_1_b to gpu_0/comp_15_spatbn_1_b INFO:model_loader:copying comp_15_conv_2_middle_w to gpu_0/comp_15_conv_2_middle_w INFO:model_loader:copying comp_15_spatbn_2_middle_s to gpu_0/comp_15_spatbn_2_middle_s INFO:model_loader:copying comp_15_spatbn_2_middle_b to gpu_0/comp_15_spatbn_2_middle_b INFO:model_loader:copying comp_15_conv_2_w to gpu_0/comp_15_conv_2_w INFO:model_loader:copying comp_15_spatbn_2_s to gpu_0/comp_15_spatbn_2_s INFO:model_loader:copying comp_15_spatbn_2_b to gpu_0/comp_15_spatbn_2_b INFO:model_loader:last_out_L101_w not found INFO:model_loader:last_out_L101_b not found INFO:model_loader:copying conv1_middle_spatbn_relu_rm to gpu_0/conv1_middle_spatbn_relu_rm INFO:model_loader:copying conv1_middle_spatbn_relu_riv to gpu_0/conv1_middle_spatbn_relu_riv INFO:model_loader:copying conv1_spatbn_relu_rm to gpu_0/conv1_spatbn_relu_rm INFO:model_loader:copying conv1_spatbn_relu_riv to gpu_0/conv1_spatbn_relu_riv INFO:model_loader:copying comp_0_spatbn_1_middle_rm to gpu_0/comp_0_spatbn_1_middle_rm INFO:model_loader:copying comp_0_spatbn_1_middle_riv to gpu_0/comp_0_spatbn_1_middle_riv INFO:model_loader:copying comp_0_spatbn_1_rm to gpu_0/comp_0_spatbn_1_rm INFO:model_loader:copying comp_0_spatbn_1_riv to gpu_0/comp_0_spatbn_1_riv INFO:model_loader:copying comp_0_spatbn_2_middle_rm to gpu_0/comp_0_spatbn_2_middle_rm INFO:model_loader:copying comp_0_spatbn_2_middle_riv to gpu_0/comp_0_spatbn_2_middle_riv INFO:model_loader:copying comp_0_spatbn_2_rm to gpu_0/comp_0_spatbn_2_rm INFO:model_loader:copying comp_0_spatbn_2_riv to gpu_0/comp_0_spatbn_2_riv INFO:model_loader:copying comp_1_spatbn_1_middle_rm to gpu_0/comp_1_spatbn_1_middle_rm INFO:model_loader:copying comp_1_spatbn_1_middle_riv to gpu_0/comp_1_spatbn_1_middle_riv INFO:model_loader:copying comp_1_spatbn_1_rm to gpu_0/comp_1_spatbn_1_rm INFO:model_loader:copying comp_1_spatbn_1_riv to gpu_0/comp_1_spatbn_1_riv INFO:model_loader:copying comp_1_spatbn_2_middle_rm to gpu_0/comp_1_spatbn_2_middle_rm INFO:model_loader:copying comp_1_spatbn_2_middle_riv to gpu_0/comp_1_spatbn_2_middle_riv INFO:model_loader:copying comp_1_spatbn_2_rm to gpu_0/comp_1_spatbn_2_rm INFO:model_loader:copying comp_1_spatbn_2_riv to gpu_0/comp_1_spatbn_2_riv INFO:model_loader:copying comp_2_spatbn_1_middle_rm to gpu_0/comp_2_spatbn_1_middle_rm INFO:model_loader:copying comp_2_spatbn_1_middle_riv to gpu_0/comp_2_spatbn_1_middle_riv INFO:model_loader:copying comp_2_spatbn_1_rm to gpu_0/comp_2_spatbn_1_rm INFO:model_loader:copying comp_2_spatbn_1_riv to gpu_0/comp_2_spatbn_1_riv INFO:model_loader:copying comp_2_spatbn_2_middle_rm to gpu_0/comp_2_spatbn_2_middle_rm INFO:model_loader:copying comp_2_spatbn_2_middle_riv to gpu_0/comp_2_spatbn_2_middle_riv INFO:model_loader:copying comp_2_spatbn_2_rm to gpu_0/comp_2_spatbn_2_rm INFO:model_loader:copying comp_2_spatbn_2_riv to gpu_0/comp_2_spatbn_2_riv INFO:model_loader:copying comp_3_spatbn_1_middle_rm to gpu_0/comp_3_spatbn_1_middle_rm INFO:model_loader:copying comp_3_spatbn_1_middle_riv to gpu_0/comp_3_spatbn_1_middle_riv INFO:model_loader:copying comp_3_spatbn_1_rm to gpu_0/comp_3_spatbn_1_rm INFO:model_loader:copying comp_3_spatbn_1_riv to gpu_0/comp_3_spatbn_1_riv INFO:model_loader:copying comp_3_spatbn_2_middle_rm to gpu_0/comp_3_spatbn_2_middle_rm INFO:model_loader:copying comp_3_spatbn_2_middle_riv to gpu_0/comp_3_spatbn_2_middle_riv INFO:model_loader:copying comp_3_spatbn_2_rm to gpu_0/comp_3_spatbn_2_rm INFO:model_loader:copying comp_3_spatbn_2_riv to gpu_0/comp_3_spatbn_2_riv INFO:model_loader:copying shortcut_projection_3_spatbn_rm to gpu_0/shortcut_projection_3_spatbn_rm INFO:model_loader:copying shortcut_projection_3_spatbn_riv to gpu_0/shortcut_projection_3_spatbn_riv INFO:model_loader:copying comp_4_spatbn_1_middle_rm to gpu_0/comp_4_spatbn_1_middle_rm INFO:model_loader:copying comp_4_spatbn_1_middle_riv to gpu_0/comp_4_spatbn_1_middle_riv INFO:model_loader:copying comp_4_spatbn_1_rm to gpu_0/comp_4_spatbn_1_rm INFO:model_loader:copying comp_4_spatbn_1_riv to gpu_0/comp_4_spatbn_1_riv INFO:model_loader:copying comp_4_spatbn_2_middle_rm to gpu_0/comp_4_spatbn_2_middle_rm INFO:model_loader:copying comp_4_spatbn_2_middle_riv to gpu_0/comp_4_spatbn_2_middle_riv INFO:model_loader:copying comp_4_spatbn_2_rm to gpu_0/comp_4_spatbn_2_rm INFO:model_loader:copying comp_4_spatbn_2_riv to gpu_0/comp_4_spatbn_2_riv INFO:model_loader:copying comp_5_spatbn_1_middle_rm to gpu_0/comp_5_spatbn_1_middle_rm INFO:model_loader:copying comp_5_spatbn_1_middle_riv to gpu_0/comp_5_spatbn_1_middle_riv INFO:model_loader:copying comp_5_spatbn_1_rm to gpu_0/comp_5_spatbn_1_rm INFO:model_loader:copying comp_5_spatbn_1_riv to gpu_0/comp_5_spatbn_1_riv INFO:model_loader:copying comp_5_spatbn_2_middle_rm to gpu_0/comp_5_spatbn_2_middle_rm INFO:model_loader:copying comp_5_spatbn_2_middle_riv to gpu_0/comp_5_spatbn_2_middle_riv INFO:model_loader:copying comp_5_spatbn_2_rm to gpu_0/comp_5_spatbn_2_rm INFO:model_loader:copying comp_5_spatbn_2_riv to gpu_0/comp_5_spatbn_2_riv INFO:model_loader:copying comp_6_spatbn_1_middle_rm to gpu_0/comp_6_spatbn_1_middle_rm INFO:model_loader:copying comp_6_spatbn_1_middle_riv to gpu_0/comp_6_spatbn_1_middle_riv INFO:model_loader:copying comp_6_spatbn_1_rm to gpu_0/comp_6_spatbn_1_rm INFO:model_loader:copying comp_6_spatbn_1_riv to gpu_0/comp_6_spatbn_1_riv INFO:model_loader:copying comp_6_spatbn_2_middle_rm to gpu_0/comp_6_spatbn_2_middle_rm INFO:model_loader:copying comp_6_spatbn_2_middle_riv to gpu_0/comp_6_spatbn_2_middle_riv INFO:model_loader:copying comp_6_spatbn_2_rm to gpu_0/comp_6_spatbn_2_rm INFO:model_loader:copying comp_6_spatbn_2_riv to gpu_0/comp_6_spatbn_2_riv INFO:model_loader:copying comp_7_spatbn_1_middle_rm to gpu_0/comp_7_spatbn_1_middle_rm INFO:model_loader:copying comp_7_spatbn_1_middle_riv to gpu_0/comp_7_spatbn_1_middle_riv INFO:model_loader:copying comp_7_spatbn_1_rm to gpu_0/comp_7_spatbn_1_rm INFO:model_loader:copying comp_7_spatbn_1_riv to gpu_0/comp_7_spatbn_1_riv INFO:model_loader:copying comp_7_spatbn_2_middle_rm to gpu_0/comp_7_spatbn_2_middle_rm INFO:model_loader:copying comp_7_spatbn_2_middle_riv to gpu_0/comp_7_spatbn_2_middle_riv INFO:model_loader:copying comp_7_spatbn_2_rm to gpu_0/comp_7_spatbn_2_rm INFO:model_loader:copying comp_7_spatbn_2_riv to gpu_0/comp_7_spatbn_2_riv INFO:model_loader:copying shortcut_projection_7_spatbn_rm to gpu_0/shortcut_projection_7_spatbn_rm INFO:model_loader:copying shortcut_projection_7_spatbn_riv to gpu_0/shortcut_projection_7_spatbn_riv INFO:model_loader:copying comp_8_spatbn_1_middle_rm to gpu_0/comp_8_spatbn_1_middle_rm INFO:model_loader:copying comp_8_spatbn_1_middle_riv to gpu_0/comp_8_spatbn_1_middle_riv INFO:model_loader:copying comp_8_spatbn_1_rm to gpu_0/comp_8_spatbn_1_rm INFO:model_loader:copying comp_8_spatbn_1_riv to gpu_0/comp_8_spatbn_1_riv INFO:model_loader:copying comp_8_spatbn_2_middle_rm to gpu_0/comp_8_spatbn_2_middle_rm INFO:model_loader:copying comp_8_spatbn_2_middle_riv to gpu_0/comp_8_spatbn_2_middle_riv INFO:model_loader:copying comp_8_spatbn_2_rm to gpu_0/comp_8_spatbn_2_rm INFO:model_loader:copying comp_8_spatbn_2_riv to gpu_0/comp_8_spatbn_2_riv INFO:model_loader:copying comp_9_spatbn_1_middle_rm to gpu_0/comp_9_spatbn_1_middle_rm INFO:model_loader:copying comp_9_spatbn_1_middle_riv to gpu_0/comp_9_spatbn_1_middle_riv INFO:model_loader:copying comp_9_spatbn_1_rm to gpu_0/comp_9_spatbn_1_rm INFO:model_loader:copying comp_9_spatbn_1_riv to gpu_0/comp_9_spatbn_1_riv INFO:model_loader:copying comp_9_spatbn_2_middle_rm to gpu_0/comp_9_spatbn_2_middle_rm INFO:model_loader:copying comp_9_spatbn_2_middle_riv to gpu_0/comp_9_spatbn_2_middle_riv INFO:model_loader:copying comp_9_spatbn_2_rm to gpu_0/comp_9_spatbn_2_rm INFO:model_loader:copying comp_9_spatbn_2_riv to gpu_0/comp_9_spatbn_2_riv INFO:model_loader:copying comp_10_spatbn_1_middle_rm to gpu_0/comp_10_spatbn_1_middle_rm INFO:model_loader:copying comp_10_spatbn_1_middle_riv to gpu_0/comp_10_spatbn_1_middle_riv INFO:model_loader:copying comp_10_spatbn_1_rm to gpu_0/comp_10_spatbn_1_rm INFO:model_loader:copying comp_10_spatbn_1_riv to gpu_0/comp_10_spatbn_1_riv INFO:model_loader:copying comp_10_spatbn_2_middle_rm to gpu_0/comp_10_spatbn_2_middle_rm INFO:model_loader:copying comp_10_spatbn_2_middle_riv to gpu_0/comp_10_spatbn_2_middle_riv INFO:model_loader:copying comp_10_spatbn_2_rm to gpu_0/comp_10_spatbn_2_rm INFO:model_loader:copying comp_10_spatbn_2_riv to gpu_0/comp_10_spatbn_2_riv INFO:model_loader:copying comp_11_spatbn_1_middle_rm to gpu_0/comp_11_spatbn_1_middle_rm INFO:model_loader:copying comp_11_spatbn_1_middle_riv to gpu_0/comp_11_spatbn_1_middle_riv INFO:model_loader:copying comp_11_spatbn_1_rm to gpu_0/comp_11_spatbn_1_rm INFO:model_loader:copying comp_11_spatbn_1_riv to gpu_0/comp_11_spatbn_1_riv INFO:model_loader:copying comp_11_spatbn_2_middle_rm to gpu_0/comp_11_spatbn_2_middle_rm INFO:model_loader:copying comp_11_spatbn_2_middle_riv to gpu_0/comp_11_spatbn_2_middle_riv INFO:model_loader:copying comp_11_spatbn_2_rm to gpu_0/comp_11_spatbn_2_rm INFO:model_loader:copying comp_11_spatbn_2_riv to gpu_0/comp_11_spatbn_2_riv INFO:model_loader:copying comp_12_spatbn_1_middle_rm to gpu_0/comp_12_spatbn_1_middle_rm INFO:model_loader:copying comp_12_spatbn_1_middle_riv to gpu_0/comp_12_spatbn_1_middle_riv INFO:model_loader:copying comp_12_spatbn_1_rm to gpu_0/comp_12_spatbn_1_rm INFO:model_loader:copying comp_12_spatbn_1_riv to gpu_0/comp_12_spatbn_1_riv INFO:model_loader:copying comp_12_spatbn_2_middle_rm to gpu_0/comp_12_spatbn_2_middle_rm INFO:model_loader:copying comp_12_spatbn_2_middle_riv to gpu_0/comp_12_spatbn_2_middle_riv INFO:model_loader:copying comp_12_spatbn_2_rm to gpu_0/comp_12_spatbn_2_rm INFO:model_loader:copying comp_12_spatbn_2_riv to gpu_0/comp_12_spatbn_2_riv INFO:model_loader:copying comp_13_spatbn_1_middle_rm to gpu_0/comp_13_spatbn_1_middle_rm INFO:model_loader:copying comp_13_spatbn_1_middle_riv to gpu_0/comp_13_spatbn_1_middle_riv INFO:model_loader:copying comp_13_spatbn_1_rm to gpu_0/comp_13_spatbn_1_rm INFO:model_loader:copying comp_13_spatbn_1_riv to gpu_0/comp_13_spatbn_1_riv INFO:model_loader:copying comp_13_spatbn_2_middle_rm to gpu_0/comp_13_spatbn_2_middle_rm INFO:model_loader:copying comp_13_spatbn_2_middle_riv to gpu_0/comp_13_spatbn_2_middle_riv INFO:model_loader:copying comp_13_spatbn_2_rm to gpu_0/comp_13_spatbn_2_rm INFO:model_loader:copying comp_13_spatbn_2_riv to gpu_0/comp_13_spatbn_2_riv INFO:model_loader:copying shortcut_projection_13_spatbn_rm to gpu_0/shortcut_projection_13_spatbn_rm INFO:model_loader:copying shortcut_projection_13_spatbn_riv to gpu_0/shortcut_projection_13_spatbn_riv INFO:model_loader:copying comp_14_spatbn_1_middle_rm to gpu_0/comp_14_spatbn_1_middle_rm INFO:model_loader:copying comp_14_spatbn_1_middle_riv to gpu_0/comp_14_spatbn_1_middle_riv INFO:model_loader:copying comp_14_spatbn_1_rm to gpu_0/comp_14_spatbn_1_rm INFO:model_loader:copying comp_14_spatbn_1_riv to gpu_0/comp_14_spatbn_1_riv INFO:model_loader:copying comp_14_spatbn_2_middle_rm to gpu_0/comp_14_spatbn_2_middle_rm INFO:model_loader:copying comp_14_spatbn_2_middle_riv to gpu_0/comp_14_spatbn_2_middle_riv INFO:model_loader:copying comp_14_spatbn_2_rm to gpu_0/comp_14_spatbn_2_rm INFO:model_loader:copying comp_14_spatbn_2_riv to gpu_0/comp_14_spatbn_2_riv INFO:model_loader:copying comp_15_spatbn_1_middle_rm to gpu_0/comp_15_spatbn_1_middle_rm INFO:model_loader:copying comp_15_spatbn_1_middle_riv to gpu_0/comp_15_spatbn_1_middle_riv INFO:model_loader:copying comp_15_spatbn_1_rm to gpu_0/comp_15_spatbn_1_rm INFO:model_loader:copying comp_15_spatbn_1_riv to gpu_0/comp_15_spatbn_1_riv INFO:model_loader:copying comp_15_spatbn_2_middle_rm to gpu_0/comp_15_spatbn_2_middle_rm INFO:model_loader:copying comp_15_spatbn_2_middle_riv to gpu_0/comp_15_spatbn_2_middle_riv INFO:model_loader:copying comp_15_spatbn_2_rm to gpu_0/comp_15_spatbn_2_rm INFO:model_loader:copying comp_15_spatbn_2_riv to gpu_0/comp_15_spatbn_2_riv INFO:data_parallel_model:Creating checkpoint synchronization net INFO:data_parallel_model:Run checkpoint net INFO:train_net:Starting epoch 0/8 Aborted at 1528652698 (unix time) try "date -d @1528652698" if you are using GNU date PC: @ 0x7fb7def1add6 memcpy_ssse3_back SIGSEGV (@0x7fb7f1fcd4d0) received by PID 260 (TID 0x7db6517f2700) from PID 18446744073474462928; stack trace: @ 0x7fb7df8aa6d0 (unknown) @ 0x7fb7def1add6 memcpy_ssse3_back @ 0x7fb773a3638e (unknown) @ 0x7fb773a363fd (unknown) @ 0x7fb7d97b725a caffe2::db::LMDBCursor::key() @ 0x7fb7c1a76898 caffe2::VideoInputOp<>::Prefetch() @ 0x7fb7c1a666e6 caffe2::PrefetchOperator<>::PrefetchWorker() @ 0x7fb773a2c070 (unknown) @ 0x7fb7df8a2e25 start_thread @ 0x7fb7deec3bad __clone Segmentation fault (core dumped)

KamranAlipour commented 6 years ago

the only suspicious thing I noticed was in the loading section where I have: INFO:model_loader:last_out_L101_w not found INFO:model_loader:last_out_L101_b not found

dutran commented 6 years ago

This part is normal, as you have different layer name (to predict 101 classes instead of 400 as Kinetics). The issue seems to be around LMBD and data reader. When you created lmdb, which option did you use? e.g. use_local_file = 1 or 0. This is needed to adjust the same for the finetuning option. As the VideoInputOp will read the key,value pair from LMDB, then if use local file is true, it use value as path to read video, if not, it will use value as binary data from video and decode frames from that. It seems you have this option mismatch.

dutran commented 6 years ago

@KamranAlipour have you got your issue solved?

KamranAlipour commented 6 years ago

@dutran I did not try your solution as I ended up not using finetuning on UCF101