search

chuckcho / video-caffe

Video-friendly caffe -- comes with the most recent version of Caffe (as of Jan 2019), a video reader, 3D(ND) pooling layer, and an example training script for C3D network and UCF-101 data
Other
175 stars 93 forks source link

VGG network doesn't converge #67

Closed livst closed 6 years ago

livst commented 7 years ago

Hi,

I tried to use the 3D conv method with the VGG network, but the network doesn't converge, the loss doesn't go down. Since the prototxt is very large, i've posted just the first layers, all of the other layers are set just as these ones.


  name: "conv1_1"
  type: "NdConvolution"
  bottom: "data"
  top: "conv1_1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 64
    pad_shape {
      dim: 1
      dim: 1
      dim: 1
    }
    kernel_shape {
      dim: 3
      dim: 3
      dim: 3
    }
  }
}
layer {
  name: "relu1_1"
  type: "ReLU"
  bottom: "conv1_1"
  top: "conv1_1"
}
layer {
  name: "conv1_2"
  type: "NdConvolution"
  bottom: "conv1_1"
  top: "conv1_2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 64
    pad_shape {
      dim: 1
      dim: 1
      dim: 1
    }
    kernel_shape {
      dim: 3
      dim: 3
      dim: 3
    }
  }
}
layer {
  name: "relu1_2"
  type: "ReLU"
  bottom: "conv1_2"
  top: "conv1_2"
}
layer {
  name: "pool1"
  type: "NdPooling"
  bottom: "conv1_2"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_shape {
      dim: 1
      dim: 2
      dim: 2
    }
    stride_shape {
      dim: 1
      dim: 2
      dim: 2
    }
  }
}
layer {
  name: "conv2_1"
  type: "NdConvolution"
  bottom: "pool1"
  top: "conv2_1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 128
    pad_shape {
      dim: 1
      dim: 1
      dim: 1
    }
    kernel_shape {
      dim: 3
      dim: 3
      dim: 3
    }
  }
}
layer {
  name: "relu2_1"
  type: "ReLU"
  bottom: "conv2_1"
  top: "conv2_1"
}
layer {
  name: "conv2_2"
  type: "NdConvolution"
  bottom: "conv2_1"
  top: "conv2_2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 128
    pad_shape {
      dim: 1
      dim: 1
      dim: 1
    }
    kernel_shape {
      dim: 3
      dim: 3
      dim: 3
    }
  }
}
layer {
  name: "relu2_2"
  type: "ReLU"
  bottom: "conv2_2"
  top: "conv2_2"
}
layer {
  name: "pool2"
  type: "NdPooling"
  bottom: "conv2_2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_shape {
      dim: 2
      dim: 2
      dim: 2
    }
    stride_shape {
      dim: 2
      dim: 2
      dim: 2
    }
  }
}
...
...
...
I1127 16:25:17.269923 16840 layer_factory.hpp:77] Creating layer data
I1127 16:25:17.269954 16840 net.cpp:100] Creating Layer data
I1127 16:25:17.269963 16840 net.cpp:408] data -> data
I1127 16:25:17.269981 16840 net.cpp:408] data -> label
I1127 16:25:17.270308 16840 video_data_layer.cpp:39] Opening file /ucf101_train_split1.txt
I1127 16:25:17.369125 16840 video_data_layer.cpp:53] Shuffling data
I1127 16:25:17.421939 16840 video_data_layer.cpp:58] A total of 174973 video chunks.
I1127 16:25:17.471613 16840 video_data_layer.cpp:98] output data size: 15,3,10,112,112
I1127 16:25:17.502660 16840 net.cpp:150] Setting up data
I1127 16:25:17.502686 16840 net.cpp:157] Top shape: 15 3 10 112 112 (5644800)
I1127 16:25:17.502691 16840 net.cpp:157] Top shape: 15 (15)
I1127 16:25:17.502693 16840 net.cpp:165] Memory required for data: 22579260
I1127 16:25:17.502701 16840 layer_factory.hpp:77] Creating layer conv1_1
I1127 16:25:17.502725 16840 net.cpp:100] Creating Layer conv1_1
I1127 16:25:17.502732 16840 net.cpp:434] conv1_1 <- data
I1127 16:25:17.502744 16840 net.cpp:408] conv1_1 -> conv1_1
I1127 16:25:17.650192 16840 net.cpp:150] Setting up conv1_1
I1127 16:25:17.650223 16840 net.cpp:157] Top shape: 15 64 10 112 112 (120422400)
I1127 16:25:17.650229 16840 net.cpp:165] Memory required for data: 504268860
I1127 16:25:17.650256 16840 layer_factory.hpp:77] Creating layer relu1_1
I1127 16:25:17.650274 16840 net.cpp:100] Creating Layer relu1_1
I1127 16:25:17.650284 16840 net.cpp:434] relu1_1 <- conv1_1
I1127 16:25:17.650290 16840 net.cpp:395] relu1_1 -> conv1_1 (in-place)
I1127 16:25:17.650478 16840 net.cpp:150] Setting up relu1_1
I1127 16:25:17.650490 16840 net.cpp:157] Top shape: 15 64 10 112 112 (120422400)
I1127 16:25:17.650493 16840 net.cpp:165] Memory required for data: 985958460
I1127 16:25:17.650501 16840 layer_factory.hpp:77] Creating layer conv1_2
I1127 16:25:17.650516 16840 net.cpp:100] Creating Layer conv1_2
I1127 16:25:17.650521 16840 net.cpp:434] conv1_2 <- conv1_1
I1127 16:25:17.650532 16840 net.cpp:408] conv1_2 -> conv1_2
I1127 16:25:17.651731 16840 net.cpp:150] Setting up conv1_2
I1127 16:25:17.651746 16840 net.cpp:157] Top shape: 15 64 10 112 112 (120422400)
I1127 16:25:17.651751 16840 net.cpp:165] Memory required for data: 1467648060
I1127 16:25:17.651762 16840 layer_factory.hpp:77] Creating layer relu1_2
I1127 16:25:17.651770 16840 net.cpp:100] Creating Layer relu1_2
I1127 16:25:17.651777 16840 net.cpp:434] relu1_2 <- conv1_2
I1127 16:25:17.651782 16840 net.cpp:395] relu1_2 -> conv1_2 (in-place)
I1127 16:25:17.652070 16840 net.cpp:150] Setting up relu1_2
I1127 16:25:17.652083 16840 net.cpp:157] Top shape: 15 64 10 112 112 (120422400)
I1127 16:25:17.652113 16840 net.cpp:165] Memory required for data: 1949337660
I1127 16:25:17.652119 16840 layer_factory.hpp:77] Creating layer pool1
I1127 16:25:17.652130 16840 net.cpp:100] Creating Layer pool1
I1127 16:25:17.652134 16840 net.cpp:434] pool1 <- conv1_2
I1127 16:25:17.652139 16840 net.cpp:408] pool1 -> pool1
I1127 16:25:17.652343 16840 net.cpp:150] Setting up pool1
I1127 16:25:17.652356 16840 net.cpp:157] Top shape: 15 64 10 56 56 (30105600)
I1127 16:25:17.652360 16840 net.cpp:165] Memory required for data: 2069760060
I1127 16:25:17.652367 16840 layer_factory.hpp:77] Creating layer conv2_1
I1127 16:25:17.652381 16840 net.cpp:100] Creating Layer conv2_1
I1127 16:25:17.652386 16840 net.cpp:434] conv2_1 <- pool1
I1127 16:25:17.652395 16840 net.cpp:408] conv2_1 -> conv2_1
I1127 16:25:17.653584 16840 net.cpp:150] Setting up conv2_1
I1127 16:25:17.653596 16840 net.cpp:157] Top shape: 15 128 10 56 56 (60211200)
I1127 16:25:17.653601 16840 net.cpp:165] Memory required for data: 2310604860
I1127 16:25:17.653614 16840 layer_factory.hpp:77] Creating layer relu2_1
I1127 16:25:17.653620 16840 net.cpp:100] Creating Layer relu2_1
I1127 16:25:17.653625 16840 net.cpp:434] relu2_1 <- conv2_1
I1127 16:25:17.653631 16840 net.cpp:395] relu2_1 -> conv2_1 (in-place)
I1127 16:25:17.653916 16840 net.cpp:150] Setting up relu2_1
I1127 16:25:17.653928 16840 net.cpp:157] Top shape: 15 128 10 56 56 (60211200)
I1127 16:25:17.653933 16840 net.cpp:165] Memory required for data: 2551449660
I1127 16:25:17.653937 16840 layer_factory.hpp:77] Creating layer conv2_2
I1127 16:25:17.653951 16840 net.cpp:100] Creating Layer conv2_2
I1127 16:25:17.653955 16840 net.cpp:434] conv2_2 <- conv2_1
I1127 16:25:17.653970 16840 net.cpp:408] conv2_2 -> conv2_2
I1127 16:25:17.655359 16840 net.cpp:150] Setting up conv2_2
I1127 16:25:17.655374 16840 net.cpp:157] Top shape: 15 128 10 56 56 (60211200)
I1127 16:25:17.655377 16840 net.cpp:165] Memory required for data: 2792294460
I1127 16:25:17.655386 16840 layer_factory.hpp:77] Creating layer relu2_2
I1127 16:25:17.655395 16840 net.cpp:100] Creating Layer relu2_2
I1127 16:25:17.655398 16840 net.cpp:434] relu2_2 <- conv2_2
I1127 16:25:17.655407 16840 net.cpp:395] relu2_2 -> conv2_2 (in-place)
I1127 16:25:17.655691 16840 net.cpp:150] Setting up relu2_2
I1127 16:25:17.655704 16840 net.cpp:157] Top shape: 15 128 10 56 56 (60211200)
I1127 16:25:17.655709 16840 net.cpp:165] Memory required for data: 3033139260
I1127 16:25:17.655714 16840 layer_factory.hpp:77] Creating layer pool2
I1127 16:25:17.655722 16840 net.cpp:100] Creating Layer pool2
I1127 16:25:17.655726 16840 net.cpp:434] pool2 <- conv2_2
I1127 16:25:17.655735 16840 net.cpp:408] pool2 -> pool2
I1127 16:25:17.656054 16840 net.cpp:150] Setting up pool2
I1127 16:25:17.656066 16840 net.cpp:157] Top shape: 15 128 5 28 28 (7526400)
I1127 16:25:17.656070 16840 net.cpp:165] Memory required for data: 3063244860
I1127 16:25:17.656075 16840 layer_factory.hpp:77] Creating layer conv3_1
I1127 16:25:17.656088 16840 net.cpp:100] Creating Layer conv3_1
I1127 16:25:17.656092 16840 net.cpp:434] conv3_1 <- pool2
I1127 16:25:17.656102 16840 net.cpp:408] conv3_1 -> conv3_1
I1127 16:25:17.658361 16840 net.cpp:150] Setting up conv3_1
I1127 16:25:17.658375 16840 net.cpp:157] Top shape: 15 256 5 28 28 (15052800)
I1127 16:25:17.658378 16840 net.cpp:165] Memory required for data: 3123456060
I1127 16:25:17.658396 16840 layer_factory.hpp:77] Creating layer relu3_1
I1127 16:25:17.658404 16840 net.cpp:100] Creating Layer relu3_1
I1127 16:25:17.658411 16840 net.cpp:434] relu3_1 <- conv3_1
I1127 16:25:17.658418 16840 net.cpp:395] relu3_1 -> conv3_1 (in-place)
I1127 16:25:17.658720 16840 net.cpp:150] Setting up relu3_1
I1127 16:25:17.658731 16840 net.cpp:157] Top shape: 15 256 5 28 28 (15052800)
I1127 16:25:17.658735 16840 net.cpp:165] Memory required for data: 3183667260
I1127 16:25:17.658740 16840 layer_factory.hpp:77] Creating layer conv3_2
I1127 16:25:17.658752 16840 net.cpp:100] Creating Layer conv3_2
I1127 16:25:17.658757 16840 net.cpp:434] conv3_2 <- conv3_1
I1127 16:25:17.658782 16840 net.cpp:408] conv3_2 -> conv3_2
I1127 16:25:17.661945 16840 net.cpp:150] Setting up conv3_2
I1127 16:25:17.661962 16840 net.cpp:157] Top shape: 15 256 5 28 28 (15052800)
I1127 16:25:17.661967 16840 net.cpp:165] Memory required for data: 3243878460
I1127 16:25:17.661974 16840 layer_factory.hpp:77] Creating layer relu3_2
I1127 16:25:17.661981 16840 net.cpp:100] Creating Layer relu3_2
I1127 16:25:17.661985 16840 net.cpp:434] relu3_2 <- conv3_2
I1127 16:25:17.661993 16840 net.cpp:395] relu3_2 -> conv3_2 (in-place)
I1127 16:25:17.662165 16840 net.cpp:150] Setting up relu3_2
I1127 16:25:17.662175 16840 net.cpp:157] Top shape: 15 256 5 28 28 (15052800)
I1127 16:25:17.662179 16840 net.cpp:165] Memory required for data: 3304089660
I1127 16:25:17.662184 16840 layer_factory.hpp:77] Creating layer conv3_3
I1127 16:25:17.662195 16840 net.cpp:100] Creating Layer conv3_3
I1127 16:25:17.662204 16840 net.cpp:434] conv3_3 <- conv3_2
I1127 16:25:17.662212 16840 net.cpp:408] conv3_3 -> conv3_3
I1127 16:25:17.665505 16840 net.cpp:150] Setting up conv3_3
I1127 16:25:17.665520 16840 net.cpp:157] Top shape: 15 256 5 28 28 (15052800)
I1127 16:25:17.665525 16840 net.cpp:165] Memory required for data: 3364300860
I1127 16:25:17.665534 16840 layer_factory.hpp:77] Creating layer relu3_3
I1127 16:25:17.665545 16840 net.cpp:100] Creating Layer relu3_3
I1127 16:25:17.665555 16840 net.cpp:434] relu3_3 <- conv3_3
I1127 16:25:17.665562 16840 net.cpp:395] relu3_3 -> conv3_3 (in-place)
I1127 16:25:17.665863 16840 net.cpp:150] Setting up relu3_3
I1127 16:25:17.665874 16840 net.cpp:157] Top shape: 15 256 5 28 28 (15052800)
I1127 16:25:17.665879 16840 net.cpp:165] Memory required for data: 3424512060
I1127 16:25:17.665884 16840 layer_factory.hpp:77] Creating layer pool3
I1127 16:25:17.665896 16840 net.cpp:100] Creating Layer pool3
I1127 16:25:17.665899 16840 net.cpp:434] pool3 <- conv3_3
I1127 16:25:17.665910 16840 net.cpp:408] pool3 -> pool3
I1127 16:25:17.666259 16840 net.cpp:150] Setting up pool3
I1127 16:25:17.666270 16840 net.cpp:157] Top shape: 15 256 3 14 14 (2257920)
I1127 16:25:17.666275 16840 net.cpp:165] Memory required for data: 3433543740
I1127 16:25:17.666280 16840 layer_factory.hpp:77] Creating layer conv4_1
I1127 16:25:17.666291 16840 net.cpp:100] Creating Layer conv4_1
I1127 16:25:17.666296 16840 net.cpp:434] conv4_1 <- pool3
I1127 16:25:17.666308 16840 net.cpp:408] conv4_1 -> conv4_1
I1127 16:25:17.672219 16840 net.cpp:150] Setting up conv4_1
I1127 16:25:17.672241 16840 net.cpp:157] Top shape: 15 512 3 14 14 (4515840)
I1127 16:25:17.672245 16840 net.cpp:165] Memory required for data: 3451607100
I1127 16:25:17.672255 16840 layer_factory.hpp:77] Creating layer relu4_1
I1127 16:25:17.672266 16840 net.cpp:100] Creating Layer relu4_1
I1127 16:25:17.672271 16840 net.cpp:434] relu4_1 <- conv4_1
I1127 16:25:17.672284 16840 net.cpp:395] relu4_1 -> conv4_1 (in-place)
I1127 16:25:17.672456 16840 net.cpp:150] Setting up relu4_1
I1127 16:25:17.672467 16840 net.cpp:157] Top shape: 15 512 3 14 14 (4515840)
I1127 16:25:17.672473 16840 net.cpp:165] Memory required for data: 3469670460
I1127 16:25:17.672477 16840 layer_factory.hpp:77] Creating layer conv4_2
I1127 16:25:17.672487 16840 net.cpp:100] Creating Layer conv4_2
I1127 16:25:17.672490 16840 net.cpp:434] conv4_2 <- conv4_1
I1127 16:25:17.672497 16840 net.cpp:408] conv4_2 -> conv4_2
I1127 16:25:17.685304 16840 net.cpp:150] Setting up conv4_2
I1127 16:25:17.685333 16840 net.cpp:157] Top shape: 15 512 3 14 14 (4515840)
I1127 16:25:17.685338 16840 net.cpp:165] Memory required for data: 3487733820
I1127 16:25:17.685356 16840 layer_factory.hpp:77] Creating layer relu4_2
I1127 16:25:17.685371 16840 net.cpp:100] Creating Layer relu4_2
I1127 16:25:17.685376 16840 net.cpp:434] relu4_2 <- conv4_2
I1127 16:25:17.685384 16840 net.cpp:395] relu4_2 -> conv4_2 (in-place)
I1127 16:25:17.685693 16840 net.cpp:150] Setting up relu4_2
I1127 16:25:17.685705 16840 net.cpp:157] Top shape: 15 512 3 14 14 (4515840)
I1127 16:25:17.685709 16840 net.cpp:165] Memory required for data: 3505797180
I1127 16:25:17.685737 16840 layer_factory.hpp:77] Creating layer conv4_3
I1127 16:25:17.685752 16840 net.cpp:100] Creating Layer conv4_3
I1127 16:25:17.685799 16840 net.cpp:434] conv4_3 <- conv4_2
I1127 16:25:17.685811 16840 net.cpp:408] conv4_3 -> conv4_3
I1127 16:25:17.699196 16840 net.cpp:150] Setting up conv4_3
I1127 16:25:17.699232 16840 net.cpp:157] Top shape: 15 512 3 14 14 (4515840)
I1127 16:25:17.699236 16840 net.cpp:165] Memory required for data: 3523860540
I1127 16:25:17.699246 16840 layer_factory.hpp:77] Creating layer relu4_3
I1127 16:25:17.699261 16840 net.cpp:100] Creating Layer relu4_3
I1127 16:25:17.699267 16840 net.cpp:434] relu4_3 <- conv4_3
I1127 16:25:17.699275 16840 net.cpp:395] relu4_3 -> conv4_3 (in-place)
I1127 16:25:17.699467 16840 net.cpp:150] Setting up relu4_3
I1127 16:25:17.699479 16840 net.cpp:157] Top shape: 15 512 3 14 14 (4515840)
I1127 16:25:17.699484 16840 net.cpp:165] Memory required for data: 3541923900
I1127 16:25:17.699491 16840 layer_factory.hpp:77] Creating layer pool4
I1127 16:25:17.699504 16840 net.cpp:100] Creating Layer pool4
I1127 16:25:17.699511 16840 net.cpp:434] pool4 <- conv4_3
I1127 16:25:17.699520 16840 net.cpp:408] pool4 -> pool4
I1127 16:25:17.700089 16840 net.cpp:150] Setting up pool4
I1127 16:25:17.700104 16840 net.cpp:157] Top shape: 15 512 2 7 7 (752640)
I1127 16:25:17.700109 16840 net.cpp:165] Memory required for data: 3544934460
I1127 16:25:17.700115 16840 layer_factory.hpp:77] Creating layer conv5_1
I1127 16:25:17.700129 16840 net.cpp:100] Creating Layer conv5_1
I1127 16:25:17.700135 16840 net.cpp:434] conv5_1 <- pool4
I1127 16:25:17.700145 16840 net.cpp:408] conv5_1 -> conv5_1
I1127 16:25:17.714501 16840 net.cpp:150] Setting up conv5_1
I1127 16:25:17.714529 16840 net.cpp:157] Top shape: 15 512 2 7 7 (752640)
I1127 16:25:17.714534 16840 net.cpp:165] Memory required for data: 3547945020
I1127 16:25:17.714545 16840 layer_factory.hpp:77] Creating layer relu5_1
I1127 16:25:17.714561 16840 net.cpp:100] Creating Layer relu5_1
I1127 16:25:17.714566 16840 net.cpp:434] relu5_1 <- conv5_1
I1127 16:25:17.714573 16840 net.cpp:395] relu5_1 -> conv5_1 (in-place)
I1127 16:25:17.714917 16840 net.cpp:150] Setting up relu5_1
I1127 16:25:17.714975 16840 net.cpp:157] Top shape: 15 512 2 7 7 (752640)
I1127 16:25:17.714980 16840 net.cpp:165] Memory required for data: 3550955580
I1127 16:25:17.714987 16840 layer_factory.hpp:77] Creating layer conv5_2
I1127 16:25:17.715001 16840 net.cpp:100] Creating Layer conv5_2
I1127 16:25:17.715008 16840 net.cpp:434] conv5_2 <- conv5_1
I1127 16:25:17.715018 16840 net.cpp:408] conv5_2 -> conv5_2
I1127 16:25:17.727910 16840 net.cpp:150] Setting up conv5_2
I1127 16:25:17.727939 16840 net.cpp:157] Top shape: 15 512 2 7 7 (752640)
I1127 16:25:17.727942 16840 net.cpp:165] Memory required for data: 3553966140
I1127 16:25:17.727952 16840 layer_factory.hpp:77] Creating layer relu5_2
I1127 16:25:17.727962 16840 net.cpp:100] Creating Layer relu5_2
I1127 16:25:17.727967 16840 net.cpp:434] relu5_2 <- conv5_2
I1127 16:25:17.727973 16840 net.cpp:395] relu5_2 -> conv5_2 (in-place)
I1127 16:25:17.728133 16840 net.cpp:150] Setting up relu5_2
I1127 16:25:17.728142 16840 net.cpp:157] Top shape: 15 512 2 7 7 (752640)
I1127 16:25:17.728144 16840 net.cpp:165] Memory required for data: 3556976700
I1127 16:25:17.728148 16840 layer_factory.hpp:77] Creating layer conv5_3
I1127 16:25:17.728157 16840 net.cpp:100] Creating Layer conv5_3
I1127 16:25:17.728160 16840 net.cpp:434] conv5_3 <- conv5_2
I1127 16:25:17.728168 16840 net.cpp:408] conv5_3 -> conv5_3
I1127 16:25:17.741930 16840 net.cpp:150] Setting up conv5_3
I1127 16:25:17.741958 16840 net.cpp:157] Top shape: 15 512 2 7 7 (752640)
I1127 16:25:17.741962 16840 net.cpp:165] Memory required for data: 3559987260
I1127 16:25:17.741971 16840 layer_factory.hpp:77] Creating layer relu5_3
I1127 16:25:17.741981 16840 net.cpp:100] Creating Layer relu5_3
I1127 16:25:17.741986 16840 net.cpp:434] relu5_3 <- conv5_3
I1127 16:25:17.741994 16840 net.cpp:395] relu5_3 -> conv5_3 (in-place)
I1127 16:25:17.742285 16840 net.cpp:150] Setting up relu5_3
I1127 16:25:17.742311 16840 net.cpp:157] Top shape: 15 512 2 7 7 (752640)
I1127 16:25:17.742316 16840 net.cpp:165] Memory required for data: 3562997820
I1127 16:25:17.742319 16840 layer_factory.hpp:77] Creating layer pool5
I1127 16:25:17.742333 16840 net.cpp:100] Creating Layer pool5
I1127 16:25:17.742337 16840 net.cpp:434] pool5 <- conv5_3
I1127 16:25:17.742343 16840 net.cpp:408] pool5 -> pool5
I1127 16:25:17.742537 16840 net.cpp:150] Setting up pool5
I1127 16:25:17.742547 16840 net.cpp:157] Top shape: 15 512 1 4 4 (122880)
I1127 16:25:17.742549 16840 net.cpp:165] Memory required for data: 3563489340
I1127 16:25:17.742552 16840 layer_factory.hpp:77] Creating layer fc6
I1127 16:25:17.742568 16840 net.cpp:100] Creating Layer fc6
I1127 16:25:17.742573 16840 net.cpp:434] fc6 <- pool5
I1127 16:25:17.742578 16840 net.cpp:408] fc6 -> fc6
I1127 16:25:17.799993 16840 net.cpp:150] Setting up fc6
I1127 16:25:17.800020 16840 net.cpp:157] Top shape: 15 4096 (61440)
I1127 16:25:17.800024 16840 net.cpp:165] Memory required for data: 3563735100
I1127 16:25:17.800034 16840 layer_factory.hpp:77] Creating layer relu6
I1127 16:25:17.800045 16840 net.cpp:100] Creating Layer relu6
I1127 16:25:17.800050 16840 net.cpp:434] relu6 <- fc6
I1127 16:25:17.800057 16840 net.cpp:395] relu6 -> fc6 (in-place)
I1127 16:25:17.800489 16840 net.cpp:150] Setting up relu6
I1127 16:25:17.800500 16840 net.cpp:157] Top shape: 15 4096 (61440)
I1127 16:25:17.800503 16840 net.cpp:165] Memory required for data: 3563980860
I1127 16:25:17.800506 16840 layer_factory.hpp:77] Creating layer drop6
I1127 16:25:17.800514 16840 net.cpp:100] Creating Layer drop6
I1127 16:25:17.800518 16840 net.cpp:434] drop6 <- fc6
I1127 16:25:17.800523 16840 net.cpp:395] drop6 -> fc6 (in-place)
I1127 16:25:17.800559 16840 net.cpp:150] Setting up drop6
I1127 16:25:17.800565 16840 net.cpp:157] Top shape: 15 4096 (61440)
I1127 16:25:17.800567 16840 net.cpp:165] Memory required for data: 3564226620
I1127 16:25:17.800571 16840 layer_factory.hpp:77] Creating layer fc7
I1127 16:25:17.800578 16840 net.cpp:100] Creating Layer fc7
I1127 16:25:17.800581 16840 net.cpp:434] fc7 <- fc6
I1127 16:25:17.800586 16840 net.cpp:408] fc7 -> fc7
I1127 16:25:17.828896 16840 net.cpp:150] Setting up fc7
I1127 16:25:17.828922 16840 net.cpp:157] Top shape: 15 4096 (61440)
I1127 16:25:17.828925 16840 net.cpp:165] Memory required for data: 3564472380
I1127 16:25:17.828936 16840 layer_factory.hpp:77] Creating layer relu7
I1127 16:25:17.828949 16840 net.cpp:100] Creating Layer relu7
I1127 16:25:17.828954 16840 net.cpp:434] relu7 <- fc7
I1127 16:25:17.828960 16840 net.cpp:395] relu7 -> fc7 (in-place)
I1127 16:25:17.829270 16840 net.cpp:150] Setting up relu7
I1127 16:25:17.829280 16840 net.cpp:157] Top shape: 15 4096 (61440)
I1127 16:25:17.829283 16840 net.cpp:165] Memory required for data: 3564718140
I1127 16:25:17.829288 16840 layer_factory.hpp:77] Creating layer drop7
I1127 16:25:17.829298 16840 net.cpp:100] Creating Layer drop7
I1127 16:25:17.829301 16840 net.cpp:434] drop7 <- fc7
I1127 16:25:17.829308 16840 net.cpp:395] drop7 -> fc7 (in-place)
I1127 16:25:17.829337 16840 net.cpp:150] Setting up drop7
I1127 16:25:17.829344 16840 net.cpp:157] Top shape: 15 4096 (61440)
I1127 16:25:17.829346 16840 net.cpp:165] Memory required for data: 3564963900
I1127 16:25:17.829349 16840 layer_factory.hpp:77] Creating layer fc8-1
I1127 16:25:17.829358 16840 net.cpp:100] Creating Layer fc8-1
I1127 16:25:17.829361 16840 net.cpp:434] fc8-1 <- fc7
I1127 16:25:17.829367 16840 net.cpp:408] fc8-1 -> fc8-1
I1127 16:25:17.833370 16840 net.cpp:150] Setting up fc8-1
I1127 16:25:17.833382 16840 net.cpp:157] Top shape: 15 101 (1515)
I1127 16:25:17.833385 16840 net.cpp:165] Memory required for data: 3564969960
I1127 16:25:17.833391 16840 layer_factory.hpp:77] Creating layer loss
I1127 16:25:17.833405 16840 net.cpp:100] Creating Layer loss
I1127 16:25:17.833410 16840 net.cpp:434] loss <- fc8-1
I1127 16:25:17.833413 16840 net.cpp:434] loss <- label
I1127 16:25:17.833421 16840 net.cpp:408] loss -> (automatic)
I1127 16:25:17.833432 16840 layer_factory.hpp:77] Creating layer loss
I1127 16:25:17.833884 16840 net.cpp:150] Setting up loss
I1127 16:25:17.833895 16840 net.cpp:157] Top shape: (1)
I1127 16:25:17.833899 16840 net.cpp:160]     with loss weight 1
I1127 16:25:17.833919 16840 net.cpp:165] Memory required for data: 3564969964
....

I1127 16:25:21.730363 16840 solver.cpp:228] Iteration 0, loss = 4.61512
I1127 16:25:21.730406 16840 sgd_solver.cpp:106] Iteration 0, lr = 0.001
I1127 16:28:17.407043 16840 solver.cpp:228] Iteration 20, loss = 4.6148
I1127 16:28:17.407192 16840 sgd_solver.cpp:106] Iteration 20, lr = 0.001
I1127 16:31:17.338194 16840 solver.cpp:228] Iteration 40, loss = 4.61508
I1127 16:31:17.338304 16840 sgd_solver.cpp:106] Iteration 40, lr = 0.001
I1127 16:34:18.256834 16840 solver.cpp:228] Iteration 60, loss = 4.61379
I1127 16:34:18.256958 16840 sgd_solver.cpp:106] Iteration 60, lr = 0.001
I1127 16:37:24.625149 16840 solver.cpp:228] Iteration 80, loss = 4.61471
I1127 16:37:24.625257 16840 sgd_solver.cpp:106] Iteration 80, lr = 0.001
I1127 16:40:31.401072 16840 solver.cpp:228] Iteration 100, loss = 4.61211
...
...
...

I1129 13:02:16.929947 16840 sgd_solver.cpp:106] Iteration 18720, lr = 0.001
I1129 13:05:05.776618 16840 solver.cpp:228] Iteration 18740, loss = 4.60391
I1129 13:05:05.776726 16840 sgd_solver.cpp:106] Iteration 18740, lr = 0.001
I1129 13:07:53.860440 16840 solver.cpp:228] Iteration 18760, loss = 4.4449
I1129 13:07:53.860537 16840 sgd_solver.cpp:106] Iteration 18760, lr = 0.001
I1129 13:10:42.574254 16840 solver.cpp:228] Iteration 18780, loss = 4.52169
I1129 13:10:42.574362 16840 sgd_solver.cpp:106] Iteration 18780, lr = 0.001
I1129 13:13:31.981536 16840 solver.cpp:228] Iteration 18800, loss = 4.38754
I1129 13:13:31.981652 16840 sgd_solver.cpp:106] Iteration 18800, lr = 0.001
I1129 13:16:20.288398 16840 solver.cpp:228] Iteration 18820, loss = 4.47064
I1129 13:16:20.288506 16840 sgd_solver.cpp:106] Iteration 18820, lr = 0.001
I1129 13:19:08.195053 16840 solver.cpp:228] Iteration 18840, loss = 4.69076
I1129 13:19:08.195155 16840 sgd_solver.cpp:106] Iteration 18840, lr = 0.001```

Thank you for your help,
livst
chuckcho commented 7 years ago

If you post VGG+C3D network definition (at gist or here), I can try.

LuisTbx commented 7 years ago

Try adding BN layers between convolutions, they work quite well with the VGG type definition. Why? Because it makes the optimisation less sensitive to the learning rate. Hope it helps. Cheers.

chuckcho commented 6 years ago

Closing this issue.