search

NVIDIA / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
672 stars 263 forks source link

Segmentation fault trying to import Python layer #411

Closed Bidski closed 7 years ago

Bidski commented 7 years ago

I have a modified version of the detectnet network that I am attempting to train, but when caffe tries to import the caffe.layers.detectnet.clustering Python module I get a segmentation fault. Here is an excerpt from the caffe output.

I0901 11:43:28.637195 6795 layer_factory.hpp:136] Creating layer 'cluster' of type 'Python' I0901 11:43:28.637199 6795 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT I0901 11:43:28.637219 6795 layerfactory.cpp:325] Importing Python module 'caffe.layers.detectnet.clustering' Aborted at 1504230208 (unix time) try "date -d @1504230208" if you are using GNU date PC: @ 0x7f3225b1cfaf dgetrf SIGSEGV (@0x280) received by PID 6795 (TID 0x7f32ad016d40) from PID 640; stack trace: @ 0x7f32ab1a77e0 (unknown) @ 0x7f3225b1cfaf dgetrf_ @ 0x7f3238e95715 f2py_routflapack_dgetrf @ 0x7f32900820e3 PyObject_Call @ 0x7f329005bfed PyEval_EvalFrameEx @ 0x7f32900b74c5 PyEval_EvalCodeEx @ 0x7f329005c14e PyEval_EvalFrameEx @ 0x7f32900b74c5 PyEval_EvalCodeEx @ 0x7f32900d18f9 PyEval_EvalCode @ 0x7f32900ed6e8 PyImport_ExecCodeModuleEx @ 0x7f32900ed8cf load_source_module @ 0x7f329009dc87 PyImportImportModuleLevel @ 0x7f32900a0768 builtinimport.lto_priv.1230 @ 0x7f32900820e3 PyObject_Call @ 0x7f32900b6bd0 PyEval_CallObjectWithKeywords @ 0x7f329005a3c9 PyEval_EvalFrameEx @ 0x7f32900b74c5 PyEval_EvalCodeEx @ 0x7f32900d18f9 PyEval_EvalCode @ 0x7f32900ed6e8 PyImport_ExecCodeModuleEx @ 0x7f32900ed8cf load_source_module @ 0x7f32900ee33a load_package @ 0x7f329009dc87 PyImportImportModuleLevel @ 0x7f32900a0768 builtinimport__.lto_priv.1230 @ 0x7f32900820e3 PyObject_Call @ 0x7f32900b6bd0 PyEval_CallObjectWithKeywords @ 0x7f329005a3c9 PyEval_EvalFrameEx @ 0x7f32900b74c5 PyEval_EvalCodeEx @ 0x7f32900d18f9 PyEval_EvalCode @ 0x7f32900ed6e8 PyImport_ExecCodeModuleEx @ 0x7f32900ed8cf load_source_module @ 0x7f329009dc87 PyImport_ImportModuleLevel @ 0x7f32900a0768 builtin_import.lto_priv.1230

I made a quick C++ source file to try and load in the Python module using boost and this works successfully. Here is the source code for this program

#include <iostream>
#include <string>
#include <boost/python.hpp>

class PyGILAquire {
    PyGILState_STATE state_;
public:
    PyGILAquire() {
        state_ = PyGILState_Ensure();
    }
    ~PyGILAquire() {
        PyGILState_Release(state_);
    }

private:
    PyGILAquire(const PyGILAquire&) = delete;
    PyGILAquire(PyGILAquire&&) = delete;
    PyGILAquire& operator=(const PyGILAquire&) = delete;
    PyGILAquire& operator=(PyGILAquire&&) = delete;
};

class PyGILRelease {
    PyThreadState *state_;
public:
    PyGILRelease() {
        state_ = PyEval_SaveThread();
    }
    ~PyGILRelease() {
        PyEval_RestoreThread(state_);
    }

private:
    PyGILRelease(const PyGILRelease&) = delete;
    PyGILRelease(PyGILRelease&&) = delete;
    PyGILRelease& operator=(const PyGILRelease&) = delete;
    PyGILRelease& operator=(PyGILRelease&&) = delete;
};

__attribute__((constructor)) void loadso() {
//  PyEval_InitThreads();
  Py_Initialize();
}

__attribute__((destructor))  void unloadso() {
}

int main() {
    boost::python::object module;
    PyGILRelease pygr;
    {
        PyGILAquire pgil;
        module = boost::python::import("caffe.layers.detectnet.clustering");
    }
    boost::python::object layer = module.attr("cluster");

    return 0;
}

This program runs without segmentation fault.

I have Boost 1.64 Python 2.7 Arch Linux OS Latest caffe-0.16 (pulled and compiled Sep 1st 2017) Latest DIGITS (pulled Sep 1st 2017, version 6.0.0-rc.2)

My PYTHONPATH environment variable also includes CAFFE_ROOT/python

Does anyone know why caffe is failing to load this Python module?

drnikolaev commented 7 years ago

@Bidski thank you. Looks like not Caffe but LAPACK routine dgetrf executed under NumPy:

PC: @ 0x7f3225b1cfaf dgetrf_
*** SIGSEGV (@0x280) received by PID 6795 (TID 0x7f32ad016d40) from PID 640; stack trace: ***
@ 0x7f32ab1a77e0 (unknown)
@ 0x7f3225b1cfaf dgetrf_
@ 0x7f3238e95715 f2py_rout__flapack_dgetrf
@ 0x7f32900820e3 PyObject_Call

Could you try to print something from every member function of clustering? Also, Python 3 might do better job here.

Bidski commented 7 years ago

@drnikolaev I have added print statements as you suggested, but none of them are getting printed to the console. Has caffe/boost done something to suppress the output or are they just not getting called?

drnikolaev commented 7 years ago

They are just not called. Any way to try this on Ubuntu? Also, how do you build everything? Could you paste all commands you invoke and their outputs?

Bidski commented 7 years ago

I pulled the caffe branch again. I am now running caffe 0.16.4.

Any way to try this on Ubuntu? This is a little difficult. I should be able to get access to a Jetson TX2, I will try on there as soon as I can, but apart from that I have no other Ubuntu machines that I have access to.

Also, how do you build everything? I assume you mean "How do I build caffe?" If so, I use cmake + ninja. The commands I execute are

  1. mkdir build && cd build
  2. cmake .. -G Ninja
  3. ccmake ..
    • Set BLAS to Open
    • Set NCCL paths
    • Set CUDA_HOST_COMPILER to gcc-5
    • Re-generate makefiles
  4. ninja

Full cmake output

    -- Boost version: 1.64.0
    -- Found the following Boost libraries:
        --   system
        --   thread
        --   filesystem
        --   chrono
        --   date_time
        --   atomic
    -- Found gflags  (include: /usr/include, library: /usr/lib/libgflags.so)
    -- Found glog    (include: /usr/include, library: /usr/lib/libglog.so)
    -- Found PROTOBUF Compiler: /usr/bin/protoc
    -- HDF5: Using hdf5 compiler wrapper to determine C configuration
    -- HDF5: Using hdf5 compiler wrapper to determine CXX configuration
    -- Found lmdb    (include: /usr/include, library: /usr/lib/liblmdb.so)
    -- Found LevelDB (include: /usr/include, library: /usr/lib/libleveldb.so)
    -- Found Snappy  (include: /usr/include, library: /usr/lib/libsnappy.so)
    -- CUDA detected: 8.0
    -- Added CUDA NVCC flags for: sm_61
    -- Found OpenCV 2.x: /usr/local/share/OpenCV
    -- Found OpenBLAS libraries: /usr/lib/libopenblas.so
    -- Found OpenBLAS include: /usr/include
    -- NumPy ver. 1.13.1 found (include: /usr/lib/python2.7/site-packages/numpy/core/include)
    -- Boost version: 1.64.0
    -- Found the following Boost libraries:
        --   python
    -- Found NCCL (include: /opt/cuda/include, library: /opt/cuda/lib/libnccl.so)
    -- Found NVML (include: /opt/cuda/include, library: /usr/lib/libnvidia-ml.so)
    -- Detected Doxygen OUTPUT_DIRECTORY: ./doxygen/
    -- 
    -- ******************* Caffe Configuration Summary *******************
    -- General:
    --   Version           :   0.16.4
    --   Git               :   v0.16.4-dirty
    --   System            :   Linux
    --   C++ compiler      :   /usr/bin/c++
    --   Release CXX flags :   -O3 -DNDEBUG -fPIC -Wall -std=c++11 -Wno-sign-compare -Wno-  uninitialized
    --   Debug CXX flags   :   -g -DDEBUG -fPIC -Wall -std=c++11 -Wno-sign-compare -Wno-uninitialized
    --   Build type        :   Release
    -- 
    --   BUILD_SHARED_LIBS :   ON
    --   BUILD_python      :   ON
    --   BUILD_matlab      :   OFF
    --   BUILD_docs        :   ON
    --   CPU_ONLY          :   OFF
    --   USE_OPENCV        :   ON
    --   USE_LEVELDB       :   ON
    --   USE_LMDB          :   ON
    --   ALLOW_LMDB_NOLOCK :   OFF
    --   TEST_FP16         :   OFF
    -- 
    -- Dependencies:
    --   BLAS              :   Yes (Open)
    --   Boost             :   Yes (ver. 1.64)
    --   glog              :   Yes
    --   gflags            :   Yes
    --   protobuf          :   Yes (ver. 3.3.2)
    --   lmdb              :   Yes (ver. 0.9.21)
    --   LevelDB           :   Yes (ver. 1.20)
    --   Snappy            :   Yes (ver. 1.1.4)
    --   OpenCV            :   Yes (ver. 2.4.13.3)
    --   CUDA              :   Yes (ver. 8.0)
    -- 
    -- NVIDIA CUDA:
    --   Target GPU(s)     :   Auto
    --   GPU arch(s)       :   sm_61
    --   cuDNN             :   Yes (ver. 6.0)
    --   NCCL              :   Yes
    --   NVML              :   /usr/lib/libnvidia-ml.so 
    -- 
    -- Python:
    --   Interpreter       :   /usr/bin/python2.7 (ver. 2.7.13)
    --   Libraries         :   /usr/lib/libpython2.7.so (ver 2.7.13)
    --   NumPy             :   /usr/lib/python2.7/site-packages/numpy/core/include (ver 1.13.1)
    -- 
    -- Documentaion:
    --   Doxygen           :   /usr/bin/doxygen (1.8.13)
    --   config_file       :   /home/bidski/Projects/nvidia/caffe/.Doxyfile
    -- 
    -- Install:
    --   Install path      :   /home/bidski/Projects/nvidia/caffe/build/install
    -- 
    -- Configuring done
    -- Generating done
    CMake Warning (dev):
      Policy CMP0058 is not set: Ninja requires custom command byproducts to be
      explicit.  Run "cmake --help-policy CMP0058" for policy details.  Use the
      cmake_policy command to set the policy and suppress this warning.

      This project specifies custom command DEPENDS on files in the build tree
      that are not specified as the OUTPUT or BYPRODUCTS of any
      add_custom_command or add_custom_target:

       src/caffe/CMakeFiles/cuda_compile_1.dir  /cuda_compile_1_generated_data_transformer.cu.o.Release.cmake
       src/caffe/test/CMakeFiles/cuda_compile_1.dir/cuda_compile_1_generated_test_im2col_kernel.cu.o.Release.cmake

      For compatibility with versions of CMake that did not have the BYPRODUCTS
      option, CMake is generating phony rules for such files to convince 'ninja'
      to build.

      Project authors should add the missing BYPRODUCTS or OUTPUT options to the
      custom commands that produce these files.
     This warning is for project developers.  Use -Wno-dev to suppress it.

    -- Build files have been written to: /home/bidski/Projects/nvidia/caffe/build

Full output from attempting to train the network

I0904 09:16:56.011200 14545 caffe.cpp:470] This is NVCaffe 0.16.4 started at Mon Sep  4 09:16:55 2017
I0904 09:16:56.011343 14545 caffe.cpp:473] CuDNN version: 6021
I0904 09:16:56.011366 14545 caffe.cpp:474] CuBLAS version: 8000
I0904 09:16:56.011369 14545 caffe.cpp:475] CUDA version: 8000
I0904 09:16:56.011373 14545 caffe.cpp:476] CUDA driver version: 9000
I0904 09:16:56.012531 14545 upgrade_proto.cpp:1044] Attempting to upgrade input file specified using deprecated 'solver_type' field (enum)': /home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170901-131346-f68c/solver.prototxt
I0904 09:16:56.012547 14545 upgrade_proto.cpp:1051] Successfully upgraded file specified using deprecated 'solver_type' field (enum) to 'type' field (string).
W0904 09:16:56.012550 14545 upgrade_proto.cpp:1053] Note that future Caffe releases will only support 'type' field (string) for a solver's type.
I0904 09:16:56.012959 14545 gpu_memory.cpp:159] GPUMemory::Manager initialized with Caching (CUB) GPU Allocator
I0904 09:16:56.012969 14545 gpu_memory.cpp:161] Total memory: 2098593792, Free: 2005270528, dev_info[0]: total=2098593792 free=2005270528
I0904 09:16:56.013309 14545 caffe.cpp:198] Using GPUs 0
I0904 09:16:56.013541 14545 caffe.cpp:203] GPU 0: GeForce GTX 1050
I0904 09:16:56.013566 14545 solver.cpp:41] Solver data type: FLOAT
I0904 09:16:56.013571 14545 solver.cpp:44] Initializing solver from parameters: 
test_iter: 400
test_interval: 300
base_lr: 0.001
display: 37
max_iter: 600000
lr_policy: "step"
gamma: 0.1
momentum: 0.9
weight_decay: 1e-05
stepsize: 150000
snapshot: 300
snapshot_prefix: "snapshot"
solver_mode: GPU
device_id: 0
net: "/home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170901-131346-f68c/train_val.prototxt"
type: "Adam"
I0904 09:16:56.013705 14545 solver.cpp:85] Creating training net from net file: /home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170901-131346-f68c/train_val.prototxt
I0904 09:16:56.014008 14545 net.cpp:441] The NetState phase (0) differed from the phase (1) specified by a rule in layer val-data
I0904 09:16:56.014014 14545 net.cpp:441] The NetState phase (0) differed from the phase (1) specified by a rule in layer val-label
I0904 09:16:56.014017 14545 net.cpp:441] The NetState phase (0) differed from the phase (1) specified by a rule in layer data_transform
I0904 09:16:56.014027 14545 net.cpp:441] The NetState phase (0) differed from the phase (1) specified by a rule in layer cluster
I0904 09:16:56.014029 14545 net.cpp:441] The NetState phase (0) differed from the phase (1) specified by a rule in layer cluster_gt
I0904 09:16:56.014031 14545 net.cpp:441] The NetState phase (0) differed from the phase (1) specified by a rule in layer score
I0904 09:16:56.014034 14545 net.cpp:441] The NetState phase (0) differed from the phase (1) specified by a rule in layer mAP
I0904 09:16:56.014052 14545 net.cpp:70] Initializing net from parameters: 
state {
  phase: TRAIN
}
layer {
  name: "train-data"
  type: "Data"
  top: "data"
  include {
    phase: TRAIN
  }
  data_param {
    source: "/home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170807-135507-149f/train_db/features"
    batch_size: 128
    backend: LMDB
  }
}
layer {
  name: "train-label"
  type: "Data"
  top: "label"
  include {
    phase: TRAIN
  }
  data_param {
    source: "/home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170807-135507-149f/train_db/labels"
    batch_size: 128
    backend: LMDB
  }
}
layer {
  name: "data_transform"
  type: "DetectNetTransformation"
  bottom: "data"
  bottom: "label"
  top: "transformed-data"
  top: "transformed-label"
  include {
    phase: TRAIN
  }
  transform_param {
    mean_value: 127
  }
  detectnet_groundtruth_param {
    stride: 16
    scale_cvg: 0.4
    gridbox_type: GRIDBOX_MIN
    min_cvg_len: 20
    coverage_type: RECTANGULAR
    image_size_x: 32
    image_size_y: 32
    obj_norm: true
    crop_bboxes: false
  }
  detectnet_augmentation_param {
    crop_prob: 1
    shift_x: 32
    shift_y: 32
    scale_prob: 0.4
    scale_min: 0.8
    scale_max: 1.2
    flip_prob: 0.5
    rotation_prob: 0
    max_rotate_degree: 5
    hue_rotation_prob: 0.8
    hue_rotation: 30
    desaturation_prob: 0.8
    desaturation_max: 0.8
  }
}
layer {
  name: "slice-label"
  type: "Slice"
  bottom: "transformed-label"
  top: "foreground-label"
  top: "bbox-label"
  top: "size-label"
  top: "obj-label"
  top: "coverage-label"
  slice_param {
    slice_point: 1
    slice_point: 5
    slice_point: 7
    slice_point: 8
  }
}
layer {
  name: "coverage-block"
  type: "Concat"
  bottom: "foreground-label"
  bottom: "foreground-label"
  bottom: "foreground-label"
  bottom: "foreground-label"
  top: "coverage-block"
}
layer {
  name: "size-block"
  type: "Concat"
  bottom: "size-label"
  bottom: "size-label"
  top: "size-block"
}
layer {
  name: "obj-block"
  type: "Concat"
  bottom: "obj-label"
  bottom: "obj-label"
  bottom: "obj-label"
  bottom: "obj-label"
  top: "obj-block"
}
layer {
  name: "bb-label-norm"
  type: "Eltwise"
  bottom: "bbox-label"
  bottom: "size-block"
  top: "bbox-label-norm"
  eltwise_param {
    operation: PROD
  }
}
layer {
  name: "bb-obj-norm"
  type: "Eltwise"
  bottom: "bbox-label-norm"
  bottom: "obj-block"
  top: "bbox-obj-label-norm"
  eltwise_param {
    operation: PROD
  }
}
layer {
  name: "conv1"
  type: "Convolution"
  bottom: "transformed-data"
  top: "conv1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 108
    kernel_size: 5
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "relu1"
  type: "ReLU"
  bottom: "conv1"
  top: "conv1"
}
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "pool1"
  top: "conv2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 108
    pad: 2
    kernel_size: 5
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0.1
    }
  }
}
layer {
  name: "relu2"
  type: "ReLU"
  bottom: "conv2"
  top: "conv2"
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "pool3"
  type: "Pooling"
  bottom: "pool1"
  top: "pool3"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "concat"
  type: "Concat"
  bottom: "pool3"
  bottom: "pool2"
  top: "concat"
  concat_param {
    axis: 1
  }
}
layer {
  name: "conv3"
  type: "Convolution"
  bottom: "concat"
  top: "conv3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 100
    pad: 0
    kernel_size: 7
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0.1
    }
  }
}
layer {
  name: "relu3"
  type: "ReLU"
  bottom: "conv3"
  top: "conv3"
}
layer {
  name: "drop3"
  type: "Dropout"
  bottom: "conv3"
  top: "conv3"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "conv4"
  type: "Convolution"
  bottom: "conv3"
  top: "conv4"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 55
    pad: 0
    kernel_size: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0.1
    }
  }
}
layer {
  name: "cvg/classifier"
  type: "Convolution"
  bottom: "conv4"
  top: "cvg/classifier"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 1
    pad: 2
    kernel_size: 2
    stride: 2
    weight_filler {
      type: "xavier"
      std: 0.03
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "coverage/sig"
  type: "Sigmoid"
  bottom: "cvg/classifier"
  top: "coverage"
}
layer {
  name: "bbox/regressor"
  type: "Convolution"
  bottom: "conv4"
  top: "bboxes"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 4
    pad: 1
    kernel_size: 2
    stride: 1
    weight_filler {
      type: "xavier"
      std: 0.03
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "bbox_mask"
  type: "Eltwise"
  bottom: "bboxes"
  bottom: "coverage-block"
  top: "bboxes-masked"
  eltwise_param {
    operation: PROD
  }
}
layer {
  name: "bbox-norm"
  type: "Eltwise"
  bottom: "bboxes-masked"
  bottom: "size-block"
  top: "bboxes-masked-norm"
  eltwise_param {
    operation: PROD
  }
}
layer {
  name: "bbox-obj-norm"
  type: "Eltwise"
  bottom: "bboxes-masked-norm"
  bottom: "obj-block"
  top: "bboxes-obj-masked-norm"
  eltwise_param {
    operation: PROD
  }
}
layer {
  name: "bbox_loss"
  type: "L1Loss"
  bottom: "bboxes-obj-masked-norm"
  bottom: "bbox-obj-label-norm"
  top: "loss_bbox"
  loss_weight: 2
}
layer {
  name: "coverage_loss"
  type: "EuclideanLoss"
  bottom: "coverage"
  bottom: "coverage-label"
  top: "loss_coverage"
}
I0904 09:16:56.014288 14545 net.cpp:102] Using FLOAT as default forward math type
I0904 09:16:56.014294 14545 net.cpp:108] Using FLOAT as default backward math type
I0904 09:16:56.014297 14545 layer_factory.hpp:136] Creating layer 'train-data' of type 'Data'
I0904 09:16:56.014300 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.014348 14545 net.cpp:182] Created Layer train-data (0)
I0904 09:16:56.014353 14545 net.cpp:528] train-data -> data
I0904 09:16:56.014386 14545 data_reader.cpp:52] Sample Data Reader threads: 1, out queues: 1, depth: 128
I0904 09:16:56.014447 14545 internal_thread.cpp:19] Starting 1 internal thread(s) on device 0
I0904 09:16:56.020126 14554 db_lmdb.cpp:24] Opened lmdb /home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170807-135507-149f/train_db/features
I0904 09:16:56.022285 14545 data_layer.cpp:187] [0] ReshapePrefetch 128, 3, 384, 384
I0904 09:16:56.022316 14545 data_layer.cpp:211] [0] Output data size: 128, 3, 384, 384
I0904 09:16:56.022331 14545 internal_thread.cpp:19] Starting 1 internal thread(s) on device 0
I0904 09:16:56.022359 14545 net.cpp:243] Setting up train-data
I0904 09:16:56.022364 14545 net.cpp:250] TRAIN Top shape for layer 0 'train-data' 128 3 384 384 (56623104)
I0904 09:16:56.022408 14545 layer_factory.hpp:136] Creating layer 'train-label' of type 'Data'
I0904 09:16:56.022413 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.022433 14545 net.cpp:182] Created Layer train-label (1)
I0904 09:16:56.022438 14545 net.cpp:528] train-label -> label
I0904 09:16:56.022446 14545 data_reader.cpp:52] Sample Data Reader threads: 1, out queues: 1, depth: 128
I0904 09:16:56.022464 14545 internal_thread.cpp:19] Starting 1 internal thread(s) on device 0
I0904 09:16:56.023104 14556 db_lmdb.cpp:24] Opened lmdb /home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170807-135507-149f/train_db/labels
I0904 09:16:56.023144 14545 data_layer.cpp:187] [0] ReshapePrefetch 128, 1, 2, 16
I0904 09:16:56.023159 14545 data_layer.cpp:211] [0] Output data size: 128, 1, 2, 16
I0904 09:16:56.023164 14545 internal_thread.cpp:19] Starting 1 internal thread(s) on device 0
I0904 09:16:56.023356 14545 net.cpp:243] Setting up train-label
I0904 09:16:56.023375 14545 net.cpp:250] TRAIN Top shape for layer 1 'train-label' 128 1 2 16 (4096)
I0904 09:16:56.023399 14545 layer_factory.hpp:136] Creating layer 'data_transform' of type 'DetectNetTransformation'
I0904 09:16:56.023403 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.023442 14545 net.cpp:182] Created Layer data_transform (2)
I0904 09:16:56.023459 14545 net.cpp:559] data_transform <- data
I0904 09:16:56.023468 14545 net.cpp:559] data_transform <- label
I0904 09:16:56.023471 14545 net.cpp:528] data_transform -> transformed-data
I0904 09:16:56.023481 14545 net.cpp:528] data_transform -> transformed-label
I0904 09:16:56.024720 14545 net.cpp:243] Setting up data_transform
I0904 09:16:56.024734 14545 net.cpp:250] TRAIN Top shape for layer 2 'data_transform' 128 3 32 32 (393216)
I0904 09:16:56.024740 14545 net.cpp:250] TRAIN Top shape for layer 2 'data_transform' 128 9 2 2 (4608)
I0904 09:16:56.024744 14545 layer_factory.hpp:136] Creating layer 'slice-label' of type 'Slice'
I0904 09:16:56.024749 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.024755 14545 net.cpp:182] Created Layer slice-label (3)
I0904 09:16:56.024757 14545 net.cpp:559] slice-label <- transformed-label
I0904 09:16:56.024761 14545 net.cpp:528] slice-label -> foreground-label
I0904 09:16:56.024766 14545 net.cpp:528] slice-label -> bbox-label
I0904 09:16:56.024770 14545 net.cpp:528] slice-label -> size-label
I0904 09:16:56.024773 14545 net.cpp:528] slice-label -> obj-label
I0904 09:16:56.024776 14545 net.cpp:528] slice-label -> coverage-label
I0904 09:16:56.024862 14545 net.cpp:243] Setting up slice-label
I0904 09:16:56.024866 14545 net.cpp:250] TRAIN Top shape for layer 3 'slice-label' 128 1 2 2 (512)
I0904 09:16:56.024870 14545 net.cpp:250] TRAIN Top shape for layer 3 'slice-label' 128 4 2 2 (2048)
I0904 09:16:56.024873 14545 net.cpp:250] TRAIN Top shape for layer 3 'slice-label' 128 2 2 2 (1024)
I0904 09:16:56.024876 14545 net.cpp:250] TRAIN Top shape for layer 3 'slice-label' 128 1 2 2 (512)
I0904 09:16:56.024879 14545 net.cpp:250] TRAIN Top shape for layer 3 'slice-label' 128 1 2 2 (512)
I0904 09:16:56.024883 14545 layer_factory.hpp:136] Creating layer 'foreground-label_slice-label_0_split' of type 'Split'
I0904 09:16:56.024885 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.024891 14545 net.cpp:182] Created Layer foreground-label_slice-label_0_split (4)
I0904 09:16:56.024894 14545 net.cpp:559] foreground-label_slice-label_0_split <- foreground-label
I0904 09:16:56.024897 14545 net.cpp:528] foreground-label_slice-label_0_split -> foreground-label_slice-label_0_split_0
I0904 09:16:56.024901 14545 net.cpp:528] foreground-label_slice-label_0_split -> foreground-label_slice-label_0_split_1
I0904 09:16:56.024904 14545 net.cpp:528] foreground-label_slice-label_0_split -> foreground-label_slice-label_0_split_2
I0904 09:16:56.024909 14545 net.cpp:528] foreground-label_slice-label_0_split -> foreground-label_slice-label_0_split_3
I0904 09:16:56.024947 14545 net.cpp:243] Setting up foreground-label_slice-label_0_split
I0904 09:16:56.024951 14545 net.cpp:250] TRAIN Top shape for layer 4 'foreground-label_slice-label_0_split' 128 1 2 2 (512)
I0904 09:16:56.024955 14545 net.cpp:250] TRAIN Top shape for layer 4 'foreground-label_slice-label_0_split' 128 1 2 2 (512)
I0904 09:16:56.024957 14545 net.cpp:250] TRAIN Top shape for layer 4 'foreground-label_slice-label_0_split' 128 1 2 2 (512)
I0904 09:16:56.024960 14545 net.cpp:250] TRAIN Top shape for layer 4 'foreground-label_slice-label_0_split' 128 1 2 2 (512)
I0904 09:16:56.024963 14545 layer_factory.hpp:136] Creating layer 'size-label_slice-label_2_split' of type 'Split'
I0904 09:16:56.024966 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.024971 14545 net.cpp:182] Created Layer size-label_slice-label_2_split (5)
I0904 09:16:56.024973 14545 net.cpp:559] size-label_slice-label_2_split <- size-label
I0904 09:16:56.024976 14545 net.cpp:528] size-label_slice-label_2_split -> size-label_slice-label_2_split_0
I0904 09:16:56.024981 14545 net.cpp:528] size-label_slice-label_2_split -> size-label_slice-label_2_split_1
I0904 09:16:56.025002 14545 net.cpp:243] Setting up size-label_slice-label_2_split
I0904 09:16:56.025004 14545 net.cpp:250] TRAIN Top shape for layer 5 'size-label_slice-label_2_split' 128 2 2 2 (1024)
I0904 09:16:56.025008 14545 net.cpp:250] TRAIN Top shape for layer 5 'size-label_slice-label_2_split' 128 2 2 2 (1024)
I0904 09:16:56.025012 14545 layer_factory.hpp:136] Creating layer 'obj-label_slice-label_3_split' of type 'Split'
I0904 09:16:56.025018 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.025024 14545 net.cpp:182] Created Layer obj-label_slice-label_3_split (6)
I0904 09:16:56.025027 14545 net.cpp:559] obj-label_slice-label_3_split <- obj-label
I0904 09:16:56.025032 14545 net.cpp:528] obj-label_slice-label_3_split -> obj-label_slice-label_3_split_0
I0904 09:16:56.025035 14545 net.cpp:528] obj-label_slice-label_3_split -> obj-label_slice-label_3_split_1
I0904 09:16:56.025039 14545 net.cpp:528] obj-label_slice-label_3_split -> obj-label_slice-label_3_split_2
I0904 09:16:56.025043 14545 net.cpp:528] obj-label_slice-label_3_split -> obj-label_slice-label_3_split_3
I0904 09:16:56.025077 14545 net.cpp:243] Setting up obj-label_slice-label_3_split
I0904 09:16:56.025080 14545 net.cpp:250] TRAIN Top shape for layer 6 'obj-label_slice-label_3_split' 128 1 2 2 (512)
I0904 09:16:56.025084 14545 net.cpp:250] TRAIN Top shape for layer 6 'obj-label_slice-label_3_split' 128 1 2 2 (512)
I0904 09:16:56.025087 14545 net.cpp:250] TRAIN Top shape for layer 6 'obj-label_slice-label_3_split' 128 1 2 2 (512)
I0904 09:16:56.025090 14545 net.cpp:250] TRAIN Top shape for layer 6 'obj-label_slice-label_3_split' 128 1 2 2 (512)
I0904 09:16:56.025094 14545 layer_factory.hpp:136] Creating layer 'coverage-block' of type 'Concat'
I0904 09:16:56.025096 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.025101 14545 net.cpp:182] Created Layer coverage-block (7)
I0904 09:16:56.025104 14545 net.cpp:559] coverage-block <- foreground-label_slice-label_0_split_0
I0904 09:16:56.025107 14545 net.cpp:559] coverage-block <- foreground-label_slice-label_0_split_1
I0904 09:16:56.025111 14545 net.cpp:559] coverage-block <- foreground-label_slice-label_0_split_2
I0904 09:16:56.025115 14545 net.cpp:559] coverage-block <- foreground-label_slice-label_0_split_3
I0904 09:16:56.025117 14545 net.cpp:528] coverage-block -> coverage-block
I0904 09:16:56.025132 14545 net.cpp:243] Setting up coverage-block
I0904 09:16:56.025136 14545 net.cpp:250] TRAIN Top shape for layer 7 'coverage-block' 128 4 2 2 (2048)
I0904 09:16:56.025141 14545 layer_factory.hpp:136] Creating layer 'size-block' of type 'Concat'
I0904 09:16:56.025142 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.025146 14545 net.cpp:182] Created Layer size-block (8)
I0904 09:16:56.025149 14545 net.cpp:559] size-block <- size-label_slice-label_2_split_0
I0904 09:16:56.025152 14545 net.cpp:559] size-block <- size-label_slice-label_2_split_1
I0904 09:16:56.025156 14545 net.cpp:528] size-block -> size-block
I0904 09:16:56.025171 14545 net.cpp:243] Setting up size-block
I0904 09:16:56.025173 14545 net.cpp:250] TRAIN Top shape for layer 8 'size-block' 128 4 2 2 (2048)
I0904 09:16:56.025177 14545 layer_factory.hpp:136] Creating layer 'size-block_size-block_0_split' of type 'Split'
I0904 09:16:56.025179 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.025183 14545 net.cpp:182] Created Layer size-block_size-block_0_split (9)
I0904 09:16:56.025185 14545 net.cpp:559] size-block_size-block_0_split <- size-block
I0904 09:16:56.025189 14545 net.cpp:528] size-block_size-block_0_split -> size-block_size-block_0_split_0
I0904 09:16:56.025192 14545 net.cpp:528] size-block_size-block_0_split -> size-block_size-block_0_split_1
I0904 09:16:56.025213 14545 net.cpp:243] Setting up size-block_size-block_0_split
I0904 09:16:56.025215 14545 net.cpp:250] TRAIN Top shape for layer 9 'size-block_size-block_0_split' 128 4 2 2 (2048)
I0904 09:16:56.025218 14545 net.cpp:250] TRAIN Top shape for layer 9 'size-block_size-block_0_split' 128 4 2 2 (2048)
I0904 09:16:56.025223 14545 layer_factory.hpp:136] Creating layer 'obj-block' of type 'Concat'
I0904 09:16:56.025224 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.025231 14545 net.cpp:182] Created Layer obj-block (10)
I0904 09:16:56.025234 14545 net.cpp:559] obj-block <- obj-label_slice-label_3_split_0
I0904 09:16:56.025239 14545 net.cpp:559] obj-block <- obj-label_slice-label_3_split_1
I0904 09:16:56.025243 14545 net.cpp:559] obj-block <- obj-label_slice-label_3_split_2
I0904 09:16:56.025246 14545 net.cpp:559] obj-block <- obj-label_slice-label_3_split_3
I0904 09:16:56.025249 14545 net.cpp:528] obj-block -> obj-block
I0904 09:16:56.025264 14545 net.cpp:243] Setting up obj-block
I0904 09:16:56.025267 14545 net.cpp:250] TRAIN Top shape for layer 10 'obj-block' 128 4 2 2 (2048)
I0904 09:16:56.025271 14545 layer_factory.hpp:136] Creating layer 'obj-block_obj-block_0_split' of type 'Split'
I0904 09:16:56.025274 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.025276 14545 net.cpp:182] Created Layer obj-block_obj-block_0_split (11)
I0904 09:16:56.025279 14545 net.cpp:559] obj-block_obj-block_0_split <- obj-block
I0904 09:16:56.025282 14545 net.cpp:528] obj-block_obj-block_0_split -> obj-block_obj-block_0_split_0
I0904 09:16:56.025286 14545 net.cpp:528] obj-block_obj-block_0_split -> obj-block_obj-block_0_split_1
I0904 09:16:56.025306 14545 net.cpp:243] Setting up obj-block_obj-block_0_split
I0904 09:16:56.025310 14545 net.cpp:250] TRAIN Top shape for layer 11 'obj-block_obj-block_0_split' 128 4 2 2 (2048)
I0904 09:16:56.025312 14545 net.cpp:250] TRAIN Top shape for layer 11 'obj-block_obj-block_0_split' 128 4 2 2 (2048)
I0904 09:16:56.025315 14545 layer_factory.hpp:136] Creating layer 'bb-label-norm' of type 'Eltwise'
I0904 09:16:56.025319 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.025326 14545 net.cpp:182] Created Layer bb-label-norm (12)
I0904 09:16:56.025328 14545 net.cpp:559] bb-label-norm <- bbox-label
I0904 09:16:56.025331 14545 net.cpp:559] bb-label-norm <- size-block_size-block_0_split_0
I0904 09:16:56.025334 14545 net.cpp:528] bb-label-norm -> bbox-label-norm
I0904 09:16:56.025347 14545 net.cpp:243] Setting up bb-label-norm
I0904 09:16:56.025351 14545 net.cpp:250] TRAIN Top shape for layer 12 'bb-label-norm' 128 4 2 2 (2048)
I0904 09:16:56.025354 14545 layer_factory.hpp:136] Creating layer 'bb-obj-norm' of type 'Eltwise'
I0904 09:16:56.025357 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.025360 14545 net.cpp:182] Created Layer bb-obj-norm (13)
I0904 09:16:56.025363 14545 net.cpp:559] bb-obj-norm <- bbox-label-norm
I0904 09:16:56.025367 14545 net.cpp:559] bb-obj-norm <- obj-block_obj-block_0_split_0
I0904 09:16:56.025370 14545 net.cpp:528] bb-obj-norm -> bbox-obj-label-norm
I0904 09:16:56.025382 14545 net.cpp:243] Setting up bb-obj-norm
I0904 09:16:56.025385 14545 net.cpp:250] TRAIN Top shape for layer 13 'bb-obj-norm' 128 4 2 2 (2048)
I0904 09:16:56.025388 14545 layer_factory.hpp:136] Creating layer 'conv1' of type 'Convolution'
I0904 09:16:56.025391 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.025404 14545 net.cpp:182] Created Layer conv1 (14)
I0904 09:16:56.025408 14545 net.cpp:559] conv1 <- transformed-data
I0904 09:16:56.025410 14545 net.cpp:528] conv1 -> conv1
I0904 09:16:56.361979 14545 net.cpp:243] Setting up conv1
I0904 09:16:56.362004 14545 net.cpp:250] TRAIN Top shape for layer 14 'conv1' 128 108 28 28 (10838016)
I0904 09:16:56.362067 14545 layer_factory.hpp:136] Creating layer 'relu1' of type 'ReLU'
I0904 09:16:56.362073 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.362082 14545 net.cpp:182] Created Layer relu1 (15)
I0904 09:16:56.362087 14545 net.cpp:559] relu1 <- conv1
I0904 09:16:56.362090 14545 net.cpp:511] relu1 -> conv1 (in-place)
I0904 09:16:56.362123 14545 net.cpp:243] Setting up relu1
I0904 09:16:56.362125 14545 net.cpp:250] TRAIN Top shape for layer 15 'relu1' 128 108 28 28 (10838016)
I0904 09:16:56.362129 14545 layer_factory.hpp:136] Creating layer 'pool1' of type 'Pooling'
I0904 09:16:56.362150 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.362169 14545 net.cpp:182] Created Layer pool1 (16)
I0904 09:16:56.362174 14545 net.cpp:559] pool1 <- conv1
I0904 09:16:56.362179 14545 net.cpp:528] pool1 -> pool1
I0904 09:16:56.362241 14545 net.cpp:243] Setting up pool1
I0904 09:16:56.362243 14545 net.cpp:250] TRAIN Top shape for layer 16 'pool1' 128 108 14 14 (2709504)
I0904 09:16:56.362248 14545 layer_factory.hpp:136] Creating layer 'pool1_pool1_0_split' of type 'Split'
I0904 09:16:56.362251 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.362256 14545 net.cpp:182] Created Layer pool1_pool1_0_split (17)
I0904 09:16:56.362258 14545 net.cpp:559] pool1_pool1_0_split <- pool1
I0904 09:16:56.362262 14545 net.cpp:528] pool1_pool1_0_split -> pool1_pool1_0_split_0
I0904 09:16:56.362265 14545 net.cpp:528] pool1_pool1_0_split -> pool1_pool1_0_split_1
I0904 09:16:56.362290 14545 net.cpp:243] Setting up pool1_pool1_0_split
I0904 09:16:56.362293 14545 net.cpp:250] TRAIN Top shape for layer 17 'pool1_pool1_0_split' 128 108 14 14 (2709504)
I0904 09:16:56.362298 14545 net.cpp:250] TRAIN Top shape for layer 17 'pool1_pool1_0_split' 128 108 14 14 (2709504)
I0904 09:16:56.362300 14545 layer_factory.hpp:136] Creating layer 'conv2' of type 'Convolution'
I0904 09:16:56.362303 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.362313 14545 net.cpp:182] Created Layer conv2 (18)
I0904 09:16:56.362315 14545 net.cpp:559] conv2 <- pool1_pool1_0_split_0
I0904 09:16:56.362318 14545 net.cpp:528] conv2 -> conv2
I0904 09:16:56.366840 14545 net.cpp:243] Setting up conv2
I0904 09:16:56.366852 14545 net.cpp:250] TRAIN Top shape for layer 18 'conv2' 128 108 14 14 (2709504)
I0904 09:16:56.366863 14545 layer_factory.hpp:136] Creating layer 'relu2' of type 'ReLU'
I0904 09:16:56.366866 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.366870 14545 net.cpp:182] Created Layer relu2 (19)
I0904 09:16:56.366873 14545 net.cpp:559] relu2 <- conv2
I0904 09:16:56.366876 14545 net.cpp:511] relu2 -> conv2 (in-place)
I0904 09:16:56.366883 14545 net.cpp:243] Setting up relu2
I0904 09:16:56.366885 14545 net.cpp:250] TRAIN Top shape for layer 19 'relu2' 128 108 14 14 (2709504)
I0904 09:16:56.366888 14545 layer_factory.hpp:136] Creating layer 'pool2' of type 'Pooling'
I0904 09:16:56.366891 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.366896 14545 net.cpp:182] Created Layer pool2 (20)
I0904 09:16:56.366899 14545 net.cpp:559] pool2 <- conv2
I0904 09:16:56.366901 14545 net.cpp:528] pool2 -> pool2
I0904 09:16:56.366940 14545 net.cpp:243] Setting up pool2
I0904 09:16:56.366943 14545 net.cpp:250] TRAIN Top shape for layer 20 'pool2' 128 108 7 7 (677376)
I0904 09:16:56.366947 14545 layer_factory.hpp:136] Creating layer 'pool3' of type 'Pooling'
I0904 09:16:56.366950 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.366953 14545 net.cpp:182] Created Layer pool3 (21)
I0904 09:16:56.366956 14545 net.cpp:559] pool3 <- pool1_pool1_0_split_1
I0904 09:16:56.366960 14545 net.cpp:528] pool3 -> pool3
I0904 09:16:56.366992 14545 net.cpp:243] Setting up pool3
I0904 09:16:56.366996 14545 net.cpp:250] TRAIN Top shape for layer 21 'pool3' 128 108 7 7 (677376)
I0904 09:16:56.366999 14545 layer_factory.hpp:136] Creating layer 'concat' of type 'Concat'
I0904 09:16:56.367002 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.367007 14545 net.cpp:182] Created Layer concat (22)
I0904 09:16:56.367010 14545 net.cpp:559] concat <- pool3
I0904 09:16:56.367013 14545 net.cpp:559] concat <- pool2
I0904 09:16:56.367017 14545 net.cpp:528] concat -> concat
I0904 09:16:56.367033 14545 net.cpp:243] Setting up concat
I0904 09:16:56.367041 14545 net.cpp:250] TRAIN Top shape for layer 22 'concat' 128 216 7 7 (1354752)
I0904 09:16:56.367053 14545 layer_factory.hpp:136] Creating layer 'conv3' of type 'Convolution'
I0904 09:16:56.367056 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.367064 14545 net.cpp:182] Created Layer conv3 (23)
I0904 09:16:56.367067 14545 net.cpp:559] conv3 <- concat
I0904 09:16:56.367070 14545 net.cpp:528] conv3 -> conv3
I0904 09:16:56.381214 14545 net.cpp:243] Setting up conv3
I0904 09:16:56.381233 14545 net.cpp:250] TRAIN Top shape for layer 23 'conv3' 128 100 1 1 (12800)
I0904 09:16:56.381247 14545 layer_factory.hpp:136] Creating layer 'relu3' of type 'ReLU'
I0904 09:16:56.381250 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.381256 14545 net.cpp:182] Created Layer relu3 (24)
I0904 09:16:56.381259 14545 net.cpp:559] relu3 <- conv3
I0904 09:16:56.381263 14545 net.cpp:511] relu3 -> conv3 (in-place)
I0904 09:16:56.381269 14545 net.cpp:243] Setting up relu3
I0904 09:16:56.381271 14545 net.cpp:250] TRAIN Top shape for layer 24 'relu3' 128 100 1 1 (12800)
I0904 09:16:56.381275 14545 layer_factory.hpp:136] Creating layer 'drop3' of type 'Dropout'
I0904 09:16:56.381278 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.381287 14545 net.cpp:182] Created Layer drop3 (25)
I0904 09:16:56.381290 14545 net.cpp:559] drop3 <- conv3
I0904 09:16:56.381294 14545 net.cpp:511] drop3 -> conv3 (in-place)
I0904 09:16:56.385807 14545 net.cpp:243] Setting up drop3
I0904 09:16:56.385817 14545 net.cpp:250] TRAIN Top shape for layer 25 'drop3' 128 100 1 1 (12800)
I0904 09:16:56.385823 14545 layer_factory.hpp:136] Creating layer 'conv4' of type 'Convolution'
I0904 09:16:56.385826 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.385835 14545 net.cpp:182] Created Layer conv4 (26)
I0904 09:16:56.385838 14545 net.cpp:559] conv4 <- conv3
I0904 09:16:56.385843 14545 net.cpp:528] conv4 -> conv4
I0904 09:16:56.386155 14545 net.cpp:243] Setting up conv4
I0904 09:16:56.386162 14545 net.cpp:250] TRAIN Top shape for layer 26 'conv4' 128 55 1 1 (7040)
I0904 09:16:56.386170 14545 layer_factory.hpp:136] Creating layer 'conv4_conv4_0_split' of type 'Split'
I0904 09:16:56.386173 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.386178 14545 net.cpp:182] Created Layer conv4_conv4_0_split (27)
I0904 09:16:56.386180 14545 net.cpp:559] conv4_conv4_0_split <- conv4
I0904 09:16:56.386184 14545 net.cpp:528] conv4_conv4_0_split -> conv4_conv4_0_split_0
I0904 09:16:56.386188 14545 net.cpp:528] conv4_conv4_0_split -> conv4_conv4_0_split_1
I0904 09:16:56.386219 14545 net.cpp:243] Setting up conv4_conv4_0_split
I0904 09:16:56.386222 14545 net.cpp:250] TRAIN Top shape for layer 27 'conv4_conv4_0_split' 128 55 1 1 (7040)
I0904 09:16:56.386226 14545 net.cpp:250] TRAIN Top shape for layer 27 'conv4_conv4_0_split' 128 55 1 1 (7040)
I0904 09:16:56.386230 14545 layer_factory.hpp:136] Creating layer 'cvg/classifier' of type 'Convolution'
I0904 09:16:56.386232 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.386240 14545 net.cpp:182] Created Layer cvg/classifier (28)
I0904 09:16:56.386243 14545 net.cpp:559] cvg/classifier <- conv4_conv4_0_split_0
I0904 09:16:56.386246 14545 net.cpp:528] cvg/classifier -> cvg/classifier
I0904 09:16:56.386479 14545 net.cpp:243] Setting up cvg/classifier
I0904 09:16:56.386485 14545 net.cpp:250] TRAIN Top shape for layer 28 'cvg/classifier' 128 1 2 2 (512)
I0904 09:16:56.386494 14545 layer_factory.hpp:136] Creating layer 'coverage/sig' of type 'Sigmoid'
I0904 09:16:56.386497 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.386503 14545 net.cpp:182] Created Layer coverage/sig (29)
I0904 09:16:56.386507 14545 net.cpp:559] coverage/sig <- cvg/classifier
I0904 09:16:56.386517 14545 net.cpp:528] coverage/sig -> coverage
I0904 09:16:56.386535 14545 net.cpp:243] Setting up coverage/sig
I0904 09:16:56.386545 14545 net.cpp:250] TRAIN Top shape for layer 29 'coverage/sig' 128 1 2 2 (512)
I0904 09:16:56.386549 14545 layer_factory.hpp:136] Creating layer 'bbox/regressor' of type 'Convolution'
I0904 09:16:56.386553 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.386564 14545 net.cpp:182] Created Layer bbox/regressor (30)
I0904 09:16:56.386567 14545 net.cpp:559] bbox/regressor <- conv4_conv4_0_split_1
I0904 09:16:56.386570 14545 net.cpp:528] bbox/regressor -> bboxes
I0904 09:16:56.387358 14545 net.cpp:243] Setting up bbox/regressor
I0904 09:16:56.387367 14545 net.cpp:250] TRAIN Top shape for layer 30 'bbox/regressor' 128 4 2 2 (2048)
I0904 09:16:56.387374 14545 layer_factory.hpp:136] Creating layer 'bbox_mask' of type 'Eltwise'
I0904 09:16:56.387377 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.387384 14545 net.cpp:182] Created Layer bbox_mask (31)
I0904 09:16:56.387387 14545 net.cpp:559] bbox_mask <- bboxes
I0904 09:16:56.387392 14545 net.cpp:559] bbox_mask <- coverage-block
I0904 09:16:56.387395 14545 net.cpp:528] bbox_mask -> bboxes-masked
I0904 09:16:56.387415 14545 net.cpp:243] Setting up bbox_mask
I0904 09:16:56.387418 14545 net.cpp:250] TRAIN Top shape for layer 31 'bbox_mask' 128 4 2 2 (2048)
I0904 09:16:56.387423 14545 layer_factory.hpp:136] Creating layer 'bbox-norm' of type 'Eltwise'
I0904 09:16:56.387425 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.387432 14545 net.cpp:182] Created Layer bbox-norm (32)
I0904 09:16:56.387435 14545 net.cpp:559] bbox-norm <- bboxes-masked
I0904 09:16:56.387439 14545 net.cpp:559] bbox-norm <- size-block_size-block_0_split_1
I0904 09:16:56.387444 14545 net.cpp:528] bbox-norm -> bboxes-masked-norm
I0904 09:16:56.387459 14545 net.cpp:243] Setting up bbox-norm
I0904 09:16:56.387462 14545 net.cpp:250] TRAIN Top shape for layer 32 'bbox-norm' 128 4 2 2 (2048)
I0904 09:16:56.387466 14545 layer_factory.hpp:136] Creating layer 'bbox-obj-norm' of type 'Eltwise'
I0904 09:16:56.387468 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.387473 14545 net.cpp:182] Created Layer bbox-obj-norm (33)
I0904 09:16:56.387476 14545 net.cpp:559] bbox-obj-norm <- bboxes-masked-norm
I0904 09:16:56.387480 14545 net.cpp:559] bbox-obj-norm <- obj-block_obj-block_0_split_1
I0904 09:16:56.387483 14545 net.cpp:528] bbox-obj-norm -> bboxes-obj-masked-norm
I0904 09:16:56.387501 14545 net.cpp:243] Setting up bbox-obj-norm
I0904 09:16:56.387504 14545 net.cpp:250] TRAIN Top shape for layer 33 'bbox-obj-norm' 128 4 2 2 (2048)
I0904 09:16:56.387508 14545 layer_factory.hpp:136] Creating layer 'bbox_loss' of type 'L1Loss'
I0904 09:16:56.387511 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.387521 14545 net.cpp:182] Created Layer bbox_loss (34)
I0904 09:16:56.387523 14545 net.cpp:559] bbox_loss <- bboxes-obj-masked-norm
I0904 09:16:56.387526 14545 net.cpp:559] bbox_loss <- bbox-obj-label-norm
I0904 09:16:56.387531 14545 net.cpp:528] bbox_loss -> loss_bbox
I0904 09:16:56.387591 14545 net.cpp:243] Setting up bbox_loss
I0904 09:16:56.387596 14545 net.cpp:250] TRAIN Top shape for layer 34 'bbox_loss' (1)
I0904 09:16:56.387600 14545 net.cpp:254]     with loss weight 2
I0904 09:16:56.387622 14545 layer_factory.hpp:136] Creating layer 'coverage_loss' of type 'EuclideanLoss'
I0904 09:16:56.387625 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.387632 14545 net.cpp:182] Created Layer coverage_loss (35)
I0904 09:16:56.387635 14545 net.cpp:559] coverage_loss <- coverage
I0904 09:16:56.387639 14545 net.cpp:559] coverage_loss <- coverage-label
I0904 09:16:56.387642 14545 net.cpp:528] coverage_loss -> loss_coverage
I0904 09:16:56.387693 14545 net.cpp:243] Setting up coverage_loss
I0904 09:16:56.387698 14545 net.cpp:250] TRAIN Top shape for layer 35 'coverage_loss' (1)
I0904 09:16:56.387712 14545 net.cpp:254]     with loss weight 1
I0904 09:16:56.387717 14545 net.cpp:321] coverage_loss needs backward computation.
I0904 09:16:56.387720 14545 net.cpp:321] bbox_loss needs backward computation.
I0904 09:16:56.387723 14545 net.cpp:321] bbox-obj-norm needs backward computation.
I0904 09:16:56.387727 14545 net.cpp:321] bbox-norm needs backward computation.
I0904 09:16:56.387729 14545 net.cpp:321] bbox_mask needs backward computation.
I0904 09:16:56.387732 14545 net.cpp:321] bbox/regressor needs backward computation.
I0904 09:16:56.387735 14545 net.cpp:321] coverage/sig needs backward computation.
I0904 09:16:56.387737 14545 net.cpp:321] cvg/classifier needs backward computation.
I0904 09:16:56.387740 14545 net.cpp:321] conv4_conv4_0_split needs backward computation.
I0904 09:16:56.387743 14545 net.cpp:321] conv4 needs backward computation.
I0904 09:16:56.387745 14545 net.cpp:321] drop3 needs backward computation.
I0904 09:16:56.387748 14545 net.cpp:321] relu3 needs backward computation.
I0904 09:16:56.387750 14545 net.cpp:321] conv3 needs backward computation.
I0904 09:16:56.387753 14545 net.cpp:321] concat needs backward computation.
I0904 09:16:56.387756 14545 net.cpp:321] pool3 needs backward computation.
I0904 09:16:56.387758 14545 net.cpp:321] pool2 needs backward computation.
I0904 09:16:56.387763 14545 net.cpp:321] relu2 needs backward computation.
I0904 09:16:56.387765 14545 net.cpp:321] conv2 needs backward computation.
I0904 09:16:56.387768 14545 net.cpp:321] pool1_pool1_0_split needs backward computation.
I0904 09:16:56.387770 14545 net.cpp:321] pool1 needs backward computation.
I0904 09:16:56.387773 14545 net.cpp:321] relu1 needs backward computation.
I0904 09:16:56.387775 14545 net.cpp:321] conv1 needs backward computation.
I0904 09:16:56.387778 14545 net.cpp:323] bb-obj-norm does not need backward computation.
I0904 09:16:56.387783 14545 net.cpp:323] bb-label-norm does not need backward computation.
I0904 09:16:56.387786 14545 net.cpp:323] obj-block_obj-block_0_split does not need backward computation.
I0904 09:16:56.387789 14545 net.cpp:323] obj-block does not need backward computation.
I0904 09:16:56.387794 14545 net.cpp:323] size-block_size-block_0_split does not need backward computation.
I0904 09:16:56.387797 14545 net.cpp:323] size-block does not need backward computation.
I0904 09:16:56.387800 14545 net.cpp:323] coverage-block does not need backward computation.
I0904 09:16:56.387806 14545 net.cpp:323] obj-label_slice-label_3_split does not need backward computation.
I0904 09:16:56.387810 14545 net.cpp:323] size-label_slice-label_2_split does not need backward computation.
I0904 09:16:56.387814 14545 net.cpp:323] foreground-label_slice-label_0_split does not need backward computation.
I0904 09:16:56.387817 14545 net.cpp:323] slice-label does not need backward computation.
I0904 09:16:56.387820 14545 net.cpp:323] data_transform does not need backward computation.
I0904 09:16:56.387825 14545 net.cpp:323] train-label does not need backward computation.
I0904 09:16:56.387827 14545 net.cpp:323] train-data does not need backward computation.
I0904 09:16:56.387830 14545 net.cpp:365] This network produces output loss_bbox
I0904 09:16:56.387832 14545 net.cpp:365] This network produces output loss_coverage
I0904 09:16:56.387859 14545 net.cpp:387] Top memory (TRAIN) required for data: 380224016 diff: 380224016
I0904 09:16:56.387863 14545 net.cpp:390] Bottom memory (TRAIN) required for data: 380224000 diff: 380224000
I0904 09:16:56.387866 14545 net.cpp:393] Shared (in-place) memory (TRAIN) by data: 54292480 diff: 54292480
I0904 09:16:56.387867 14545 net.cpp:396] Parameters memory (TRAIN) required for data: 5460312 diff: 5460312
I0904 09:16:56.387869 14545 net.cpp:399] Parameters shared memory (TRAIN) by data: 0 diff: 0
I0904 09:16:56.387872 14545 net.cpp:405] Network initialization done.
I0904 09:16:56.388172 14545 solver.cpp:174] Creating test net (#0) specified by net file: /home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170901-131346-f68c/train_val.prototxt
I0904 09:16:56.388212 14545 net.cpp:441] The NetState phase (1) differed from the phase (0) specified by a rule in layer train-data
I0904 09:16:56.388216 14545 net.cpp:441] The NetState phase (1) differed from the phase (0) specified by a rule in layer train-label
I0904 09:16:56.388219 14545 net.cpp:441] The NetState phase (1) differed from the phase (0) specified by a rule in layer data_transform
I0904 09:16:56.388247 14545 net.cpp:70] Initializing net from parameters: 
state {
  phase: TEST
}
layer {
  name: "val-data"
  type: "Data"
  top: "data"
  include {
    phase: TEST
  }
  data_param {
    source: "/home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170807-135507-149f/val_db/features"
    batch_size: 32
    backend: LMDB
  }
}
layer {
  name: "val-label"
  type: "Data"
  top: "label"
  include {
    phase: TEST
  }
  data_param {
    source: "/home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170807-135507-149f/val_db/labels"
    batch_size: 32
    backend: LMDB
  }
}
layer {
  name: "data_transform"
  type: "DetectNetTransformation"
  bottom: "data"
  bottom: "label"
  top: "transformed-data"
  top: "transformed-label"
  include {
    phase: TEST
  }
  transform_param {
    mean_value: 127
  }
  detectnet_groundtruth_param {
    stride: 16
    scale_cvg: 0.4
    gridbox_type: GRIDBOX_MIN
    min_cvg_len: 20
    coverage_type: RECTANGULAR
    image_size_x: 32
    image_size_y: 32
    obj_norm: true
    crop_bboxes: false
  }
}
layer {
  name: "slice-label"
  type: "Slice"
  bottom: "transformed-label"
  top: "foreground-label"
  top: "bbox-label"
  top: "size-label"
  top: "obj-label"
  top: "coverage-label"
  slice_param {
    slice_point: 1
    slice_point: 5
    slice_point: 7
    slice_point: 8
  }
}
layer {
  name: "coverage-block"
  type: "Concat"
  bottom: "foreground-label"
  bottom: "foreground-label"
  bottom: "foreground-label"
  bottom: "foreground-label"
  top: "coverage-block"
}
layer {
  name: "size-block"
  type: "Concat"
  bottom: "size-label"
  bottom: "size-label"
  top: "size-block"
}
layer {
  name: "obj-block"
  type: "Concat"
  bottom: "obj-label"
  bottom: "obj-label"
  bottom: "obj-label"
  bottom: "obj-label"
  top: "obj-block"
}
layer {
  name: "bb-label-norm"
  type: "Eltwise"
  bottom: "bbox-label"
  bottom: "size-block"
  top: "bbox-label-norm"
  eltwise_param {
    operation: PROD
  }
}
layer {
  name: "bb-obj-norm"
  type: "Eltwise"
  bottom: "bbox-label-norm"
  bottom: "obj-block"
  top: "bbox-obj-label-norm"
  eltwise_param {
    operation: PROD
  }
}
layer {
  name: "conv1"
  type: "Convolution"
  bottom: "transformed-data"
  top: "conv1"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 108
    kernel_size: 5
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "relu1"
  type: "ReLU"
  bottom: "conv1"
  top: "conv1"
}
layer {
  name: "pool1"
  type: "Pooling"
  bottom: "conv1"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "conv2"
  type: "Convolution"
  bottom: "pool1"
  top: "conv2"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 108
    pad: 2
    kernel_size: 5
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0.1
    }
  }
}
layer {
  name: "relu2"
  type: "ReLU"
  bottom: "conv2"
  top: "conv2"
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "conv2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "pool3"
  type: "Pooling"
  bottom: "pool1"
  top: "pool3"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layer {
  name: "concat"
  type: "Concat"
  bottom: "pool3"
  bottom: "pool2"
  top: "concat"
  concat_param {
    axis: 1
  }
}
layer {
  name: "conv3"
  type: "Convolution"
  bottom: "concat"
  top: "conv3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 100
    pad: 0
    kernel_size: 7
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0.1
    }
  }
}
layer {
  name: "relu3"
  type: "ReLU"
  bottom: "conv3"
  top: "conv3"
}
layer {
  name: "drop3"
  type: "Dropout"
  bottom: "conv3"
  top: "conv3"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layer {
  name: "conv4"
  type: "Convolution"
  bottom: "conv3"
  top: "conv4"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 55
    pad: 0
    kernel_size: 1
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0.1
    }
  }
}
layer {
  name: "cvg/classifier"
  type: "Convolution"
  bottom: "conv4"
  top: "cvg/classifier"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 1
    pad: 2
    kernel_size: 2
    stride: 2
    weight_filler {
      type: "xavier"
      std: 0.03
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "coverage/sig"
  type: "Sigmoid"
  bottom: "cvg/classifier"
  top: "coverage"
}
layer {
  name: "bbox/regressor"
  type: "Convolution"
  bottom: "conv4"
  top: "bboxes"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  param {
    lr_mult: 2
    decay_mult: 0
  }
  convolution_param {
    num_output: 4
    pad: 1
    kernel_size: 2
    stride: 1
    weight_filler {
      type: "xavier"
      std: 0.03
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layer {
  name: "bbox_mask"
  type: "Eltwise"
  bottom: "bboxes"
  bottom: "coverage-block"
  top: "bboxes-masked"
  eltwise_param {
    operation: PROD
  }
}
layer {
  name: "bbox-norm"
  type: "Eltwise"
  bottom: "bboxes-masked"
  bottom: "size-block"
  top: "bboxes-masked-norm"
  eltwise_param {
    operation: PROD
  }
}
layer {
  name: "bbox-obj-norm"
  type: "Eltwise"
  bottom: "bboxes-masked-norm"
  bottom: "obj-block"
  top: "bboxes-obj-masked-norm"
  eltwise_param {
    operation: PROD
  }
}
layer {
  name: "bbox_loss"
  type: "L1Loss"
  bottom: "bboxes-obj-masked-norm"
  bottom: "bbox-obj-label-norm"
  top: "loss_bbox"
  loss_weight: 2
}
layer {
  name: "coverage_loss"
  type: "EuclideanLoss"
  bottom: "coverage"
  bottom: "coverage-label"
  top: "loss_coverage"
}
layer {
  name: "cluster"
  type: "Python"
  bottom: "coverage"
  bottom: "bboxes"
  top: "bbox-list"
  include {
    phase: TEST
  }
  python_param {
    module: "caffe.layers.detectnet.clustering"
    layer: "ClusterDetections"
    param_str: "520, 520, 16, 0.3, 1, 0.001, 22, 1"
  }
}
layer {
  name: "cluster_gt"
  type: "Python"
  bottom: "coverage-label"
  bottom: "bbox-label"
  top: "bbox-list-label"
  include {
    phase: TEST
  }
  python_param {
    module: "caffe.layers.detectnet.clustering"
    layer: "ClusterGroundtruth"
    param_str: "520, 520, 16, 1"
  }
}
layer {
  name: "score"
  type: "Python"
  bottom: "bbox-list-label"
  bottom: "bbox-list"
  top: "bbox-list-scored"
  include {
    phase: TEST
  }
  python_param {
    module: "caffe.layers.detectnet.mean_ap"
    layer: "ScoreDetections"
  }
}
layer {
  name: "mAP"
  type: "Python"
  bottom: "bbox-list-scored"
  top: "mAP"
  top: "precision"
  top: "recall"
  include {
    phase: TEST
  }
  python_param {
    module: "caffe.layers.detectnet.mean_ap"
    layer: "mAP"
    param_str: "520, 520, 16"
  }
}
I0904 09:16:56.388509 14545 net.cpp:102] Using FLOAT as default forward math type
I0904 09:16:56.388514 14545 net.cpp:108] Using FLOAT as default backward math type
I0904 09:16:56.388517 14545 layer_factory.hpp:136] Creating layer 'val-data' of type 'Data'
I0904 09:16:56.388520 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.388535 14545 net.cpp:182] Created Layer val-data (0)
I0904 09:16:56.388541 14545 net.cpp:528] val-data -> data
I0904 09:16:56.388550 14545 data_reader.cpp:52] Data Reader threads: 1, out queues: 1, depth: 32
I0904 09:16:56.388559 14545 internal_thread.cpp:19] Starting 1 internal thread(s) on device 0
I0904 09:16:56.389094 14558 db_lmdb.cpp:24] Opened lmdb /home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170807-135507-149f/val_db/features
I0904 09:16:56.390378 14545 data_layer.cpp:187] (0) ReshapePrefetch 32, 3, 384, 384
I0904 09:16:56.390399 14545 data_layer.cpp:211] (0) Output data size: 32, 3, 384, 384
I0904 09:16:56.390405 14545 internal_thread.cpp:19] Starting 1 internal thread(s) on device 0
I0904 09:16:56.390424 14545 net.cpp:243] Setting up val-data
I0904 09:16:56.390427 14545 net.cpp:250] TEST Top shape for layer 0 'val-data' 32 3 384 384 (14155776)
I0904 09:16:56.390434 14545 layer_factory.hpp:136] Creating layer 'val-label' of type 'Data'
I0904 09:16:56.390436 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.390449 14545 net.cpp:182] Created Layer val-label (1)
I0904 09:16:56.390451 14545 net.cpp:528] val-label -> label
I0904 09:16:56.390457 14545 data_reader.cpp:52] Data Reader threads: 1, out queues: 1, depth: 32
I0904 09:16:56.390465 14545 internal_thread.cpp:19] Starting 1 internal thread(s) on device 0
I0904 09:16:56.390805 14559 data_layer.cpp:101] (0) Parser threads: 1
I0904 09:16:56.390818 14559 data_layer.cpp:103] (0) Transformer threads: 1
I0904 09:16:56.391212 14560 db_lmdb.cpp:24] Opened lmdb /home/bidski/Projects/nvidia/DIGITS/digits/jobs/20170807-135507-149f/val_db/labels
I0904 09:16:56.391294 14545 data_layer.cpp:187] (0) ReshapePrefetch 32, 1, 2, 16
I0904 09:16:56.391319 14545 data_layer.cpp:211] (0) Output data size: 32, 1, 2, 16
I0904 09:16:56.391325 14545 internal_thread.cpp:19] Starting 1 internal thread(s) on device 0
I0904 09:16:56.391495 14545 net.cpp:243] Setting up val-label
I0904 09:16:56.391522 14545 net.cpp:250] TEST Top shape for layer 1 'val-label' 32 1 2 16 (1024)
I0904 09:16:56.391530 14545 layer_factory.hpp:136] Creating layer 'data_transform' of type 'DetectNetTransformation'
I0904 09:16:56.391535 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.391554 14545 net.cpp:182] Created Layer data_transform (2)
I0904 09:16:56.391559 14545 net.cpp:559] data_transform <- data
I0904 09:16:56.391566 14545 net.cpp:559] data_transform <- label
I0904 09:16:56.391571 14545 net.cpp:528] data_transform -> transformed-data
I0904 09:16:56.391578 14545 net.cpp:528] data_transform -> transformed-label
I0904 09:16:56.392146 14545 net.cpp:243] Setting up data_transform
I0904 09:16:56.392156 14545 net.cpp:250] TEST Top shape for layer 2 'data_transform' 32 3 32 32 (98304)
I0904 09:16:56.392163 14545 net.cpp:250] TEST Top shape for layer 2 'data_transform' 32 9 2 2 (1152)
I0904 09:16:56.392166 14545 layer_factory.hpp:136] Creating layer 'slice-label' of type 'Slice'
I0904 09:16:56.392169 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.392175 14545 net.cpp:182] Created Layer slice-label (3)
I0904 09:16:56.392179 14545 net.cpp:559] slice-label <- transformed-label
I0904 09:16:56.392182 14545 net.cpp:528] slice-label -> foreground-label
I0904 09:16:56.392181 14561 data_layer.cpp:101] (0) Parser threads: 1
I0904 09:16:56.392195 14561 data_layer.cpp:103] (0) Transformer threads: 1
I0904 09:16:56.392187 14545 net.cpp:528] slice-label -> bbox-label
I0904 09:16:56.392206 14545 net.cpp:528] slice-label -> size-label
I0904 09:16:56.392210 14545 net.cpp:528] slice-label -> obj-label
I0904 09:16:56.392213 14545 net.cpp:528] slice-label -> coverage-label
I0904 09:16:56.392354 14545 net.cpp:243] Setting up slice-label
I0904 09:16:56.392364 14545 net.cpp:250] TEST Top shape for layer 3 'slice-label' 32 1 2 2 (128)
I0904 09:16:56.392370 14545 net.cpp:250] TEST Top shape for layer 3 'slice-label' 32 4 2 2 (512)
I0904 09:16:56.392379 14545 net.cpp:250] TEST Top shape for layer 3 'slice-label' 32 2 2 2 (256)
I0904 09:16:56.392390 14545 net.cpp:250] TEST Top shape for layer 3 'slice-label' 32 1 2 2 (128)
I0904 09:16:56.392392 14545 net.cpp:250] TEST Top shape for layer 3 'slice-label' 32 1 2 2 (128)
I0904 09:16:56.392396 14545 layer_factory.hpp:136] Creating layer 'foreground-label_slice-label_0_split' of type 'Split'
I0904 09:16:56.392400 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.392405 14545 net.cpp:182] Created Layer foreground-label_slice-label_0_split (4)
I0904 09:16:56.392408 14545 net.cpp:559] foreground-label_slice-label_0_split <- foreground-label
I0904 09:16:56.392412 14545 net.cpp:528] foreground-label_slice-label_0_split -> foreground-label_slice-label_0_split_0
I0904 09:16:56.392417 14545 net.cpp:528] foreground-label_slice-label_0_split -> foreground-label_slice-label_0_split_1
I0904 09:16:56.392421 14545 net.cpp:528] foreground-label_slice-label_0_split -> foreground-label_slice-label_0_split_2
I0904 09:16:56.392424 14545 net.cpp:528] foreground-label_slice-label_0_split -> foreground-label_slice-label_0_split_3
I0904 09:16:56.402740 14545 net.cpp:243] Setting up foreground-label_slice-label_0_split
I0904 09:16:56.402822 14545 net.cpp:250] TEST Top shape for layer 4 'foreground-label_slice-label_0_split' 32 1 2 2 (128)
I0904 09:16:56.402840 14545 net.cpp:250] TEST Top shape for layer 4 'foreground-label_slice-label_0_split' 32 1 2 2 (128)
I0904 09:16:56.402844 14545 net.cpp:250] TEST Top shape for layer 4 'foreground-label_slice-label_0_split' 32 1 2 2 (128)
I0904 09:16:56.402860 14545 net.cpp:250] TEST Top shape for layer 4 'foreground-label_slice-label_0_split' 32 1 2 2 (128)
I0904 09:16:56.402866 14545 layer_factory.hpp:136] Creating layer 'bbox-label_slice-label_1_split' of type 'Split'
I0904 09:16:56.402873 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.402884 14545 net.cpp:182] Created Layer bbox-label_slice-label_1_split (5)
I0904 09:16:56.402889 14545 net.cpp:559] bbox-label_slice-label_1_split <- bbox-label
I0904 09:16:56.402896 14545 net.cpp:528] bbox-label_slice-label_1_split -> bbox-label_slice-label_1_split_0
I0904 09:16:56.402904 14545 net.cpp:528] bbox-label_slice-label_1_split -> bbox-label_slice-label_1_split_1
I0904 09:16:56.402966 14545 net.cpp:243] Setting up bbox-label_slice-label_1_split
I0904 09:16:56.402971 14545 net.cpp:250] TEST Top shape for layer 5 'bbox-label_slice-label_1_split' 32 4 2 2 (512)
I0904 09:16:56.402974 14545 net.cpp:250] TEST Top shape for layer 5 'bbox-label_slice-label_1_split' 32 4 2 2 (512)
I0904 09:16:56.402977 14545 layer_factory.hpp:136] Creating layer 'size-label_slice-label_2_split' of type 'Split'
I0904 09:16:56.402981 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.402986 14545 net.cpp:182] Created Layer size-label_slice-label_2_split (6)
I0904 09:16:56.402989 14545 net.cpp:559] size-label_slice-label_2_split <- size-label
I0904 09:16:56.402993 14545 net.cpp:528] size-label_slice-label_2_split -> size-label_slice-label_2_split_0
I0904 09:16:56.402998 14545 net.cpp:528] size-label_slice-label_2_split -> size-label_slice-label_2_split_1
I0904 09:16:56.403029 14545 net.cpp:243] Setting up size-label_slice-label_2_split
I0904 09:16:56.403033 14545 net.cpp:250] TEST Top shape for layer 6 'size-label_slice-label_2_split' 32 2 2 2 (256)
I0904 09:16:56.403038 14545 net.cpp:250] TEST Top shape for layer 6 'size-label_slice-label_2_split' 32 2 2 2 (256)
I0904 09:16:56.403040 14545 layer_factory.hpp:136] Creating layer 'obj-label_slice-label_3_split' of type 'Split'
I0904 09:16:56.403043 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.403048 14545 net.cpp:182] Created Layer obj-label_slice-label_3_split (7)
I0904 09:16:56.403050 14545 net.cpp:559] obj-label_slice-label_3_split <- obj-label
I0904 09:16:56.403069 14545 net.cpp:528] obj-label_slice-label_3_split -> obj-label_slice-label_3_split_0
I0904 09:16:56.403087 14545 net.cpp:528] obj-label_slice-label_3_split -> obj-label_slice-label_3_split_1
I0904 09:16:56.403092 14545 net.cpp:528] obj-label_slice-label_3_split -> obj-label_slice-label_3_split_2
I0904 09:16:56.403097 14545 net.cpp:528] obj-label_slice-label_3_split -> obj-label_slice-label_3_split_3
I0904 09:16:56.403148 14545 net.cpp:243] Setting up obj-label_slice-label_3_split
I0904 09:16:56.403153 14545 net.cpp:250] TEST Top shape for layer 7 'obj-label_slice-label_3_split' 32 1 2 2 (128)
I0904 09:16:56.403156 14545 net.cpp:250] TEST Top shape for layer 7 'obj-label_slice-label_3_split' 32 1 2 2 (128)
I0904 09:16:56.403159 14545 net.cpp:250] TEST Top shape for layer 7 'obj-label_slice-label_3_split' 32 1 2 2 (128)
I0904 09:16:56.403163 14545 net.cpp:250] TEST Top shape for layer 7 'obj-label_slice-label_3_split' 32 1 2 2 (128)
I0904 09:16:56.403167 14545 layer_factory.hpp:136] Creating layer 'coverage-label_slice-label_4_split' of type 'Split'
I0904 09:16:56.403169 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.403174 14545 net.cpp:182] Created Layer coverage-label_slice-label_4_split (8)
I0904 09:16:56.403178 14545 net.cpp:559] coverage-label_slice-label_4_split <- coverage-label
I0904 09:16:56.403183 14545 net.cpp:528] coverage-label_slice-label_4_split -> coverage-label_slice-label_4_split_0
I0904 09:16:56.403187 14545 net.cpp:528] coverage-label_slice-label_4_split -> coverage-label_slice-label_4_split_1
I0904 09:16:56.403215 14545 net.cpp:243] Setting up coverage-label_slice-label_4_split
I0904 09:16:56.403219 14545 net.cpp:250] TEST Top shape for layer 8 'coverage-label_slice-label_4_split' 32 1 2 2 (128)
I0904 09:16:56.403223 14545 net.cpp:250] TEST Top shape for layer 8 'coverage-label_slice-label_4_split' 32 1 2 2 (128)
I0904 09:16:56.403228 14545 layer_factory.hpp:136] Creating layer 'coverage-block' of type 'Concat'
I0904 09:16:56.403231 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.403237 14545 net.cpp:182] Created Layer coverage-block (9)
I0904 09:16:56.403241 14545 net.cpp:559] coverage-block <- foreground-label_slice-label_0_split_0
I0904 09:16:56.403245 14545 net.cpp:559] coverage-block <- foreground-label_slice-label_0_split_1
I0904 09:16:56.403249 14545 net.cpp:559] coverage-block <- foreground-label_slice-label_0_split_2
I0904 09:16:56.403252 14545 net.cpp:559] coverage-block <- foreground-label_slice-label_0_split_3
I0904 09:16:56.403256 14545 net.cpp:528] coverage-block -> coverage-block
I0904 09:16:56.403288 14545 net.cpp:243] Setting up coverage-block
I0904 09:16:56.403295 14545 net.cpp:250] TEST Top shape for layer 9 'coverage-block' 32 4 2 2 (512)
I0904 09:16:56.403298 14545 layer_factory.hpp:136] Creating layer 'size-block' of type 'Concat'
I0904 09:16:56.403301 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.403306 14545 net.cpp:182] Created Layer size-block (10)
I0904 09:16:56.403309 14545 net.cpp:559] size-block <- size-label_slice-label_2_split_0
I0904 09:16:56.403313 14545 net.cpp:559] size-block <- size-label_slice-label_2_split_1
I0904 09:16:56.403317 14545 net.cpp:528] size-block -> size-block
I0904 09:16:56.403337 14545 net.cpp:243] Setting up size-block
I0904 09:16:56.403342 14545 net.cpp:250] TEST Top shape for layer 10 'size-block' 32 4 2 2 (512)
I0904 09:16:56.403347 14545 layer_factory.hpp:136] Creating layer 'size-block_size-block_0_split' of type 'Split'
I0904 09:16:56.403349 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.403353 14545 net.cpp:182] Created Layer size-block_size-block_0_split (11)
I0904 09:16:56.403357 14545 net.cpp:559] size-block_size-block_0_split <- size-block
I0904 09:16:56.403360 14545 net.cpp:528] size-block_size-block_0_split -> size-block_size-block_0_split_0
I0904 09:16:56.403373 14545 net.cpp:528] size-block_size-block_0_split -> size-block_size-block_0_split_1
I0904 09:16:56.403411 14545 net.cpp:243] Setting up size-block_size-block_0_split
I0904 09:16:56.403415 14545 net.cpp:250] TEST Top shape for layer 11 'size-block_size-block_0_split' 32 4 2 2 (512)
I0904 09:16:56.403420 14545 net.cpp:250] TEST Top shape for layer 11 'size-block_size-block_0_split' 32 4 2 2 (512)
I0904 09:16:56.403424 14545 layer_factory.hpp:136] Creating layer 'obj-block' of type 'Concat'
I0904 09:16:56.403426 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.403432 14545 net.cpp:182] Created Layer obj-block (12)
I0904 09:16:56.403435 14545 net.cpp:559] obj-block <- obj-label_slice-label_3_split_0
I0904 09:16:56.403439 14545 net.cpp:559] obj-block <- obj-label_slice-label_3_split_1
I0904 09:16:56.403442 14545 net.cpp:559] obj-block <- obj-label_slice-label_3_split_2
I0904 09:16:56.403445 14545 net.cpp:559] obj-block <- obj-label_slice-label_3_split_3
I0904 09:16:56.403448 14545 net.cpp:528] obj-block -> obj-block
I0904 09:16:56.403468 14545 net.cpp:243] Setting up obj-block
I0904 09:16:56.403473 14545 net.cpp:250] TEST Top shape for layer 12 'obj-block' 32 4 2 2 (512)
I0904 09:16:56.403477 14545 layer_factory.hpp:136] Creating layer 'obj-block_obj-block_0_split' of type 'Split'
I0904 09:16:56.403481 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.403486 14545 net.cpp:182] Created Layer obj-block_obj-block_0_split (13)
I0904 09:16:56.403488 14545 net.cpp:559] obj-block_obj-block_0_split <- obj-block
I0904 09:16:56.403492 14545 net.cpp:528] obj-block_obj-block_0_split -> obj-block_obj-block_0_split_0
I0904 09:16:56.403496 14545 net.cpp:528] obj-block_obj-block_0_split -> obj-block_obj-block_0_split_1
I0904 09:16:56.403525 14545 net.cpp:243] Setting up obj-block_obj-block_0_split
I0904 09:16:56.403529 14545 net.cpp:250] TEST Top shape for layer 13 'obj-block_obj-block_0_split' 32 4 2 2 (512)
I0904 09:16:56.403533 14545 net.cpp:250] TEST Top shape for layer 13 'obj-block_obj-block_0_split' 32 4 2 2 (512)
I0904 09:16:56.403538 14545 layer_factory.hpp:136] Creating layer 'bb-label-norm' of type 'Eltwise'
I0904 09:16:56.403542 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.403549 14545 net.cpp:182] Created Layer bb-label-norm (14)
I0904 09:16:56.403553 14545 net.cpp:559] bb-label-norm <- bbox-label_slice-label_1_split_0
I0904 09:16:56.403558 14545 net.cpp:559] bb-label-norm <- size-block_size-block_0_split_0
I0904 09:16:56.403563 14545 net.cpp:528] bb-label-norm -> bbox-label-norm
I0904 09:16:56.403581 14545 net.cpp:243] Setting up bb-label-norm
I0904 09:16:56.403586 14545 net.cpp:250] TEST Top shape for layer 14 'bb-label-norm' 32 4 2 2 (512)
I0904 09:16:56.403590 14545 layer_factory.hpp:136] Creating layer 'bb-obj-norm' of type 'Eltwise'
I0904 09:16:56.403592 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.403599 14545 net.cpp:182] Created Layer bb-obj-norm (15)
I0904 09:16:56.403601 14545 net.cpp:559] bb-obj-norm <- bbox-label-norm
I0904 09:16:56.403604 14545 net.cpp:559] bb-obj-norm <- obj-block_obj-block_0_split_0
I0904 09:16:56.403609 14545 net.cpp:528] bb-obj-norm -> bbox-obj-label-norm
I0904 09:16:56.403626 14545 net.cpp:243] Setting up bb-obj-norm
I0904 09:16:56.403630 14545 net.cpp:250] TEST Top shape for layer 15 'bb-obj-norm' 32 4 2 2 (512)
I0904 09:16:56.403635 14545 layer_factory.hpp:136] Creating layer 'conv1' of type 'Convolution'
I0904 09:16:56.403637 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.403656 14545 net.cpp:182] Created Layer conv1 (16)
I0904 09:16:56.403661 14545 net.cpp:559] conv1 <- transformed-data
I0904 09:16:56.403666 14545 net.cpp:528] conv1 -> conv1
I0904 09:16:56.404234 14545 net.cpp:243] Setting up conv1
I0904 09:16:56.404269 14545 net.cpp:250] TEST Top shape for layer 16 'conv1' 32 108 28 28 (2709504)
I0904 09:16:56.404309 14545 layer_factory.hpp:136] Creating layer 'relu1' of type 'ReLU'
I0904 09:16:56.404330 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.404340 14545 net.cpp:182] Created Layer relu1 (17)
I0904 09:16:56.404345 14545 net.cpp:559] relu1 <- conv1
I0904 09:16:56.404350 14545 net.cpp:511] relu1 -> conv1 (in-place)
I0904 09:16:56.404359 14545 net.cpp:243] Setting up relu1
I0904 09:16:56.404362 14545 net.cpp:250] TEST Top shape for layer 17 'relu1' 32 108 28 28 (2709504)
I0904 09:16:56.404366 14545 layer_factory.hpp:136] Creating layer 'pool1' of type 'Pooling'
I0904 09:16:56.404369 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.404479 14545 net.cpp:182] Created Layer pool1 (18)
I0904 09:16:56.404486 14545 net.cpp:559] pool1 <- conv1
I0904 09:16:56.404492 14545 net.cpp:528] pool1 -> pool1
I0904 09:16:56.404603 14545 net.cpp:243] Setting up pool1
I0904 09:16:56.404608 14545 net.cpp:250] TEST Top shape for layer 18 'pool1' 32 108 14 14 (677376)
I0904 09:16:56.404613 14545 layer_factory.hpp:136] Creating layer 'pool1_pool1_0_split' of type 'Split'
I0904 09:16:56.404616 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.404621 14545 net.cpp:182] Created Layer pool1_pool1_0_split (19)
I0904 09:16:56.404624 14545 net.cpp:559] pool1_pool1_0_split <- pool1
I0904 09:16:56.404628 14545 net.cpp:528] pool1_pool1_0_split -> pool1_pool1_0_split_0
I0904 09:16:56.404633 14545 net.cpp:528] pool1_pool1_0_split -> pool1_pool1_0_split_1
I0904 09:16:56.404665 14545 net.cpp:243] Setting up pool1_pool1_0_split
I0904 09:16:56.404670 14545 net.cpp:250] TEST Top shape for layer 19 'pool1_pool1_0_split' 32 108 14 14 (677376)
I0904 09:16:56.404675 14545 net.cpp:250] TEST Top shape for layer 19 'pool1_pool1_0_split' 32 108 14 14 (677376)
I0904 09:16:56.404678 14545 layer_factory.hpp:136] Creating layer 'conv2' of type 'Convolution'
I0904 09:16:56.404682 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.404697 14545 net.cpp:182] Created Layer conv2 (20)
I0904 09:16:56.404700 14545 net.cpp:559] conv2 <- pool1_pool1_0_split_0
I0904 09:16:56.404703 14545 net.cpp:528] conv2 -> conv2
I0904 09:16:56.410349 14545 net.cpp:243] Setting up conv2
I0904 09:16:56.410373 14545 net.cpp:250] TEST Top shape for layer 20 'conv2' 32 108 14 14 (677376)
I0904 09:16:56.410387 14545 layer_factory.hpp:136] Creating layer 'relu2' of type 'ReLU'
I0904 09:16:56.410392 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.410396 14545 net.cpp:182] Created Layer relu2 (21)
I0904 09:16:56.410400 14545 net.cpp:559] relu2 <- conv2
I0904 09:16:56.410405 14545 net.cpp:511] relu2 -> conv2 (in-place)
I0904 09:16:56.410411 14545 net.cpp:243] Setting up relu2
I0904 09:16:56.410414 14545 net.cpp:250] TEST Top shape for layer 21 'relu2' 32 108 14 14 (677376)
I0904 09:16:56.410418 14545 layer_factory.hpp:136] Creating layer 'pool2' of type 'Pooling'
I0904 09:16:56.410421 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.410426 14545 net.cpp:182] Created Layer pool2 (22)
I0904 09:16:56.410429 14545 net.cpp:559] pool2 <- conv2
I0904 09:16:56.410432 14545 net.cpp:528] pool2 -> pool2
I0904 09:16:56.410478 14545 net.cpp:243] Setting up pool2
I0904 09:16:56.410481 14545 net.cpp:250] TEST Top shape for layer 22 'pool2' 32 108 7 7 (169344)
I0904 09:16:56.410486 14545 layer_factory.hpp:136] Creating layer 'pool3' of type 'Pooling'
I0904 09:16:56.410488 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.410492 14545 net.cpp:182] Created Layer pool3 (23)
I0904 09:16:56.410495 14545 net.cpp:559] pool3 <- pool1_pool1_0_split_1
I0904 09:16:56.410498 14545 net.cpp:528] pool3 -> pool3
I0904 09:16:56.410532 14545 net.cpp:243] Setting up pool3
I0904 09:16:56.410542 14545 net.cpp:250] TEST Top shape for layer 23 'pool3' 32 108 7 7 (169344)
I0904 09:16:56.410547 14545 layer_factory.hpp:136] Creating layer 'concat' of type 'Concat'
I0904 09:16:56.410557 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.410562 14545 net.cpp:182] Created Layer concat (24)
I0904 09:16:56.410564 14545 net.cpp:559] concat <- pool3
I0904 09:16:56.410567 14545 net.cpp:559] concat <- pool2
I0904 09:16:56.410570 14545 net.cpp:528] concat -> concat
I0904 09:16:56.410590 14545 net.cpp:243] Setting up concat
I0904 09:16:56.410594 14545 net.cpp:250] TEST Top shape for layer 24 'concat' 32 216 7 7 (338688)
I0904 09:16:56.410598 14545 layer_factory.hpp:136] Creating layer 'conv3' of type 'Convolution'
I0904 09:16:56.410600 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.410609 14545 net.cpp:182] Created Layer conv3 (25)
I0904 09:16:56.410611 14545 net.cpp:559] conv3 <- concat
I0904 09:16:56.410614 14545 net.cpp:528] conv3 -> conv3
I0904 09:16:56.427047 14545 net.cpp:243] Setting up conv3
I0904 09:16:56.427072 14545 net.cpp:250] TEST Top shape for layer 25 'conv3' 32 100 1 1 (3200)
I0904 09:16:56.427085 14545 layer_factory.hpp:136] Creating layer 'relu3' of type 'ReLU'
I0904 09:16:56.427089 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.427095 14545 net.cpp:182] Created Layer relu3 (26)
I0904 09:16:56.427099 14545 net.cpp:559] relu3 <- conv3
I0904 09:16:56.427103 14545 net.cpp:511] relu3 -> conv3 (in-place)
I0904 09:16:56.427110 14545 net.cpp:243] Setting up relu3
I0904 09:16:56.427114 14545 net.cpp:250] TEST Top shape for layer 26 'relu3' 32 100 1 1 (3200)
I0904 09:16:56.427116 14545 layer_factory.hpp:136] Creating layer 'drop3' of type 'Dropout'
I0904 09:16:56.427120 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.427139 14545 net.cpp:182] Created Layer drop3 (27)
I0904 09:16:56.427141 14545 net.cpp:559] drop3 <- conv3
I0904 09:16:56.427145 14545 net.cpp:511] drop3 -> conv3 (in-place)
I0904 09:16:56.431524 14545 net.cpp:243] Setting up drop3
I0904 09:16:56.431533 14545 net.cpp:250] TEST Top shape for layer 27 'drop3' 32 100 1 1 (3200)
I0904 09:16:56.431540 14545 layer_factory.hpp:136] Creating layer 'conv4' of type 'Convolution'
I0904 09:16:56.431543 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.431552 14545 net.cpp:182] Created Layer conv4 (28)
I0904 09:16:56.431555 14545 net.cpp:559] conv4 <- conv3
I0904 09:16:56.431560 14545 net.cpp:528] conv4 -> conv4
I0904 09:16:56.431869 14545 net.cpp:243] Setting up conv4
I0904 09:16:56.431876 14545 net.cpp:250] TEST Top shape for layer 28 'conv4' 32 55 1 1 (1760)
I0904 09:16:56.431884 14545 layer_factory.hpp:136] Creating layer 'conv4_conv4_0_split' of type 'Split'
I0904 09:16:56.431886 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.431891 14545 net.cpp:182] Created Layer conv4_conv4_0_split (29)
I0904 09:16:56.431895 14545 net.cpp:559] conv4_conv4_0_split <- conv4
I0904 09:16:56.431898 14545 net.cpp:528] conv4_conv4_0_split -> conv4_conv4_0_split_0
I0904 09:16:56.431902 14545 net.cpp:528] conv4_conv4_0_split -> conv4_conv4_0_split_1
I0904 09:16:56.431932 14545 net.cpp:243] Setting up conv4_conv4_0_split
I0904 09:16:56.431936 14545 net.cpp:250] TEST Top shape for layer 29 'conv4_conv4_0_split' 32 55 1 1 (1760)
I0904 09:16:56.431939 14545 net.cpp:250] TEST Top shape for layer 29 'conv4_conv4_0_split' 32 55 1 1 (1760)
I0904 09:16:56.431942 14545 layer_factory.hpp:136] Creating layer 'cvg/classifier' of type 'Convolution'
I0904 09:16:56.431946 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.431952 14545 net.cpp:182] Created Layer cvg/classifier (30)
I0904 09:16:56.431957 14545 net.cpp:559] cvg/classifier <- conv4_conv4_0_split_0
I0904 09:16:56.431967 14545 net.cpp:528] cvg/classifier -> cvg/classifier
I0904 09:16:56.432183 14545 net.cpp:243] Setting up cvg/classifier
I0904 09:16:56.432199 14545 net.cpp:250] TEST Top shape for layer 30 'cvg/classifier' 32 1 2 2 (128)
I0904 09:16:56.432207 14545 layer_factory.hpp:136] Creating layer 'coverage/sig' of type 'Sigmoid'
I0904 09:16:56.432210 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.432214 14545 net.cpp:182] Created Layer coverage/sig (31)
I0904 09:16:56.432217 14545 net.cpp:559] coverage/sig <- cvg/classifier
I0904 09:16:56.432221 14545 net.cpp:528] coverage/sig -> coverage
I0904 09:16:56.432252 14545 net.cpp:243] Setting up coverage/sig
I0904 09:16:56.432258 14545 net.cpp:250] TEST Top shape for layer 31 'coverage/sig' 32 1 2 2 (128)
I0904 09:16:56.432262 14545 layer_factory.hpp:136] Creating layer 'coverage_coverage/sig_0_split' of type 'Split'
I0904 09:16:56.432265 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.432270 14545 net.cpp:182] Created Layer coverage_coverage/sig_0_split (32)
I0904 09:16:56.432272 14545 net.cpp:559] coverage_coverage/sig_0_split <- coverage
I0904 09:16:56.432276 14545 net.cpp:528] coverage_coverage/sig_0_split -> coverage_coverage/sig_0_split_0
I0904 09:16:56.432279 14545 net.cpp:528] coverage_coverage/sig_0_split -> coverage_coverage/sig_0_split_1
I0904 09:16:56.432305 14545 net.cpp:243] Setting up coverage_coverage/sig_0_split
I0904 09:16:56.432309 14545 net.cpp:250] TEST Top shape for layer 32 'coverage_coverage/sig_0_split' 32 1 2 2 (128)
I0904 09:16:56.432312 14545 net.cpp:250] TEST Top shape for layer 32 'coverage_coverage/sig_0_split' 32 1 2 2 (128)
I0904 09:16:56.432317 14545 layer_factory.hpp:136] Creating layer 'bbox/regressor' of type 'Convolution'
I0904 09:16:56.432318 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.432325 14545 net.cpp:182] Created Layer bbox/regressor (33)
I0904 09:16:56.432329 14545 net.cpp:559] bbox/regressor <- conv4_conv4_0_split_1
I0904 09:16:56.432332 14545 net.cpp:528] bbox/regressor -> bboxes
I0904 09:16:56.432569 14545 net.cpp:243] Setting up bbox/regressor
I0904 09:16:56.432574 14545 net.cpp:250] TEST Top shape for layer 33 'bbox/regressor' 32 4 2 2 (512)
I0904 09:16:56.432580 14545 layer_factory.hpp:136] Creating layer 'bboxes_bbox/regressor_0_split' of type 'Split'
I0904 09:16:56.432584 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.432587 14545 net.cpp:182] Created Layer bboxes_bbox/regressor_0_split (34)
I0904 09:16:56.432590 14545 net.cpp:559] bboxes_bbox/regressor_0_split <- bboxes
I0904 09:16:56.432593 14545 net.cpp:528] bboxes_bbox/regressor_0_split -> bboxes_bbox/regressor_0_split_0
I0904 09:16:56.432597 14545 net.cpp:528] bboxes_bbox/regressor_0_split -> bboxes_bbox/regressor_0_split_1
I0904 09:16:56.432624 14545 net.cpp:243] Setting up bboxes_bbox/regressor_0_split
I0904 09:16:56.432627 14545 net.cpp:250] TEST Top shape for layer 34 'bboxes_bbox/regressor_0_split' 32 4 2 2 (512)
I0904 09:16:56.432631 14545 net.cpp:250] TEST Top shape for layer 34 'bboxes_bbox/regressor_0_split' 32 4 2 2 (512)
I0904 09:16:56.432634 14545 layer_factory.hpp:136] Creating layer 'bbox_mask' of type 'Eltwise'
I0904 09:16:56.432637 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.432642 14545 net.cpp:182] Created Layer bbox_mask (35)
I0904 09:16:56.432646 14545 net.cpp:559] bbox_mask <- bboxes_bbox/regressor_0_split_0
I0904 09:16:56.432649 14545 net.cpp:559] bbox_mask <- coverage-block
I0904 09:16:56.432652 14545 net.cpp:528] bbox_mask -> bboxes-masked
I0904 09:16:56.432670 14545 net.cpp:243] Setting up bbox_mask
I0904 09:16:56.432674 14545 net.cpp:250] TEST Top shape for layer 35 'bbox_mask' 32 4 2 2 (512)
I0904 09:16:56.432678 14545 layer_factory.hpp:136] Creating layer 'bbox-norm' of type 'Eltwise'
I0904 09:16:56.432680 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.432687 14545 net.cpp:182] Created Layer bbox-norm (36)
I0904 09:16:56.432693 14545 net.cpp:559] bbox-norm <- bboxes-masked
I0904 09:16:56.432698 14545 net.cpp:559] bbox-norm <- size-block_size-block_0_split_1
I0904 09:16:56.432701 14545 net.cpp:528] bbox-norm -> bboxes-masked-norm
I0904 09:16:56.432718 14545 net.cpp:243] Setting up bbox-norm
I0904 09:16:56.432721 14545 net.cpp:250] TEST Top shape for layer 36 'bbox-norm' 32 4 2 2 (512)
I0904 09:16:56.432725 14545 layer_factory.hpp:136] Creating layer 'bbox-obj-norm' of type 'Eltwise'
I0904 09:16:56.432727 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.432731 14545 net.cpp:182] Created Layer bbox-obj-norm (37)
I0904 09:16:56.432734 14545 net.cpp:559] bbox-obj-norm <- bboxes-masked-norm
I0904 09:16:56.432737 14545 net.cpp:559] bbox-obj-norm <- obj-block_obj-block_0_split_1
I0904 09:16:56.432741 14545 net.cpp:528] bbox-obj-norm -> bboxes-obj-masked-norm
I0904 09:16:56.432757 14545 net.cpp:243] Setting up bbox-obj-norm
I0904 09:16:56.432760 14545 net.cpp:250] TEST Top shape for layer 37 'bbox-obj-norm' 32 4 2 2 (512)
I0904 09:16:56.432765 14545 layer_factory.hpp:136] Creating layer 'bbox_loss' of type 'L1Loss'
I0904 09:16:56.432766 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.432773 14545 net.cpp:182] Created Layer bbox_loss (38)
I0904 09:16:56.432776 14545 net.cpp:559] bbox_loss <- bboxes-obj-masked-norm
I0904 09:16:56.432780 14545 net.cpp:559] bbox_loss <- bbox-obj-label-norm
I0904 09:16:56.432783 14545 net.cpp:528] bbox_loss -> loss_bbox
I0904 09:16:56.432838 14545 net.cpp:243] Setting up bbox_loss
I0904 09:16:56.432843 14545 net.cpp:250] TEST Top shape for layer 38 'bbox_loss' (1)
I0904 09:16:56.432847 14545 net.cpp:254]     with loss weight 2
I0904 09:16:56.432855 14545 layer_factory.hpp:136] Creating layer 'coverage_loss' of type 'EuclideanLoss'
I0904 09:16:56.432858 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.432864 14545 net.cpp:182] Created Layer coverage_loss (39)
I0904 09:16:56.432868 14545 net.cpp:559] coverage_loss <- coverage_coverage/sig_0_split_0
I0904 09:16:56.432870 14545 net.cpp:559] coverage_loss <- coverage-label_slice-label_4_split_0
I0904 09:16:56.432873 14545 net.cpp:528] coverage_loss -> loss_coverage
I0904 09:16:56.432920 14545 net.cpp:243] Setting up coverage_loss
I0904 09:16:56.432925 14545 net.cpp:250] TEST Top shape for layer 39 'coverage_loss' (1)
I0904 09:16:56.432929 14545 net.cpp:254]     with loss weight 1
I0904 09:16:56.432932 14545 layer_factory.hpp:136] Creating layer 'cluster' of type 'Python'
I0904 09:16:56.432935 14545 layer_factory.hpp:148] Layer's types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT
I0904 09:16:56.432955 14545 layer_factory.cpp:325] Importing Python module 'caffe.layers.detectnet.clustering'
*** Aborted at 1504480616 (unix time) try "date -d @1504480616" if you are using GNU date ***
PC: @     0x7ff58d1a9faf dgetrf_
*** SIGSEGV (@0x280) received by PID 14545 (TID 0x7ff6174e3a00) from PID 640; stack trace: ***
    @     0x7ff61564b7e0 (unknown)
    @     0x7ff58d1a9faf dgetrf_
    @     0x7ff58cab4715 f2py_rout__flapack_dgetrf
    @     0x7ff5fa5260e3 PyObject_Call
    @     0x7ff5fa4fffed PyEval_EvalFrameEx
    @     0x7ff5fa55b4c5 PyEval_EvalCodeEx
    @     0x7ff5fa50014e PyEval_EvalFrameEx
    @     0x7ff5fa55b4c5 PyEval_EvalCodeEx
    @     0x7ff5fa5758f9 PyEval_EvalCode
    @     0x7ff5fa5916e8 PyImport_ExecCodeModuleEx
    @     0x7ff5fa5918cf load_source_module
    @     0x7ff5fa541c87 PyImport_ImportModuleLevel
    @     0x7ff5fa544768 builtin___import__.lto_priv.1230
    @     0x7ff5fa5260e3 PyObject_Call
    @     0x7ff5fa55abd0 PyEval_CallObjectWithKeywords
    @     0x7ff5fa4fe3c9 PyEval_EvalFrameEx
    @     0x7ff5fa55b4c5 PyEval_EvalCodeEx
    @     0x7ff5fa5758f9 PyEval_EvalCode
    @     0x7ff5fa5916e8 PyImport_ExecCodeModuleEx
    @     0x7ff5fa5918cf load_source_module
    @     0x7ff5fa59233a load_package
    @     0x7ff5fa541c87 PyImport_ImportModuleLevel
    @     0x7ff5fa544768 builtin___import__.lto_priv.1230
    @     0x7ff5fa5260e3 PyObject_Call
    @     0x7ff5fa55abd0 PyEval_CallObjectWithKeywords
    @     0x7ff5fa4fe3c9 PyEval_EvalFrameEx
    @     0x7ff5fa55b4c5 PyEval_EvalCodeEx
    @     0x7ff5fa5758f9 PyEval_EvalCode
    @     0x7ff5fa5916e8 PyImport_ExecCodeModuleEx
    @     0x7ff5fa5918cf load_source_module
    @     0x7ff5fa541c87 PyImport_ImportModuleLevel
    @     0x7ff5fa544768 builtin___import__.lto_priv.1230
Segmentation fault (core dumped)
Bidski commented 7 years ago

Just ran it on the Jetson TX2, the layer loads fine and training starts (and the print statements that I added print out as well).

I shall look into my OpenBLAS installation and see if I can find anything interesting there

Bidski commented 7 years ago

OK, I believe I have resolved this issue now.

For those who have a similar problem I did the following: Note: These instructions are specific to Arch Linux

  1. Remove cblas, blas, or any other packages that provide blas functionality (the openblas-lapack install procedure will complain about any conflicting packages)
  2. Remove python2-numpy and any of its dependencies (specifically matplotlib)
  3. Download, build, and install openblas-lapack package from AUR
  4. Download, build, and install python2-numpy-openblas package from AUR
  5. Reinstall any of the numpy dependencies that you removed earlier