caffe digits ERROR: error code -11

[zhong-xin]

When training orientation head,I met the following questions.

Setting up train-data Top shape: 64 3 180 320 (11059200) Top shape: 64 (64) Memory required for data: 44237056 Creating layer data_aug

System configuration

Operating system: ubuntu 16.04
Compiler:
CUDA version (if applicable): 9.0
CUDNN version (if applicable): 9.0.176
Python version (if using pycaffe): tried both 2.7 and 3.5.2

[Alexey-Kamenev]

What is the GPU that are you using? Try reducing minibatch size (e.g. to 8) and see if it works.

[zhong-xin]

@Alexey-Kamenev Titan XP.Reducing minibatch size is not help.

[zhong-xin]

@Alexey-Kamenev

I0424 22:31:13.564702  1215 upgrade_proto.cpp:1084] Attempting to upgrade input file specified using deprecated 'solver_type' field (enum)': /home/xin/digits/digits/jobs/20180424-223112-f147/solver.prototxt
I0424 22:31:13.565052  1215 upgrade_proto.cpp:1091] Successfully upgraded file specified using deprecated 'solver_type' field (enum) to 'type' field (string).
W0424 22:31:13.565063  1215 upgrade_proto.cpp:1093] Note that future Caffe releases will only support 'type' field (string) for a solver's type.
I0424 22:31:13.565243  1215 caffe.cpp:204] Using GPUs 0
I0424 22:31:13.586935  1215 caffe.cpp:209] GPU 0: TITAN Xp
I0424 22:31:14.909298  1215 solver.cpp:45] Initializing solver from parameters:
test_iter: 164
test_interval: 3482
base_lr: 0.001
display: 79
max_iter: 34820
lr_policy: "poly"
power: 1
momentum: 0.9
weight_decay: 1.0000001e-05
snapshot: 8705
snapshot_prefix: "snapshot"
solver_mode: GPU
device_id: 0
net: "train_val.prototxt"
train_state {
level: 0
stage: ""
}
type: "Nesterov"
I0424 22:31:14.909458  1215 solver.cpp:102] Creating training net from net file: train_val.prototxt
I0424 22:31:14.910451  1215 net.cpp:294] The NetState phase (0) differed from the phase (1) specified by a rule in layer val-data
I0424 22:31:14.910498  1215 net.cpp:294] The NetState phase (0) differed from the phase (1) specified by a rule in layer accuracy
I0424 22:31:14.910944  1215 net.cpp:51] Initializing net from parameters:
state {
phase: TRAIN
level: 0
stage: ""
}
layer {
name: "train-data"
type: "Data"
top: "data"
top: "label"
include {
phase: TRAIN
}
data_param {
source: "/home/xin/digits/digits/jobs/20180424-221713-4a74/train_db"
batch_size: 64
backend: LMDB
}
}
layer {
name: "data_aug"
type: "Python"
bottom: "data"
bottom: "label"
top: "data"
top: "label"
include {
phase: TRAIN
}
python_param {
module: "digits_python_layers"
layer: "TrailAugLayer"
param_str: "{\'debug\': False, \'hflip3\': True, \'blurProb\': 0.1, \'contrastRadius\': 0.2, \'brightnessRadius\': 0.2, \'saturationRadius\': 0.3, \'sharpnessRadius\': 0.3, \'scaleMin\': 0.9, \'scaleMax\': 1.2, \'rotateAngle\': 15, \'numThreads\': 32}"
}
}
layer {
name: "sub_mean"
type: "Scale"
bottom: "data"
top: "sub_mean"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 0.00390625
}
bias_term: true
bias_filler {
value: -0.5
}
}
}
layer {
name: "conv1"
type: "Convolution"
bottom: "sub_mean"
top: "conv1"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 64
bias_term: true
kernel_size: 7
stride: 2
weight_filler {
type: "xavier"
}
}
}
layer {
name: "conv1_srelu1_1"
type: "Scale"
bottom: "conv1"
top: "conv1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "conv1_srelu1_2"
type: "ReLU"
bottom: "conv1"
top: "conv1"
}
layer {
name: "conv1_srelu1_3"
type: "Scale"
bottom: "conv1"
top: "conv1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "pool1"
type: "Pooling"
bottom: "conv1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 3
stride: 2
}
}
layer {
name: "res1_1_1"
type: "Convolution"
bottom: "pool1"
top: "res1_1_1"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 64
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res1_1_1_srelu_1"
type: "Scale"
bottom: "res1_1_1"
top: "res1_1_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res1_1_1_srelu_2"
type: "ReLU"
bottom: "res1_1_1"
top: "res1_1_1"
}
layer {
name: "res1_1_1_srelu_3"
type: "Scale"
bottom: "res1_1_1"
top: "res1_1_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res1_1_2"
type: "Convolution"
bottom: "res1_1_1"
top: "res1_1_2"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 64
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res1_1_sum"
type: "Eltwise"
bottom: "pool1"
bottom: "res1_1_2"
top: "res1_1"
eltwise_param {
operation: SUM
}
}
layer {
name: "res1_1_srelu_1"
type: "Scale"
bottom: "res1_1"
top: "res1_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res1_1_srelu_2"
type: "ReLU"
bottom: "res1_1"
top: "res1_1"
}
layer {
name: "res1_1_srelu_3"
type: "Scale"
bottom: "res1_1"
top: "res1_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res1_2_1"
type: "Convolution"
bottom: "res1_1"
top: "res1_2_1"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 64
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res1_2_1_srelu_1"
type: "Scale"
bottom: "res1_2_1"
top: "res1_2_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res1_2_1_srelu_2"
type: "ReLU"
bottom: "res1_2_1"
top: "res1_2_1"
}
layer {
name: "res1_2_1_srelu_3"
type: "Scale"
bottom: "res1_2_1"
top: "res1_2_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res1_2_2"
type: "Convolution"
bottom: "res1_2_1"
top: "res1_2_2"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 64
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res1_2_sum"
type: "Eltwise"
bottom: "res1_1"
bottom: "res1_2_2"
top: "res1_2"
eltwise_param {
operation: SUM
}
}
layer {
name: "res1_2_srelu_1"
type: "Scale"
bottom: "res1_2"
top: "res1_2"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res1_2_srelu_2"
type: "ReLU"
bottom: "res1_2"
top: "res1_2"
}
layer {
name: "res1_2_srelu_3"
type: "Scale"
bottom: "res1_2"
top: "res1_2"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res2_1_1"
type: "Convolution"
bottom: "res1_2"
top: "res2_1_1"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 128
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res2_1_1_srelu_1"
type: "Scale"
bottom: "res2_1_1"
top: "res2_1_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res2_1_1_srelu_2"
type: "ReLU"
bottom: "res2_1_1"
top: "res2_1_1"
}
layer {
name: "res2_1_1_srelu_3"
type: "Scale"
bottom: "res2_1_1"
top: "res2_1_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res2_1_2"
type: "Convolution"
bottom: "res2_1_1"
top: "res2_1_2"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 128
bias_term: true
pad: 1
kernel_size: 3
stride: 2
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res2_1_proj"
type: "Convolution"
bottom: "res1_2"
top: "res2_1_proj"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 128
bias_term: true
pad: 0
kernel_size: 1
stride: 2
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res2_1_sum"
type: "Eltwise"
bottom: "res2_1_proj"
bottom: "res2_1_2"
top: "res2_1"
eltwise_param {
operation: SUM
}
}
layer {
name: "res2_1_srelu_1"
type: "Scale"
bottom: "res2_1"
top: "res2_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res2_1_srelu_2"
type: "ReLU"
bottom: "res2_1"
top: "res2_1"
}
layer {
name: "res2_1_srelu_3"
type: "Scale"
bottom: "res2_1"
top: "res2_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res2_2_1"
type: "Convolution"
bottom: "res2_1"
top: "res2_2_1"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 128
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res2_2_1_srelu_1"
type: "Scale"
bottom: "res2_2_1"
top: "res2_2_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res2_2_1_srelu_2"
type: "ReLU"
bottom: "res2_2_1"
top: "res2_2_1"
}
layer {
name: "res2_2_1_srelu_3"
type: "Scale"
bottom: "res2_2_1"
top: "res2_2_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res2_2_2"
type: "Convolution"
bottom: "res2_2_1"
top: "res2_2_2"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 128
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res2_2_sum"
type: "Eltwise"
bottom: "res2_1"
bottom: "res2_2_2"
top: "res2_2"
eltwise_param {
operation: SUM
}
}
layer {
name: "res2_2_srelu_1"
type: "Scale"
bottom: "res2_2"
top: "res2_2"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res2_2_srelu_2"
type: "ReLU"
bottom: "res2_2"
top: "res2_2"
}
layer {
name: "res2_2_srelu_3"
type: "Scale"
bottom: "res2_2"
top: "res2_2"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res3_1_1"
type: "Convolution"
bottom: "res2_2"
top: "res3_1_1"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 256
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res3_1_1_srelu_1"
type: "Scale"
bottom: "res3_1_1"
top: "res3_1_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res3_1_1_srelu_2"
type: "ReLU"
bottom: "res3_1_1"
top: "res3_1_1"
}
layer {
name: "res3_1_1_srelu_3"
type: "Scale"
bottom: "res3_1_1"
top: "res3_1_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res3_1_2"
type: "Convolution"
bottom: "res3_1_1"
top: "res3_1_2"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 256
bias_term: true
pad: 1
kernel_size: 3
stride: 2
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res3_1_proj"
type: "Convolution"
bottom: "res2_2"
top: "res3_1_proj"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 256
bias_term: true
pad: 0
kernel_size: 1
stride: 2
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res3_1_sum"
type: "Eltwise"
bottom: "res3_1_proj"
bottom: "res3_1_2"
top: "res3_1"
eltwise_param {
operation: SUM
}
}
layer {
name: "res3_1_srelu_1"
type: "Scale"
bottom: "res3_1"
top: "res3_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res3_1_srelu_2"
type: "ReLU"
bottom: "res3_1"
top: "res3_1"
}
layer {
name: "res3_1_srelu_3"
type: "Scale"
bottom: "res3_1"
top: "res3_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res3_2_1"
type: "Convolution"
bottom: "res3_1"
top: "res3_2_1"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 256
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res3_2_1_srelu_1"
type: "Scale"
bottom: "res3_2_1"
top: "res3_2_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res3_2_1_srelu_2"
type: "ReLU"
bottom: "res3_2_1"
top: "res3_2_1"
}
layer {
name: "res3_2_1_srelu_3"
type: "Scale"
bottom: "res3_2_1"
top: "res3_2_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res3_2_2"
type: "Convolution"
bottom: "res3_2_1"
top: "res3_2_2"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 256
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res3_2_sum"
type: "Eltwise"
bottom: "res3_1"
bottom: "res3_2_2"
top: "res3_2"
eltwise_param {
operation: SUM
}
}
layer {
name: "res3_2_srelu_1"
type: "Scale"
bottom: "res3_2"
top: "res3_2"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res3_2_srelu_2"
type: "ReLU"
bottom: "res3_2"
top: "res3_2"
}
layer {
name: "res3_2_srelu_3"
type: "Scale"
bottom: "res3_2"
top: "res3_2"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res4_1_1"
type: "Convolution"
bottom: "res3_2"
top: "res4_1_1"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 512
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res4_1_1_srelu_1"
type: "Scale"
bottom: "res4_1_1"
top: "res4_1_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res4_1_1_srelu_2"
type: "ReLU"
bottom: "res4_1_1"
top: "res4_1_1"
}
layer {
name: "res4_1_1_srelu_3"
type: "Scale"
bottom: "res4_1_1"
top: "res4_1_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res4_1_2"
type: "Convolution"
bottom: "res4_1_1"
top: "res4_1_2"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 512
bias_term: true
pad: 1
kernel_size: 3
stride: 2
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res4_1_proj"
type: "Convolution"
bottom: "res3_2"
top: "res4_1_proj"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 512
bias_term: true
pad: 0
kernel_size: 1
stride: 2
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res4_1_sum"
type: "Eltwise"
bottom: "res4_1_proj"
bottom: "res4_1_2"
top: "res4_1"
eltwise_param {
operation: SUM
}
}
layer {
name: "res4_1_srelu_1"
type: "Scale"
bottom: "res4_1"
top: "res4_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res4_1_srelu_2"
type: "ReLU"
bottom: "res4_1"
top: "res4_1"
}
layer {
name: "res4_1_srelu_3"
type: "Scale"
bottom: "res4_1"
top: "res4_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res4_2_1"
type: "Convolution"
bottom: "res4_1"
top: "res4_2_1"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 512
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res4_2_1_srelu_1"
type: "Scale"
bottom: "res4_2_1"
top: "res4_2_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res4_2_1_srelu_2"
type: "ReLU"
bottom: "res4_2_1"
top: "res4_2_1"
}
layer {
name: "res4_2_1_srelu_3"
type: "Scale"
bottom: "res4_2_1"
top: "res4_2_1"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "res4_2_2"
type: "Convolution"
bottom: "res4_2_1"
top: "res4_2_2"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
convolution_param {
num_output: 512
bias_term: true
pad: 1
kernel_size: 3
stride: 1
weight_filler {
type: "xavier"
}
}
}
layer {
name: "res4_2_sum"
type: "Eltwise"
bottom: "res4_1"
bottom: "res4_2_2"
top: "res4_2"
eltwise_param {
operation: SUM
}
}
layer {
name: "res4_2_srelu_1"
type: "Scale"
bottom: "res4_2"
top: "res4_2"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: 1
}
}
}
layer {
name: "res4_2_srelu_2"
type: "ReLU"
bottom: "res4_2"
top: "res4_2"
}
layer {
name: "res4_2_srelu_3"
type: "Scale"
bottom: "res4_2"
top: "res4_2"
param {
lr_mult: 0
decay_mult: 0
}
param {
lr_mult: 0
decay_mult: 0
}
scale_param {
filler {
value: 1
}
bias_term: true
bias_filler {
value: -1
}
}
}
layer {
name: "pool_avg"
type: "Pooling"
bottom: "res4_2"
top: "pool_avg"
pooling_param {
pool: AVE
stride: 1
kernel_h: 6
kernel_w: 10
}
}
layer {
name: "fc3"
type: "InnerProduct"
bottom: "pool_avg"
top: "fc3"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 1
decay_mult: 0
}
inner_product_param {
num_output: 3
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
value: 0
}
}
}
layer {
name: "cee_loss"
type: "Python"
bottom: "fc3"
bottom: "label"
top: "cee_loss"
loss_weight: 1
python_param {
module: "digits_python_layers"
layer: "CrossEntropySoftmaxWithEntropyLossLayer"
param_str: "{ \'entScale\': 0.01, \'pScale\': 0.0001, \'label_eps\': 0.01 }"
}
}
I0424 22:31:14.911382  1215 layer_factory.hpp:77] Creating layer train-data
I0424 22:31:14.911535  1215 db_lmdb.cpp:35] Opened lmdb /home/xin/digits/digits/jobs/20180424-221713-4a74/train_db
I0424 22:31:14.911571  1215 net.cpp:84] Creating Layer train-data
I0424 22:31:14.911581  1215 net.cpp:380] train-data -> data
I0424 22:31:14.911607  1215 net.cpp:380] train-data -> label
I0424 22:31:14.913704  1215 data_layer.cpp:45] output data size: 64,3,180,320
I0424 22:31:15.025135  1215 net.cpp:122] Setting up train-data
I0424 22:31:15.025193  1215 net.cpp:129] Top shape: 64 3 180 320 (11059200)
I0424 22:31:15.025198  1215 net.cpp:129] Top shape: 64 (64)
I0424 22:31:15.025200  1215 net.cpp:137] Memory required for data: 44237056
I0424 22:31:15.025212  1215 layer_factory.hpp:77] Creating layer data_aug
*** Aborted at 1524580275 (unix time) try "date -d @1524580275" if you are using GNU date ***
PC: @     0x7f60fb1ce873 std::_Hashtable<>::clear()
*** SIGSEGV (@0x9) received by PID 1215 (TID 0x7f624ad6f740) from PID 9; stack trace: ***
@     0x7f6247fed4b0 (unknown)
@     0x7f60fb1ce873 std::_Hashtable<>::clear()
@     0x7f60fb1c0346 google::protobuf::DescriptorPool::FindFileByName()
@     0x7f60fb19eac8 google::protobuf::python::cdescriptor_pool::AddSerializedFile()
@     0x7f624864a9f0 PyEval_EvalFrameEx
@     0x7f624878005c PyEval_EvalCodeEx
@     0x7f62486d646d (unknown)
@     0x7f62486a9273 PyObject_Call
@     0x7f62486c9b75 (unknown)
@     0x7f6248660173 (unknown)
@     0x7f62486a9273 PyObject_Call
@     0x7f624864735c PyEval_EvalFrameEx
@     0x7f624878005c PyEval_EvalCodeEx
@     0x7f6248641da9 PyEval_EvalCode
@     0x7f62486e3244 PyImport_ExecCodeModuleEx
@     0x7f62486e3c1f (unknown)
@     0x7f62486e5390 (unknown)
@     0x7f62486e5658 (unknown)
@     0x7f62486e676b PyImport_ImportModuleLevel
@     0x7f62486508b8 (unknown)
@     0x7f62486a9273 PyObject_Call
@     0x7f624877f487 PyEval_CallObjectWithKeywords
@     0x7f62486457e6 PyEval_EvalFrameEx
@     0x7f624878005c PyEval_EvalCodeEx
@     0x7f6248641da9 PyEval_EvalCode
@     0x7f62486e3244 PyImport_ExecCodeModuleEx
@     0x7f62486e3c1f (unknown)
@     0x7f62486e5390 (unknown)
@     0x7f62486e5658 (unknown)
@     0x7f62486e676b PyImport_ImportModuleLevel
@     0x7f62486508b8 (unknown)
@     0x7f62486a9273 PyObject_Call

[Alexey-Kamenev]

From the log, it looks like something went wrong when data_aug layer was being created. This layer is implemented using Python custom layer which is defined here. Can you please check that you provide correct path to this file when you run a job in DIGITS and that this file is accessible by DIGITS (e.g. if you run DIGITS in Docker container). See wiki for more details.

[Alexey-Kamenev]

Hope you were able to fix the problem, but if not - feel free to re-open the issue or create a new one.