Re-implement CVPR2017 paper: "dense captioning with joint inference and visual context" and minor changes in Tensorflow. (mAP 8.296 after 500k iters of training)
exec
++ tee -a logs/s1_resnet_v2_50_vg_1.2_train.txt.2018-08-03_09-38-38
echo Logging output to logs/s1_resnet_v2_50_vg_1.2_train.txt.2018-08-03_09-38-38
Logging output to logs/s1_resnet_v2_50_vg_1.2_train.txt.2018-08-03_09-38-38
'[' 1 -lt 2 ']'
python ./tools/train_net.py --weights /home/xiewei/densecap-tensorflow/slim_models/resnet_v2_50.ckpt --imdb vg_1.2_train --imdbval vg_1.2_val --iters 200000 --cfg scripts/dense_cap_config.yml --data_dir /home/xiewei/densecap-tensorflow/visual_genome --net resnet_v2_50 --set EXP_DIR dc_conv_fixed CONTEXT_FUSION False RESNET.FIXED_BLOCKS 3
/usr/local/lib/python2.7/dist-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Couldn't import dot_parser, loading of dot files will not be possible.
------ called with args: -------
Namespace(cfg_file='scripts/dense_cap_config.yml', context_fusion=False, data_dir='/home/xiewei/densecap-tensorflow/visual_genome', device='gpu', device_id=0, embed_dim=512, imdb_name='vg_1.2_train', imdbval_name='vg_1.2_val', max_iters=200000, net='resnet_v2_50', randomize=False, set_cfgs=['EXP_DIR', 'dc_conv_fixed', 'CONTEXT_FUSION', 'False', 'RESNET.FIXED_BLOCKS', '3'], tag=None, weights='/home/xiewei/densecap-tensorflow/slim_models/resnet_v2_50.ckpt')
runing with LIMIT_RAM: True
Using config:
{'ALL_TEST': False,
'ALL_TEST_NUM_TEST': 1000,
'ALL_TEST_NUM_TRAIN': 100,
'ALL_TEST_NUM_VAL': 100,
'ANCHOR_RATIOS': [0.5, 1, 2],
'ANCHOR_SCALES': [4, 8, 16, 32],
'CACHE_DIR': '/home/xiewei/densecap-tensorflow/visual_genome/1.2',
'CONTEXT_FUSION': False,
'CONTEXT_FUSION_MODE': 'sum',
'CONTEXT_MODE': 'concat',
'DATA_DIR': '/home/xiewei/densecap-tensorflow/visual_genome',
'DEBUG_ALL': False,
'EMBED_DIM': 512,
'END_INDEX': 2,
'EXP_DIR': 'dc_conv_fixed',
'FILTER_SMALL_BOX': False,
'GLOVE_DIM': 300,
'GPU_ID': 0,
'INIT_BY_GLOVE': False,
'KEEP_AS_GLOVE_DIM': False,
'LIMIT_RAM': True,
'LOG_DIR': '/home/xiewei/densecap-tensorflow/logs',
'LOSS': {'BBOX_W': 0.01,
'CAP_W': 1.0,
'CLS_W': 0.1,
'RPN_BBOX_W': 0.05,
'RPN_CLS_W': 0.1},
'MAX_WORDS': 10,
'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
'POOLING_MODE': 'crop',
'POOLING_SIZE': 7,
'RESNET': {'FIXED_BLOCKS': 3, 'MAX_POOL': False},
'RNG_SEED': 3,
'ROOT_DIR': '/home/xiewei/densecap-tensorflow',
'RPN_CHANNELS': 512,
'SAMPLE_NUM_FIXED_REGIONS': False,
'SPLIT_DIR': '/home/xiewei/densecap-tensorflow/info',
'TEST': {'BBOX_REG': True,
'BEAM_SIZE': 3,
'HAS_RPN': True,
'LN_FACTOR': 0.0,
'MAX_SIZE': 720,
'MODE': 'nms',
'NMS': 0.5,
'PROPOSAL_METHOD': 'gt',
'RPN_MIN_SIZE': 16,
'RPN_NMS_THRESH': 0.6,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'RPN_TOP_N': 5000,
'SCALES': [600],
'SVM': False,
'USE_BEAM_SEARCH': False},
'TIME_STEPS': 12,
'TRAIN': {'ASPECT_GROUPING': True,
'BATCH_SIZE': 256,
'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'BBOX_NORMALIZE_TARGETS': True,
'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
'BBOX_REG': True,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'BIAS_DECAY': False,
'CLIP_NORM': 40.0,
'DISPLAY': 10,
'DOUBLE_BIAS': False,
'EXP_DECAY_RATE': 0.9,
'EXP_DECAY_STEPS': 5000,
'FG_FRACTION': 0.5,
'FG_THRESH': 0.5,
'GAMMA': 0.5,
'HAS_RPN': True,
'IMS_PER_BATCH': 1,
'LEARNING_RATE': 0.001,
'LR_DIY_DECAY': True,
'MAX_SIZE': 720,
'MOMENTUM': 0.98,
'OPTIMIZER': 'sgd_m',
'PROPOSAL_METHOD': 'gt',
'RPN_BATCHSIZE': 256,
'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 16,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SNAPSHOT_ITERS': 5000,
'SNAPSHOT_KEPT': 3,
'SNAPSHOT_PREFIX': 'res50_densecap',
'STEPSIZE': [100000],
'SUMMARY_INTERVAL': 10,
'USE_FLIPPED': True,
'USE_GT': False,
'WEIGHT_DECAY': 0.0001,
'WEIGHT_INITIALIZER': 'normal'},
'TRAIN_GLOVE': False,
'USE_GPU_NMS': True,
'VOCAB_END_ID': 2,
'VOCAB_SIZE': 10000,
'VOCAB_START_ID': 1}
data_path: /home/xiewei/densecap-tensorflow/visual_genome/1.2
loading splits from /home/xiewei/densecap-tensorflow/info/densecap_splits.json
Number of examples: 77398
train gt roidb could be loaded from /home/xiewei/densecap-tensorflow/visual_genome/1.2_cache/train_gt_roidb
Getting gt roidb and number of examples is:154796
output will be saved to /home/xiewei/densecap-tensorflow/output/dc_conv_fixed/vg_1.2_train
TensorFlow summaries will be saved to /home/xiewei/densecap-tensorflow/output/dc_conv_fixed/tb/vg_1.2_train/default
data_path: /home/xiewei/densecap-tensorflow/visual_genome/1.2
loading splits from /home/xiewei/densecap-tensorflow/info/densecap_splits.json
Number of examples: 5000
val gt roidb could be loaded from /home/xiewei/densecap-tensorflow/visual_genome/1.2_cache/val_gt_roidb
Getting gt roidb and number of examples is:5000
2018-08-03 09:38:39.720107: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-08-03 09:38:40.880941: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: Tesla P4 major: 6 minor: 1 memoryClockRate(GHz): 1.1135
pciBusID: 0000:01:00.0
totalMemory: 7.43GiB freeMemory: 7.32GiB
2018-08-03 09:38:40.881009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P4, pci bus id: 0000:01:00.0, compute capability: 6.1)
Solving...
LIMIT_RAM version and load index from /home/xiewei/densecap-tensorflow/visual_genome/1.2_cache/train_gt_roidb/image_index.json
LIMIT_RAM version and load index from /home/xiewei/densecap-tensorflow/visual_genome/1.2_cache/val_gt_roidb/image_index.json
Fixing 3 blocks.
Traceback (most recent call last):
File "./tools/train_net.py", line 221, in
main()
File "./tools/train_net.py", line 217, in main
max_iters=args.max_iters)
File "/home/xiewei/densecap-tensorflow/tools/../lib/dense_cap/train.py", line 485, in train_net
sw.train_model(sess, max_iters)
File "/home/xiewei/densecap-tensorflow/tools/../lib/dense_cap/train.py", line 318, in train_model
lr, train_op = self.construct_graph(sess)
File "/home/xiewei/densecap-tensorflow/tools/../lib/dense_cap/train.py", line 148, in construct_graph
layers = self.net.create_architecture('TRAIN', num_classes=1, tag='default')
File "/home/xiewei/densecap-tensorflow/tools/../lib/nets/network.py", line 812, in create_architecture
rois, cls_prob = self._build_network(training)
File "/home/xiewei/densecap-tensorflow/tools/../lib/nets/network.py", line 601, in _build_network
fc7 = self._head_to_tail(pool5, is_training)
File "/home/xiewei/densecap-tensorflow/tools/../lib/nets/resnet_v2.py", line 131, in _head_to_tail
scope=self._resnet_scope)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 221, in resnet_v2
net, activation_fn=nn_ops.relu, scope='postnorm')
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
return func(*args, current_args)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 643, in batch_norm
outputs = layer.apply(inputs, training=is_training)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 671, in apply
return self.call(inputs, args, kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 559, in call
self.build(input_shapes[0])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/normalization.py", line 189, in build
trainable=True)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 458, in add_variable
trainable=trainable and self.trainable)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 1203, in get_variable
constraint=constraint)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 1092, in get_variable
constraint=constraint)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 417, in get_variable
return custom_getter(custom_getter_kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1539, in layer_variable_getter
return _model_variable_getter(getter, args, kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1531, in _model_variable_getter
custom_getter=getter, use_resource=use_resource)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
return func(*args, *current_args)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 262, in model_variable
use_resource=use_resource)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
return func(args, **current_args)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 217, in variable
use_resource=use_resource)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 394, in _true_getter
use_resource=use_resource, constraint=constraint)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 742, in _get_single_variable
name, "".join(traceback.format_list(tb))))
ValueError: Variable resnet_v2_50/postnorm/gamma already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 217, in variable
use_resource=use_resource)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
return func(*args, **current_args)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 262, in model_variable
use_resource=use_resource)
你好,因为 tensorflow/contrib/slim/python/slim/nets/resnet_v1.py 和tensorflow/contrib/slim/python/slim/nets/resnet_v2.py 极其相似,所以我将 lib/nets/resnet_v1.py 中的出现resnet_v1的地方替换为resnet_v2,作为resnet_v2.py,但是resnet_v1.py 依旧保留。
另外,我在 tools/train_net.py 中,也做了相应的修改,导入了resnetv2(from lib.nets.resnet_v2 import resnetv2)。在#load network也做了些修改,如下:
load network
当训练模型时,出现以下错误,希望能得到您的建议,谢谢!!@InnerPeace-Wu
~/densecap-tensorflow$ bash scripts/dense_cap_train.sh visual_genome_1.2 resnet_v2_50 /home/xiewei/densecap-tensorflow/slim_models/resnet_v2_50.ckpt /home/xiewei/densecap-tensorflow/visual_genome 1
python ./tools/train_net.py --weights /home/xiewei/densecap-tensorflow/slim_models/resnet_v2_50.ckpt --imdb vg_1.2_train --imdbval vg_1.2_val --iters 200000 --cfg scripts/dense_cap_config.yml --data_dir /home/xiewei/densecap-tensorflow/visual_genome --net resnet_v2_50 --set EXP_DIR dc_conv_fixed CONTEXT_FUSION False RESNET.FIXED_BLOCKS 3 /usr/local/lib/python2.7/dist-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from
main()
File "./tools/train_net.py", line 217, in main
max_iters=args.max_iters)
File "/home/xiewei/densecap-tensorflow/tools/../lib/dense_cap/train.py", line 485, in train_net
sw.train_model(sess, max_iters)
File "/home/xiewei/densecap-tensorflow/tools/../lib/dense_cap/train.py", line 318, in train_model
lr, train_op = self.construct_graph(sess)
File "/home/xiewei/densecap-tensorflow/tools/../lib/dense_cap/train.py", line 148, in construct_graph
layers = self.net.create_architecture('TRAIN', num_classes=1, tag='default')
File "/home/xiewei/densecap-tensorflow/tools/../lib/nets/network.py", line 812, in create_architecture
rois, cls_prob = self._build_network(training)
File "/home/xiewei/densecap-tensorflow/tools/../lib/nets/network.py", line 601, in _build_network
fc7 = self._head_to_tail(pool5, is_training)
File "/home/xiewei/densecap-tensorflow/tools/../lib/nets/resnet_v2.py", line 131, in _head_to_tail
scope=self._resnet_scope)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/nets/resnet_v2.py", line 221, in resnet_v2
net, activation_fn=nn_ops.relu, scope='postnorm')
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
return func(*args, current_args)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 643, in batch_norm
outputs = layer.apply(inputs, training=is_training)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 671, in apply
return self.call(inputs, args, kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 559, in call
self.build(input_shapes[0])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/normalization.py", line 189, in build
trainable=True)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 458, in add_variable
trainable=trainable and self.trainable)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 1203, in get_variable
constraint=constraint)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 1092, in get_variable
constraint=constraint)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 417, in get_variable
return custom_getter(custom_getter_kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1539, in layer_variable_getter
return _model_variable_getter(getter, args, kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1531, in _model_variable_getter
custom_getter=getter, use_resource=use_resource)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
return func(*args, *current_args)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 262, in model_variable
use_resource=use_resource)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args
return func(args, **current_args)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 217, in variable
use_resource=use_resource)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 394, in _true_getter
use_resource=use_resource, constraint=constraint)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variable_scope.py", line 742, in _get_single_variable
name, "".join(traceback.format_list(tb))))
ValueError: Variable resnet_v2_50/postnorm/gamma already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:
float
tonp.floating
is deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type
. from ._conv import register_converters as _register_converters Couldn't import dot_parser, loading of dot files will not be possible. ------ called with args: ------- Namespace(cfg_file='scripts/dense_cap_config.yml', context_fusion=False, data_dir='/home/xiewei/densecap-tensorflow/visual_genome', device='gpu', device_id=0, embed_dim=512, imdb_name='vg_1.2_train', imdbval_name='vg_1.2_val', max_iters=200000, net='resnet_v2_50', randomize=False, set_cfgs=['EXP_DIR', 'dc_conv_fixed', 'CONTEXT_FUSION', 'False', 'RESNET.FIXED_BLOCKS', '3'], tag=None, weights='/home/xiewei/densecap-tensorflow/slim_models/resnet_v2_50.ckpt') runing with LIMIT_RAM: True Using config: {'ALL_TEST': False, 'ALL_TEST_NUM_TEST': 1000, 'ALL_TEST_NUM_TRAIN': 100, 'ALL_TEST_NUM_VAL': 100, 'ANCHOR_RATIOS': [0.5, 1, 2], 'ANCHOR_SCALES': [4, 8, 16, 32], 'CACHE_DIR': '/home/xiewei/densecap-tensorflow/visual_genome/1.2', 'CONTEXT_FUSION': False, 'CONTEXT_FUSION_MODE': 'sum', 'CONTEXT_MODE': 'concat', 'DATA_DIR': '/home/xiewei/densecap-tensorflow/visual_genome', 'DEBUG_ALL': False, 'EMBED_DIM': 512, 'END_INDEX': 2, 'EXP_DIR': 'dc_conv_fixed', 'FILTER_SMALL_BOX': False, 'GLOVE_DIM': 300, 'GPU_ID': 0, 'INIT_BY_GLOVE': False, 'KEEP_AS_GLOVE_DIM': False, 'LIMIT_RAM': True, 'LOG_DIR': '/home/xiewei/densecap-tensorflow/logs', 'LOSS': {'BBOX_W': 0.01, 'CAP_W': 1.0, 'CLS_W': 0.1, 'RPN_BBOX_W': 0.05, 'RPN_CLS_W': 0.1}, 'MAX_WORDS': 10, 'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]), 'POOLING_MODE': 'crop', 'POOLING_SIZE': 7, 'RESNET': {'FIXED_BLOCKS': 3, 'MAX_POOL': False}, 'RNG_SEED': 3, 'ROOT_DIR': '/home/xiewei/densecap-tensorflow', 'RPN_CHANNELS': 512, 'SAMPLE_NUM_FIXED_REGIONS': False, 'SPLIT_DIR': '/home/xiewei/densecap-tensorflow/info', 'TEST': {'BBOX_REG': True, 'BEAM_SIZE': 3, 'HAS_RPN': True, 'LN_FACTOR': 0.0, 'MAX_SIZE': 720, 'MODE': 'nms', 'NMS': 0.5, 'PROPOSAL_METHOD': 'gt', 'RPN_MIN_SIZE': 16, 'RPN_NMS_THRESH': 0.6, 'RPN_POST_NMS_TOP_N': 300, 'RPN_PRE_NMS_TOP_N': 6000, 'RPN_TOP_N': 5000, 'SCALES': [600], 'SVM': False, 'USE_BEAM_SEARCH': False}, 'TIME_STEPS': 12, 'TRAIN': {'ASPECT_GROUPING': True, 'BATCH_SIZE': 256, 'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0], 'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0], 'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2], 'BBOX_NORMALIZE_TARGETS': True, 'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True, 'BBOX_REG': True, 'BBOX_THRESH': 0.5, 'BG_THRESH_HI': 0.5, 'BG_THRESH_LO': 0.0, 'BIAS_DECAY': False, 'CLIP_NORM': 40.0, 'DISPLAY': 10, 'DOUBLE_BIAS': False, 'EXP_DECAY_RATE': 0.9, 'EXP_DECAY_STEPS': 5000, 'FG_FRACTION': 0.5, 'FG_THRESH': 0.5, 'GAMMA': 0.5, 'HAS_RPN': True, 'IMS_PER_BATCH': 1, 'LEARNING_RATE': 0.001, 'LR_DIY_DECAY': True, 'MAX_SIZE': 720, 'MOMENTUM': 0.98, 'OPTIMIZER': 'sgd_m', 'PROPOSAL_METHOD': 'gt', 'RPN_BATCHSIZE': 256, 'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0], 'RPN_CLOBBER_POSITIVES': False, 'RPN_FG_FRACTION': 0.5, 'RPN_MIN_SIZE': 16, 'RPN_NEGATIVE_OVERLAP': 0.3, 'RPN_NMS_THRESH': 0.7, 'RPN_POSITIVE_OVERLAP': 0.7, 'RPN_POSITIVE_WEIGHT': -1.0, 'RPN_POST_NMS_TOP_N': 2000, 'RPN_PRE_NMS_TOP_N': 12000, 'SCALES': [600], 'SNAPSHOT_ITERS': 5000, 'SNAPSHOT_KEPT': 3, 'SNAPSHOT_PREFIX': 'res50_densecap', 'STEPSIZE': [100000], 'SUMMARY_INTERVAL': 10, 'USE_FLIPPED': True, 'USE_GT': False, 'WEIGHT_DECAY': 0.0001, 'WEIGHT_INITIALIZER': 'normal'}, 'TRAIN_GLOVE': False, 'USE_GPU_NMS': True, 'VOCAB_END_ID': 2, 'VOCAB_SIZE': 10000, 'VOCAB_START_ID': 1} data_path: /home/xiewei/densecap-tensorflow/visual_genome/1.2 loading splits from /home/xiewei/densecap-tensorflow/info/densecap_splits.json Number of examples: 77398 train gt roidb could be loaded from /home/xiewei/densecap-tensorflow/visual_genome/1.2_cache/train_gt_roidb Getting gt roidb and number of examples is:154796 output will be saved to/home/xiewei/densecap-tensorflow/output/dc_conv_fixed/vg_1.2_train
TensorFlow summaries will be saved to/home/xiewei/densecap-tensorflow/output/dc_conv_fixed/tb/vg_1.2_train/default
data_path: /home/xiewei/densecap-tensorflow/visual_genome/1.2 loading splits from /home/xiewei/densecap-tensorflow/info/densecap_splits.json Number of examples: 5000 val gt roidb could be loaded from /home/xiewei/densecap-tensorflow/visual_genome/1.2_cache/val_gt_roidb Getting gt roidb and number of examples is:5000 2018-08-03 09:38:39.720107: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2018-08-03 09:38:40.880941: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: name: Tesla P4 major: 6 minor: 1 memoryClockRate(GHz): 1.1135 pciBusID: 0000:01:00.0 totalMemory: 7.43GiB freeMemory: 7.32GiB 2018-08-03 09:38:40.881009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Tesla P4, pci bus id: 0000:01:00.0, compute capability: 6.1) Solving... LIMIT_RAM version and load index from /home/xiewei/densecap-tensorflow/visual_genome/1.2_cache/train_gt_roidb/image_index.json LIMIT_RAM version and load index from /home/xiewei/densecap-tensorflow/visual_genome/1.2_cache/val_gt_roidb/image_index.json Fixing 3 blocks. Traceback (most recent call last): File "./tools/train_net.py", line 221, inFile "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 217, in variable use_resource=use_resource) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 181, in func_with_args return func(*args, **current_args) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 262, in model_variable use_resource=use_resource)