rky0930
commented
6 years ago Hi. @ilkarman I using python2.7 as well. I want to reproduce this Error. Can you post your config file?
Open ilkarman opened 6 years ago
rky0930
commented
6 years ago Hi. @ilkarman I using python2.7 as well. I want to reproduce this Error. Can you post your config file?
ilkarman
commented
6 years ago Sorry for not being clear - I was using this was py35 and tensorflow-gpu==1.4 and trying to figure out if the error was due to the script being for py2 (since it assumes zip has length) or some other reason. After restarting everything my error changes to OOM which is not true (I reduced batch to 2 and should be able to fit in around 64):
I don't know how it comes up with this shape:
[2,1,3968,2976,3]
Unless it forgets to resize my image? (BTW I've run this dataset on SSD, Faster-RCNN, etc many times on the OD-API)
File "/anaconda/envs/py35/lib/python3.5/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 775, in train
sv.stop(threads, close_summary_writer=True)
File "/anaconda/envs/py35/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "/anaconda/envs/py35/lib/python3.5/site-packages/tensorflow/python/training/supervisor.py", line 964, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/anaconda/envs/py35/lib/python3.5/site-packages/tensorflow/python/training/supervisor.py", line 792, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "/anaconda/envs/py35/lib/python3.5/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/anaconda/envs/py35/lib/python3.5/site-packages/six.py", line 693, in reraise
raise value
File "/anaconda/envs/py35/lib/python3.5/site-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
enqueue_callable()
File "/anaconda/envs/py35/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1231, in _single_operation_run
target_list_as_strings, status, None)
File "/anaconda/envs/py35/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[2,1,3968,2976,3]
[[Node: batch = QueueDequeueManyV2[component_types=[DT_STRING, DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32, DT_INT64, DT_INT32, DT_INT64, DT_INT32, DT_INT64, DT_INT32, DT_BOOL, DT_INT32, DT_FLOAT, DT_INT32, DT_STRING, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](batch/padding_fifo_queue, batch/n)]]My config is not really altered in a big way at all:
# YOLO with Darknet19 configuration for MSCOCO Dataset.
# Users should configure the fine_tune_checkpoint field in the train config as
# well as the label_map_path and input_path fields in the train_input_reader and
# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
# should be configured.
model {
ssd {
num_classes: 1
box_coder {
faster_rcnn_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}
matcher {
argmax_matcher {
matched_threshold: 0.5
unmatched_threshold: 0.5
ignore_thresholds: false
negatives_lower_than_unmatched: true
force_match_for_each_row: true
}
}
similarity_calculator {
iou_similarity {
}
}
anchor_generator {
yolo_anchor_generator {
anchors: [1.3221, 1.73145, 3.19275, 4.00944, 5.05587, 8.09892, 9.47112, 4.84053, 11.2364, 10.0071]
}
}
image_resizer {
fixed_shape_resizer {
height: 416
width: 416
}
}
box_predictor {
convolutional_box_predictor {
min_depth: 0
max_depth: 0
num_layers_before_predictor: 0
use_dropout: false
dropout_keep_probability: 0.8
kernel_size: 1
box_code_size: 4
apply_sigmoid_to_scores: false
conv_hyperparams {
activation: NONE,
regularizer {
l2_regularizer {
weight: 0.00004
}
}
initializer {
truncated_normal_initializer {
stddev: 0.03
mean: 0.0
}
}
}
}
}
feature_extractor {
type: 'yolo_v2_darknet_19'
conv_hyperparams {
activation: NONE,
regularizer {
l2_regularizer {
weight: 0.00004
}
}
initializer {
truncated_normal_initializer {
stddev: 0.03
mean: 0.0
}
}
batch_norm {
train: true,
scale: true,
center: true,
decay: 0.9997,
epsilon: 0.001,
}
}
}
loss {
classification_loss {
confidence_weighted_sigmoid {
anchorwise_output: true
object_scale: 5.0,
noobject_scale: 1.0
}
}
localization_loss {
weighted_l2 {
anchorwise_output: true
}
}
hard_example_miner {
num_hard_examples: 3000
iou_threshold: 0.99
loss_type: CLASSIFICATION
max_negatives_per_positive: 3
min_negatives_per_image: 0
}
classification_weight: 1.0
localization_weight: 1.0
}
normalize_loss_by_num_matches: true
post_processing {
batch_non_max_suppression {
score_threshold: 1e-8
iou_threshold: 0.6
max_detections_per_class: 100
max_total_detections: 100
}
score_converter: SIGMOID
}
}
}
train_config: {
batch_size: 2
optimizer {
rms_prop_optimizer: {
learning_rate: {
exponential_decay_learning_rate {
initial_learning_rate: 0.004
decay_steps: 80072
decay_factor: 0.95
}
}
momentum_optimizer_value: 0.9
decay: 0.9
epsilon: 1.0
}
}
fine_tune_checkpoint: "/data/tf_od_api/models/yolov2_with_darknet19/model.ckpt"
from_detection_checkpoint: true
# Note: The below line limits the training process to 200K steps, which we
# empirically found to be sufficient enough to train the pets dataset. This
# effectively bypasses the learning rate schedule (the learning rate will
# never decay). Remove the below line to train indefinitely.
#num_steps: 800000
data_augmentation_options {
random_horizontal_flip {}}
data_augmentation_options {
ssd_random_crop {}}
}
train_input_reader: {
tf_record_input_reader {
input_path: "/data/tf_od_api/data/all_data_final.record"
}
label_map_path: "/data/tf_od_api/data/pipes_map.pbtxt"
}
eval_config: {
num_examples: 100
num_visualizations: 100
}
eval_input_reader: {
tf_record_input_reader {
input_path: "/data/tf_od_api/data/test_big.record"
}
label_map_path: "/data/tf_od_api/data/pipes_map.pbtxt"
shuffle: false
num_readers: 1
num_epochs: 1
}And I used the checkpoint from rky
rky0930
commented
6 years ago I think "ResourceExhaustedError" means out of memory.
Image_resizer config will make your imput to size of 416x416. So input size is not a problem.
image_resizer { fixed_shape_resizer { height: 416 width: 416 } }
Batch_size 2 is relatively small. Did you check memory usage of GPU before run this model ? ex) $ nvidia-smi I can run this model with batch_size 8 on my labtop which is GTX1050 (4Gb memory).
ilkarman
commented
6 years ago @rky0930 thanks for the reply! My GPUs are free and available (11GB on 2xK80), the small batch was just to see if it would work at all.
Can you run this model on py35? I've tried re-uploading the files again from this repo, and changing line 114 in yolo_grid_anchor_generator.py to:
return [len(list(self._anchors))]
And get a diff error:
InvalidArgumentError (see above for traceback): Reduction axis 1 is empty in shape [8,0]
[[Node: Loss/Match_1/cond/ArgMax_1 = ArgMax[T=DT_FLOAT, Tidx=DT_INT32, output_type=DT_INT64, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/Match_1/cond/Shape_2/Switch:1, Loss/Match_1/cond/range/delta)]]
[[Node: Loss/ToInt32_2/_375 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1884_Loss/ToInt32_2", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
If I try py27 I get:
from tensorflow.contrib.learn.python.learn import *
File "/anaconda/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/__init__.py", line 25, in <module>
from tensorflow.contrib.learn.python.learn import estimators
File "/anaconda/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/__init__.py", line 297, in <module>
from tensorflow.contrib.learn.python.learn.estimators.dnn import DNNClassifier
File "/anaconda/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/dnn.py", line 30, in <module>
from tensorflow.contrib.learn.python.learn.estimators import dnn_linear_combined
File "/anaconda/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/dnn_linear_combined.py", line 31, in <module>
from tensorflow.contrib.learn.python.learn.estimators import estimator
File "/anaconda/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 49, in <module>
from tensorflow.contrib.learn.python.learn.learn_io import data_feeder
File "/anaconda/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_io/__init__.py", line 21, in <module>
from tensorflow.contrib.learn.python.learn.learn_io.dask_io import extract_dask_data
File "/anaconda/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_io/dask_io.py", line 26, in <module>
import dask.dataframe as dd
File "/anaconda/lib/python2.7/site-packages/dask/dataframe/__init__.py", line 3, in <module>
from .core import (DataFrame, Series, Index, _Frame, map_partitions,
File "/anaconda/lib/python2.7/site-packages/dask/dataframe/core.py", line 40, in <module>
pd.core.computation.expressions.set_use_numexpr(False)
AttributeError: 'module' object has no attribute 'expressions'In both cases tf==1.4.0
rky0930
commented
6 years ago I changed the line 114 the same as you and run using python2.7. There was no such error.
In my knowledge, Reduction axis 1 is empty in shape [8,0] error is occured when the tf.arg_max ops get empty tensor in the specific axis.
I suggest you check your record and pbtxt file again.
Or, please share the files to me via email. so, i can try to reproduce the error.
I guess python2.7 error is related with post ? https://stackoverflow.com/questions/43833081/attributeerror-module-object-has-no-attribute-computation
I hope your model work well. If the problem occured again. Please let me know :smiley:
iyeszin
commented
6 years ago I get the same error as @ilkarman, and I changes to return [len(list(self._anchors))] in function, however, another error occurs on it.
Traceback (most recent call last):
File "train.py", line 167, in
I'm using py36 and running on gpu. Im using custom data on 2 classes.
I have no idea to solve that error, please help. Many thanks！
rky0930
commented
6 years ago Hi. @iyeszin It happened when you use yolo code with newest object detection api code. I fixed the bugs with simply make the return value to list. Please try again. You just need to re-download "yolo_grid_anchor_generator.py" and run again. If another bugs occurred. please let me know.
xiaowenhe
commented
5 years ago @iyeszin @rky0930 。I also get Reduction axis 1 is empty in shape [2,0] error! I fixed the error like :
self._anchors = []
for i , j in enumerate(anchors):
if i % 2 == 0:
self._anchors.append([anchors[i],anchors[i+1]])in yolo_grid_anchor_generator.py line 86.
I was curious to give this a go and wasn't sure if the below error was due to this being python2 only or something else?
From the function:
I changed to this to make it run:
return [len(list(self._anchors))]But then ran into another issue: