Problem with getting reproducible results, set seed object detection API

[mmikolajewicz-sifd]

I am using object detection API with tensorflow v1.12. I am having troubles getting reproducible results - each time I run my code I am getting different results. Is there any way to set random seed at training / prediction level? I tried to set seed in model_main.py, but it didn't help.

def main(unused_argv): tf.random.set_random_seed(1234) flags.mark_flag_as_required('model_dir') flags.mark_flag_as_required('pipeline_config_path') config = tf.estimator.RunConfig(model_dir=FLAGS.model_dir, tf_random_seed=1234)

My pipeline.config for reference:

model { faster_rcnn { num_classes: 1 image_resizer { keep_aspect_ratio_resizer { min_dimension: 600 max_dimension: 1024 } } feature_extractor { type: "faster_rcnn_resnet101" first_stage_features_stride: 16 } first_stage_anchor_generator { grid_anchor_generator { height_stride: 16 width_stride: 16 scales: 0.25 scales: 0.5 scales: 1.0 scales: 2.0 aspect_ratios: 0.5 aspect_ratios: 1.0 aspect_ratios: 2.0 } } first_stage_box_predictor_conv_hyperparams { op: CONV regularizer { l2_regularizer { weight: 0.0 } } initializer { truncated_normal_initializer { stddev: 0.0099999998 } } } first_stage_nms_score_threshold: 0.0 first_stage_nms_iou_threshold: 0.69999999 first_stage_max_proposals: 100 first_stage_localization_loss_weight: 2.0 first_stage_objectness_loss_weight: 1.0 initial_crop_size: 14 maxpool_kernel_size: 2 maxpool_stride: 2 second_stage_box_predictor { mask_rcnn_box_predictor { fc_hyperparams { op: FC regularizer { l2_regularizer { weight: 0.0 } } initializer { variance_scaling_initializer { factor: 1.0 uniform: true mode: FAN_AVG } } } use_dropout: false dropout_keep_probability: 1.0 } } second_stage_post_processing { batch_non_max_suppression { score_threshold: 0.30000001 iou_threshold: 0.60000002 max_detections_per_class: 100 max_total_detections: 100 } score_converter: SOFTMAX } second_stage_localization_loss_weight: 2.0 second_stage_classification_loss_weight: 1.0 } } train_config { batch_size: 1 data_augmentation_options { random_horizontal_flip { } } optimizer { momentum_optimizer { learning_rate { manual_step_learning_rate { initial_learning_rate: 0.00030000001 schedule { step: 814096 learning_rate: 2.9999999e-05 } } } momentum_optimizer_value: 0.89999998 } use_moving_average: false } gradient_clipping_by_norm: 10.0 fine_tune_checkpoint: "/home/kombajn/tensorflow/models/research/object_detection/2020_20_11/model_to_send/model.ckpt" from_detection_checkpoint: false num_steps: 30000 } train_input_reader { label_map_path: "/home/kombajn/tensorflow/models/research/object_detection/2020_22_12_test/data/pack.pbtxt" tf_record_input_reader { input_path: "/home/kombajn/tensorflow/models/research/object_detection/2020_22_12_test/data/train.record" } } eval_config { num_examples: 100 use_moving_averages: false } eval_input_reader { label_map_path: "/home/kombajn/tensorflow/models/research/object_detection/2020_22_12_test/data/pack.pbtxt" shuffle: false num_readers: 1 tf_record_input_reader { input_path: "/home/kombajn/tensorflow/models/research/object_detection/2020_22_12_test/data/eval.record" } }

[jvishnuvardhan]

@mmikolajewicz-sifd Can you please try recent Tensorflow version (TF1.15.3 as your are using TF1.x) and let us know whether the issue is same with latest version. Thanks!

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[mmikolajewicz-sifd]

@mmikolajewicz-sifd Can you please try recent Tensorflow version (TF1.15.3 as your are using TF1.x) and let us know whether the issue is same with latest version. Thanks!

Thanks for your response. According to your suggestion, I uptaded the version of tensorflow and tensorflow-gpu to 1.15.3, but my results still differ.