Prerequisites

Please answer the following questions for yourself before submitting an issue.

[X ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
[ X] I am reporting the issue to the correct repository. (Model Garden official or research directory)
[ X] I checked to make sure that this issue has not been filed already.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): example script
OS Platform and Distribution : win10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): bin
TensorFlow version (use command below): tensorflow-cpu 2.4.1
Python version: 3.8.8
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory: intel integrated

Describe the current behavior

(tensorflow) C:\Users\kacpe\Desktop\tensorflow\models\research\object_detection>python model_main_tf2.py --pipeline_config_path=training/pipeline.config --model_dir=training --alsologtostderr 2021-04-06 14:21:49.605417: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found 2021-04-06 14:21:49.605814: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2021-04-06 14:22:08.223915: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-04-06 14:22:08.229736: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found 2021-04-06 14:22:08.230068: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303) 2021-04-06 14:22:08.242115: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-HO16S6U 2021-04-06 14:22:08.242701: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-HO16S6U 2021-04-06 14:22:08.244472: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-04-06 14:22:08.245897: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set WARNING:tensorflow:There are non-GPU devices in tf.distribute.Strategy, not using nccl allreduce. W0406 14:22:08.247780 9044 cross_device_ops.py:1321] There are non-GPU devices in tf.distribute.Strategy, not using nccl allreduce. INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) I0406 14:22:08.247780 9044 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',) INFO:tensorflow:Maybe overwriting train_steps: None I0406 14:22:08.263412 9044 config_util.py:552] Maybe overwriting train_steps: None INFO:tensorflow:Maybe overwriting use_bfloat16: False I0406 14:22:08.263412 9044 config_util.py:552] Maybe overwriting use_bfloat16: False I0406 14:22:08.325909 9044 ssd_efficientnet_bifpn_feature_extractor.py:142] EfficientDet EfficientNet backbone version: efficientnet-b1 I0406 14:22:08.325909 9044 ssd_efficientnet_bifpn_feature_extractor.py:144] EfficientDet BiFPN num filters: 88 I0406 14:22:08.325909 9044 ssd_efficientnet_bifpn_feature_extractor.py:145] EfficientDet BiFPN num iterations: 4 I0406 14:22:08.357155 9044 efficientnet_model.py:147] round_filter input=32 output=32 I0406 14:22:08.510153 9044 efficientnet_model.py:147] round_filter input=32 output=32 I0406 14:22:08.510153 9044 efficientnet_model.py:147] round_filter input=16 output=16 I0406 14:22:09.333078 9044 efficientnet_model.py:147] round_filter input=16 output=16 I0406 14:22:09.333078 9044 efficientnet_model.py:147] round_filter input=24 output=24 I0406 14:22:10.936553 9044 efficientnet_model.py:147] round_filter input=24 output=24 I0406 14:22:10.936553 9044 efficientnet_model.py:147] round_filter input=40 output=40 I0406 14:22:12.487046 9044 efficientnet_model.py:147] round_filter input=40 output=40 I0406 14:22:12.487046 9044 efficientnet_model.py:147] round_filter input=80 output=80 I0406 14:22:14.495594 9044 efficientnet_model.py:147] round_filter input=80 output=80 I0406 14:22:14.495594 9044 efficientnet_model.py:147] round_filter input=112 output=112 I0406 14:22:16.817428 9044 efficientnet_model.py:147] round_filter input=112 output=112 I0406 14:22:16.817428 9044 efficientnet_model.py:147] round_filter input=192 output=192 I0406 14:22:19.744267 9044 efficientnet_model.py:147] round_filter input=192 output=192 I0406 14:22:19.744267 9044 efficientnet_model.py:147] round_filter input=320 output=320 I0406 14:22:20.961132 9044 efficientnet_model.py:147] round_filter input=1280 output=1280 I0406 14:22:21.257415 9044 efficientnet_model.py:458] Building model efficientnet with params ModelConfig(width_coefficient=1.0, depth_coefficient=1.1, resolution=240, dropout_rate=0.2, blocks=(BlockConfig(input_filters=32, output_filters=16, kernel_size=3, num_repeat=1, expand_ratio=1, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=16, output_filters=24, kernel_size=3, num_repeat=2, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=24, output_filters=40, kernel_size=5, num_repeat=2, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=40, output_filters=80, kernel_size=3, num_repeat=3, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=80, output_filters=112, kernel_size=5, num_repeat=3, expand_ratio=6, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=112, output_filters=192, kernel_size=5, num_repeat=4, expand_ratio=6, strides=(2, 2), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise'), BlockConfig(input_filters=192, output_filters=320, kernel_size=3, num_repeat=1, expand_ratio=6, strides=(1, 1), se_ratio=0.25, id_skip=True, fused_conv=False, conv_type='depthwise')), stem_base_filters=32, top_base_filters=1280, activation='simple_swish', batch_norm='default', bn_momentum=0.99, bn_epsilon=0.001, weight_decay=5e-06, drop_connect_rate=0.2, depth_divisor=8, min_depth=None, use_se=True, input_channels=3, num_classes=1000, model_name='efficientnet', rescale_input=False, data_format='channels_last', dtype='float32') WARNING:tensorflow:From C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py:545: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version. Instructions for updating: rename to distribute_datasets_from_function W0406 14:22:21.567980 9044 deprecation.py:333] From C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py:545: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version. Instructions for updating: rename to distribute_datasets_from_function Traceback (most recent call last): File "model_main_tf2.py", line 113, in tf.compat.v1.app.run() File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\absl\app.py", line 303, in run _run_main(main, args) File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\absl\app.py", line 251, in _run_main sys.exit(main(argv)) File "model_main_tf2.py", line 104, in main model_lib_v2.train_loop( File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py", line 545, in train_loop train_input = strategy.experimental_distribute_datasets_from_function( File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\util\deprecation.py", line 340, in new_func return func(*args, **kwargs) File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 1143, in experimental_distribute_datasets_from_function return self.distribute_datasets_from_function(dataset_fn, options) File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 1134, in distribute_datasets_from_function return self._extended._distribute_datasets_from_function( # pylint: disable=protected-access File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\distribute\mirrored_strategy.py", line 545, in _distribute_datasets_from_function return input_lib.get_distributed_datasets_from_function( File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 161, in get_distributed_datasets_from_function return DistributedDatasetsFromFunction(dataset_fn, input_workers, File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 1272, in init _create_datasets_from_function_with_input_context( File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 1936, in _create_datasets_from_function_with_input_context dataset = dataset_fn(ctx) File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py", line 536, in train_dataset_fn train_input = inputs.train_input( File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\inputs.py", line 893, in train_input dataset = INPUT_BUILDER_UTIL_MAP['dataset_build']( File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\builders\dataset_builder.py", line 210, in build decoder = decoder_builder.build(input_reader_config) File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\builders\decoder_builder.py", line 52, in build decoder = tf_example_decoder.TfExampleDecoder( File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\data_decoders\tf_example_decoder.py", line 414, in init _ClassTensorHandler( File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\data_decoders\tf_example_decoder.py", line 88, in init name_to_id = label_map_util.get_label_map_dict( File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\utils\label_map_util.py", line 201, in get_label_map_dict label_map = load_labelmap(label_map_path_or_proto) File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\object_detection\utils\label_map_util.py", line 168, in load_labelmap label_map_string = fid.read() File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 117, in read self._preread_check() File "C:\Users\kacpe\anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 79, in _preread_check self._read_buf = _pywrap_file_io.BufferedInputStream( UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 142: invalid start byte

Describe the expected behavior

train the model

Standalone code to reproduce the issue Provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/Jupyter/any notebook.

pipeline.config :

`model { ssd { num_classes: 1 image_resizer { keep_aspect_ratio_resizer { min_dimension: 640 max_dimension: 640 pad_to_max_dimension: true } } feature_extractor { type: "ssd_efficientnet-b1_bifpn_keras" conv_hyperparams { regularizer { l2_regularizer { weight: 3.9999998989515007e-05 } } initializer { truncated_normal_initializer { mean: 0.0 stddev: 0.029999999329447746 } } activation: SWISH batch_norm { decay: 0.9900000095367432 scale: true epsilon: 0.0010000000474974513 } force_use_bias: true } bifpn { min_level: 3 max_level: 7 num_iterations: 4 num_filters: 88 } } box_coder { faster_rcnn_box_coder { y_scale: 1.0 x_scale: 1.0 height_scale: 1.0 width_scale: 1.0 } } matcher { argmax_matcher { matched_threshold: 0.5 unmatched_threshold: 0.5 ignore_thresholds: false negatives_lower_than_unmatched: true force_match_for_each_row: true use_matmul_gather: true } } similarity_calculator { iou_similarity { } } box_predictor { weight_shared_convolutional_box_predictor { conv_hyperparams { regularizer { l2_regularizer { weight: 3.9999998989515007e-05 } } initializer { random_normal_initializer { mean: 0.0 stddev: 0.009999999776482582 } } activation: SWISH batch_norm { decay: 0.9900000095367432 scale: true epsilon: 0.0010000000474974513 } force_use_bias: true } depth: 88 num_layers_before_predictor: 3 kernel_size: 3 class_prediction_bias_init: -4.599999904632568 use_depthwise: true } } anchor_generator { multiscale_anchor_generator { min_level: 3 max_level: 7 anchor_scale: 4.0 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 scales_per_octave: 3 } } post_processing { batch_non_max_suppression { score_threshold: 9.99999993922529e-09 iou_threshold: 0.5 max_detections_per_class: 100 max_total_detections: 100 } score_converter: SIGMOID } normalize_loss_by_num_matches: true loss { localization_loss { weighted_smooth_l1 { } } classification_loss { weighted_sigmoid_focal { gamma: 1.5 alpha: 0.25 } } classification_weight: 1.0 localization_weight: 1.0 } encode_background_as_zeros: true normalize_loc_loss_by_codesize: true inplace_batchnorm_update: true freeze_batchnorm: false add_background_class: false } } train_config { batch_size: 8 data_augmentation_options { random_horizontal_flip { } } data_augmentation_options { random_scale_crop_and_pad_to_square { output_size: 640 scale_min: 0.10000000149011612 scale_max: 2.0 } } sync_replicas: true optimizer { momentum_optimizer { learning_rate { cosine_decay_learning_rate { learning_rate_base: 0.07999999821186066 total_steps: 300000 warmup_learning_rate: 0.0010000000474974513 warmup_steps: 2500 } } momentum_optimizer_value: 0.8999999761581421 } use_moving_average: false } fine_tune_checkpoint: "C:/Users/kacpe/Desktop/tensorflow/models/research/object_detection/efficientdet_d1_coco17_tpu-32/checkpoint/ckpt-0" num_steps: 300000 startup_delay_steps: 0.0 replicas_to_aggregate: 8 max_number_of_boxes: 100 unpad_groundtruth_tensors: false fine_tune_checkpoint_type: "detection" use_bfloat16: true fine_tune_checkpoint_version: V2 } train_input_reader: { label_map_path: "C:/Users/kacpe/Desktop/tensorflow/models/research/object_detection/training/labelmap.bptxt" tf_record_input_reader { input_path: "C:/Users/kacpe/Desktop/tensorflow/models/research/object_detection/images/train.record" } }

eval_config: { metrics_set: "coco_detection_metrics" use_moving_averages: false batch_size: 1; }

eval_input_reader: { label_map_path: "C:/Users/kacpe/Desktop/tensorflow/models/research/object_detection/training/labelmap.bptxt" shuffle: false num_epochs: 1 tf_record_input_reader { input_path: "C:/Users/kacpe/Desktop/tensorflow/models/research/object_detection/images/test.record" } } `

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

tensorflow / models

model_main_tf2.py -> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 142: invalid start byte #9875

Prerequisites