WARNING:tensorflow:Gradients do not exist for variables...when minimizing the loss.

[nikgarhwal]

I was trying to train the "efficientdet_d1_coco17_tpu-32" model on custom dataset using "model_main_tf2.py". My code was working just fine till last month but im encountering this warning now : " WARNING:tensorflow:Gradients do not exist for variables ['stack_6/block_1/expand_bn/gamma:0', 'stack_6/block_1/expand_bn/beta:0', 'stack_6/block_1/depthwise_conv2d/depthwise_kernel:0', 'stack_6/block_1/depthwise_bn/gamma:0', 'stack_6/block_1/depthwise_bn/beta:0', 'stack_6/block_1/project_bn/gamma:0', 'stack_6/block_1/project_bn/beta:0', 'top_bn/gamma:0', 'top_bn/beta:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a loss argument? W1222 12:51:32.223813 140304483087936 utils.py:82] Gradients do not exist for variables ['stack_6/block_1/expand_bn/gamma:0', 'stack_6/block_1/expand_bn/beta:0', 'stack_6/block_1/depthwise_conv2d/depthwise_kernel:0', 'stack_6/block_1/depthwise_bn/gamma:0', 'stack_6/block_1/depthwise_bn/beta:0', 'stack_6/block_1/project_bn/gamma:0', 'stack_6/block_1/project_bn/beta:0', 'top_bn/gamma:0', 'top_bn/beta:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a loss argument? " does anyone know what it means or how can i resolve this? colab file link:

[laxmareddyp]

Hi @nikgarhwal,

To assist you more efficiently, could you please share a code sample or a link to a Google Colab notebook that reproduces the issue you're experiencing? Additionally, it would be helpful to know whether you are using the official TensorFlow Model Garden repository or a research-based one.

Thanks.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[nikgarhwal]

I’m using the efficient diet d1 model from the official tf2 model zoo repository. But I tried to configure the model by tuning the parameters.

Here’s the colab file link:

https://colab.research.google.com/drive/1suzuFZrXo97x-91u1RM86VQ0jjo1f0gi?usp=share_link

Here’s my configured pipeline link, that I’m using to train the model :

https://drive.google.com/file/d/1-XWY4Uc6OKWrOTu5TCrBpMxw6EVlplpt/view?usp=share_link

[laxmareddyp]

Hi @nikgarhwal ,

I strongly suggest utilizing the TensorFlow Official Model Garden to circumvent issues related to outdated code commonly found in research codebases. Unlike the research repositories, the Official Model Garden is consistently updated and aligned with the latest changes in TensorFlow and other libraries and there are lot of API's are available that you can define a training experiment using Python commands in the TensorFlow Model library.We have developed several Notebook examples like object detection that illustrate how to train models from the ground up.

Thanks.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[shivam-bme]

Hi @laxmareddyp i were using tensorflow model garden repositry for model developement and along with tensorflow 2.0 object detection api tutorial i also modified suggested change in pipeline.config file , your your assistantance here is model configuration file

"model { ssd { num_classes: 1 image_resizer { keep_aspect_ratio_resizer { min_dimension: 1024 max_dimension: 1024 pad_to_max_dimension:true } } feature_extractor { type: "ssd_efficientnet-b4_bifpn_keras" conv_hyperparams { regularizer { l2_regularizer { weight: 3.9999998989515007e-05 } } initializer { truncated_normal_initializer { mean: 0.0 stddev: 0.029999999329447746 } } activation: SWISH batch_norm { decay: 0.9900000095367432 scale: true epsilon: 0.0010000000474974513 } force_use_bias: true } bifpn { min_level: 3 max_level: 7 num_iterations: 7 num_filters: 224 } } box_coder { faster_rcnn_box_coder { y_scale: 1.0 x_scale: 1.0 height_scale: 1.0 width_scale: 1.0 } } matcher { argmax_matcher { matched_threshold: 0.5 unmatched_threshold: 0.5 ignore_thresholds: false negatives_lower_than_unmatched: true force_match_for_each_row: true use_matmul_gather: true } } similarity_calculator { iou_similarity { } } box_predictor { weight_shared_convolutional_box_predictor { conv_hyperparams { regularizer { l2_regularizer { weight: 3.9999998989515007e-05 } } initializer { random_normal_initializer { mean: 0.0 stddev: 0.009999999776482582 } } activation: SWISH batch_norm { decay: 0.9900000095367432 scale: true epsilon: 0.0010000000474974513 } force_use_bias: true } depth: 224 num_layers_before_predictor: 4 kernel_size: 3 class_prediction_bias_init: -4.599999904632568 use_depthwise: true } } anchor_generator { multiscale_anchor_generator { min_level: 3 max_level: 7 anchor_scale: 4.0 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 scales_per_octave: 3 } } post_processing { batch_non_max_suppression { score_threshold: 9.99999993922529e-09 iou_threshold: 0.5 max_detections_per_class: 100 max_total_detections: 100 } score_converter: SIGMOID } normalize_loss_by_num_matches: true loss { localization_loss { weighted_smooth_l1 { } } classification_loss { weighted_sigmoid_focal { gamma: 1.5 alpha: 0.25 } } classification_weight: 1.0 localization_weight: 1.0 } encode_background_as_zeros: true normalize_loc_loss_by_codesize: true inplace_batchnorm_update: true freeze_batchnorm: false add_background_class: false } } train_config { batch_size: 16 data_augmentation_options { random_horizontal_flip { } } data_augmentation_options { random_scale_crop_and_pad_to_square { output_size: 1024 scale_min: 0.10000000149011612 scale_max: 2.0 } } sync_replicas: true optimizer { momentum_optimizer { learning_rate { cosine_decay_learning_rate { learning_rate_base: 0.07999999821186066 total_steps: 30000 warmup_learning_rate: 0.0010000000474974513 warmup_steps: 2500 } } momentum_optimizer_value: 0.8999999761581421 } use_moving_average: false

} fine_tune_checkpoint: "/home/shivam_singh/models/workspace/training_demo/pre-trained_models/efficientdet_d4_coco17_tpu-32/checkpoint/ckpt-0" num_steps: 3000 startup_delay_steps: 0.0 replicas_to_aggregate: 8 max_number_of_boxes: 100 unpad_groundtruth_tensors: false fine_tune_checkpoint_type: "detection" use_bfloat16: false fine_tune_checkpoint_version: V2 } train_input_reader: { label_map_path: "/home/shivam_singh/models/workspace/training_demo/annotations/label_map.pbtxt" tf_record_input_reader { input_path: "/media/shivam_singh/New Volume/DownloadsMain/dataset_node21/cxr_images/original_data/tfrecords/test.record" } }

eval_config: { metrics_set: "coco_detection_metrics" use_moving_averages: false batch_size: 1; }

eval_input_reader: { label_map_path: "/home/shivam_singh/models/workspace/training_demo/annotations/label_map.pbtxt" shuffle: false num_epochs: 1 tf_record_input_reader { input_path: "/media/shivam_singh/New Volume/DownloadsMain/dataset_node21/cxr_images/original_data/tfrecords/test.record" } }"