Wrongs weights for pretrained models

[frost-is]

System information.

• Have I written custom code (as opposed to using a stock example script provided in Keras): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
• TensorFlow installed from (source or binary): apt
• TensorFlow version (use command below): 2.10
• Python version: 3.8
• Bazel version (if compiling from source): N/A
• GPU model and memory: NVidia A100 - 40GB
• Exact command to reproduce: below

Describe the problem.

Hi,

I have recently begun tinkering with ILSVRC2012. I have noticed that some of the pretrained models that keras offers infer random noise.

I haven't tried everything, but on my setup, Mobilenetv3 Small, MobileNetV3 Large and EfficientNetv2 S work fine, with train accuracies over 70%. However, Inception ResnetV2, NasNet Mobile and NasNet Large have train accuracies of just about 0.1%.

The functions I use to import the models are the same as in the keras documentation

resnetv2 = tensorflow.keras.applications.inception_resnet_v2(
    include_top=True, weights='imagenet', input_tensor=None,
    input_shape=None, pooling=None, classes=1000,
    classifier_activation='softmax')

mobilenet_small = tf.keras.applications.MobileNetV3Small(
    input_shape=None, alpha=1.0, minimalistic=False, include_top=True,
    weights='imagenet', input_tensor=None, classes=1000, pooling=None,
    dropout_rate=0.2, classifier_activation='softmax',
    include_preprocessing=True
)

train_dataset_224 = tf.keras.utils.image_dataset_from_directory(
    ILSVRC2012_train, labels='inferred', label_mode='int',
    class_names=None, color_mode='rgb', batch_size=32, image_size=(224,
    224), shuffle=True, seed=None, validation_split=None, subset=None,
    interpolation='bilinear', follow_links=False,
    crop_to_aspect_ratio=False
)

train_dataset_299 = tf.keras.utils.image_dataset_from_directory(
    path_train, labels='inferred', label_mode='int',
    class_names=None, color_mode='rgb', batch_size=32, image_size=(299,
    299), shuffle=True, seed=None, validation_split=None, subset=None,
    interpolation='bilinear', follow_links=False,
    crop_to_aspect_ratio=False
)

mobilenet_small.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
_, baseline_model_accuracy = mobilenet_small.evaluate(train_dataset_224.take(1000), verbose=2)

resnetv2.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
_, baseline_model_accuracy = resnetv2.evaluate(train_dataset_299.take(1000), verbose=2)

(I only infer on 1000 images on this example but it is the same when infering on the entire dataset)

The outputs I get are 1000/1000 - 10s - loss: 1.0272 - accuracy: 0.7647 - 10s/epoch - 10ms/step for mobilenet and 1000/1000 - 47s - loss: 1020.6038 - accuracy: 7.1875e-04 - 47s/epoch - 47ms/step for InceptionResnetv2

If you need to experiment on my code I'll try to create a colab - hopefully it's just a problem with the backend loading wrong weights.

Thank you very much for your time !

[sampathweb]

@frost-is - Thanks for raising the issue and providing code snippets.

Performance issue you observe in resnet_v2 is due to the missing step of preprocess_input that squashes the inputs to -1 to 1. MobileNetV3Small model has an optional parameter include_preprocessing that you set to True so you don't need any additional step to pre-process input.

Please apply the preprocess_input in your resnet_v2 pipeline and let us know if you still see any degradation in accuracy metric.

Docs: https://www.tensorflow.org/api_docs/python/tf/keras/applications/resnet_v2/preprocess_input https://keras.io/api/applications/resnet/

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No