Add `FeaturePyramidBackbone` and port weights from `timm` for `ResNetBackbone`

[james77777778]

This PR introduces FeaturePyramidBackbone, a wrapper for Backbone that adds pyramid_outputs property. If a vision backbone supports feature pyramids, it should subclass FeaturePyramidBackbone.

I modified ResNetBackbone by subclassing FeaturePyramidBackbone to include feature pyramid information.

Also, head_dtype is added in ResNetImageClassifier.

@divyashreepathihalli @mattdangerw @SamanehSaadat

EDITED: See https://github.com/keras-team/keras-nlp/pull/1769#issuecomment-2287967827 for updates

[mattdangerw]

@james77777778 I was thinking we would make FeaturePyramidBackbone a simple subclass of Backbone for most CV backbones.

So...

• Backbone is basically just a functional model with from_preset.
• FeaturePyramidBackbone extends Backbone with extra pyramid_outputs.
• ResNetBackbone extends FeaturePyramidBackbone directly.

The goal is to keep the Backbone base class as clean as we can, now that we are venturing into multi modal models with a lot of different overall patterns. dir(bert_backbone) shouldn't have feature pyramid stuff in it. If we could move the token_embedding off the backbone and into a TextBackbone or similar without breaking compat we probably would too.

Most CV models like ResNet, DenseNet, EfficientNet, etc, will be FeaturePyramidBackbones. But something like ViT can just subclass Backbone directly without needing the feature pyramid outputs.

WDYT?

[james77777778]

@mattdangerw @divyashreepathihalli Got it. That makes sense. Updated!

I have updated the PR to make compatible with timm. Additionally, the conversion logic has been added, similar to how it’s done in transformers.

Please refer to this colab for the numerical check: https://colab.research.google.com/drive/1QnmNDiFYd56fsYoaUM46QRT4gF9G06fH?usp=sharing

Supported:

• V1: resnet18.a1_in1k, resnet26.bt_in1k, resnet34.a1_in1k, resnet50.a1_in1k, resnet101.a1h_in1k, resnet152.a1h_in1k
• V2: resnetv2_50.a1h_in1k, resnetv2_101.a1h_in1k

[divyashreepathihalli]

Looks really good! Its awesome to have built in timm conversion! LGTM!

[mattdangerw]

Thanks!