search

huggingface / pytorch-image-models

The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more
https://huggingface.co/docs/timm
Apache License 2.0
32.22k stars 4.75k forks source link

[FEATURE] MaxVit implementation without using custom operators to be able to compile for tensorrt #2329

Closed Skier23 closed 9 hours ago

Skier23 commented 2 days ago

Is your feature request related to a problem? Please describe. Currently the maxvit models in this repo cannot be compiled with tensorrt which would enable massive potential speedups. According to NVidia in this thread, the reason that the model cannot be compiled or converted to tensorrt is because the implementation of MaxVIT in this repo uses custom operators for which there is no registered tensorrt importer. See details: https://github.com/NVIDIA/TensorRT/issues/3734:

[03/25/2024-09:34:30] [I] [TRT] No importer registered for op: timm_models_maxxvit_MaxxVit_base_model_1. Attempting to import as plugin.
[03/25/2024-09:34:30] [I] [TRT] Searching for plugin: timm_models_maxxvit_MaxxVit_base_model_1, plugin_version: 1, plugin_namespace:
[03/25/2024-09:34:30] [E] [TRT] 3: getPluginCreator could not find plugin: timm_models_maxxvit_MaxxVit_base_model_1 version: 1

Follow-up request from https://github.com/huggingface/pytorch-image-models/issues/2129

Describe the solution you'd like If the few custom layers could be replaced with more standard pytorch layers for which there may already be tensorrt converters/importers then hopefully a few layer changes on the timm side could enable the timm MaxVIT models to be supported with tensorrt

Describe alternatives you've considered I've tried every way possible to try to get the MaxVIT model in this repo onto TensorRT but all approaches have failed likely all because of these custom operators: https://forums.developer.nvidia.com/t/struggling-to-get-model-onto-tensorrt/287310

Additional context Add any other context or screenshots about the feature request here.

Skier23 commented 2 days ago

If this is something that would be considered, I'd be happy to assist with testing but my knowledge of the layers within the backbone architecture here is a bit limited thus the request instead of trying to tackle this myself. I created a model based on the MaxVIT_XL_512 model here (highest f1 of all the models in this repo for my use-case) and I currently have hundreds of users using the model actively which is the reason for the request as getting tensorrt support could 2x or even more the throughput of the model for all my users.

rwightman commented 2 days ago

@Skier23 there are no 'custom' operators, no custom kernels, trition, cuda, etc used in this model, not even any Autograd fns. So you might want to dig in to what pytorch ops are not supported because it's all pytorch ops. The logs provided do not point to anything specific. Will close unless there's nore actionanable info.