facebookresearch / detectron2

Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.
https://detectron2.readthedocs.io/en/latest/
Apache License 2.0
30.13k stars 7.42k forks source link

Support torchscript export for FCOS #4267

Open inakivelasco opened 2 years ago

inakivelasco commented 2 years ago

I am trying to export a model trained with a custom dataset to torchscript.

Any help would be appreciated

Instructions To Reproduce the Issue and Full Logs

I trained a model and now I am trying to export it to torchscript but I am having some problems with it

I run the following command: export_lazy_config.py --config-file "path to .yaml with the config" --format torchscript --export-method scripting --output /export_output

I got the following error: Module 'ResNet' has no attribute '_out_features' (This attribute exists on the Python module, but we failed to convert Python type: 'omegaconf.listconfig.ListConfig' to a TorchScript type. Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type ListConfig.. Its type was inferred; try adding a type annotation for the attribute.)

Here is the log:

Command line arguments: Namespace(format='torchscript', export_method='scripting', config_file='../../../../media/naeem/T7/trainers/fcos_R_50_FPN_1x.py/output/config.yaml', sample_image=None, run_eval=False, output='/export_output', opts=[]) [05/24 17:31:06 detectron2]: Command line arguments: Namespace(format='torchscript', export_method='scripting', config_file='../../../../media/naeem/T7/trainers/fcos_R_50_FPN_1x.py/output/config.yaml', sample_image=None, run_eval=False, output='/export_output', opts=[]) [05/24 17:31:06 detectron2]: Contents of args.config_file=../../../../media/naeem/T7/trainers/fcos_R_50_FPN_1x.py/output/config.yaml: dataloader: evaluator: {target: detectron2.evaluation.COCOEvaluator, dataset_name: '${..test.dataset.names}'} test: target: detectron2.data.build_detection_test_loader dataset: {target: detectron2.data.get_detection_dataset_dicts, filter_empty: false, names: maize_valid} mapper: target: detectron2.data.DatasetMapper augmentations: - {target: detectron2.data.transforms.ResizeShortestEdge, max_size: 1333, short_edge_length: 800} image_format: ${...train.mapper.image_format} is_train: false num_workers: 0 train: target: detectron2.data.build_detection_train_loader dataset: {target: detectron2.data.get_detection_dataset_dicts, names: maize_train} mapper: target: detectron2.data.DatasetMapper augmentations: - target: detectron2.data.transforms.ResizeShortestEdge max_size: 1333 sample_style: choice short_edge_length: [640, 672, 704, 736, 768, 800] - {target: detectron2.data.transforms.RandomFlip, horizontal: true} image_format: BGR is_train: true use_instance_mask: false num_workers: 4 total_batch_size: 2 lr_multiplier: target: detectron2.solver.WarmupParamScheduler scheduler: target: fvcore.common.param_scheduler.MultiStepParamScheduler milestones: [60000, 80000, 90000] values: [1.0, 0.1, 0.01] warmup_factor: 0.001 warmup_length: 0.011111111111111112 warmup_method: linear model: target: detectron2.modeling.FCOS backbone: target: detectron2.modeling.FPN bottom_up: target: detectron2.modeling.ResNet freeze_at: 2 out_features: [res3, res4, res5] stages: {target: detectron2.modeling.ResNet.make_default_stages, depth: 50, norm: FrozenBN, stride_in_1x1: true} stem: {target: detectron2.modeling.backbone.BasicStem, in_channels: 3, norm: FrozenBN, out_channels: 64} in_features: [res3, res4, res5] out_channels: 256 top_block: {target: detectron2.modeling.backbone.fpn.LastLevelP6P7, in_channels: 256, in_feature: p5, out_channels: '${..out_channels}'} focal_loss_alpha: 0.25 focal_loss_gamma: 2.0 head: target: detectron2.modeling.meta_arch.fcos.FCOSHead conv_dims: [256, 256, 256, 256] input_shape: - &id001 !!python/object/new:detectron2.layers.shape_spec.ShapeSpec [256, null, null, null] - id001 - id001 - id001 - id001 norm: GN num_classes: ${..num_classes} prior_prob: 0.01 head_in_features: [p3, p4, p5, p6, p7] num_classes: 10 pixel_mean: [103.53, 116.28, 123.675] pixel_std: [1.0, 1.0, 1.0] test_nms_thresh: 0.6 test_score_thresh: 0.2 optimizer: target: torch.optim.SGD lr: 0.01 momentum: 0.9 params: {target: detectron2.solver.get_default_optimizer_params, weight_decay_norm: 0.0} weight_decay: 0.0001 train: amp: {enabled: false} checkpointer: {max_to_keep: 100, period: 5000} ddp: {broadcast_buffers: false, find_unused_parameters: false, fp16_compression: false} device: cuda eval_period: 5000 init_checkpoint: '' log_period: 20 max_iter: 1500 output_dir: /media/naeem/T7/trainers/fcos_R_50_FPN_1x.py/output/ [05/24 17:31:06 detectron2]: Contents of args.config_file=../../../../media/naeem/T7/trainers/fcos_R_50_FPN_1x.py/output/config.yaml: dataloader: evaluator: {target: detectron2.evaluation.COCOEvaluator, dataset_name: '${..test.dataset.names}'} test: target: detectron2.data.build_detection_test_loader dataset: {target: detectron2.data.get_detection_dataset_dicts, filter_empty: false, names: maize_valid} mapper: target: detectron2.data.DatasetMapper augmentations: - {target: detectron2.data.transforms.ResizeShortestEdge, max_size: 1333, short_edge_length: 800} image_format: ${...train.mapper.image_format} is_train: false num_workers: 0 train: target: detectron2.data.build_detection_train_loader dataset: {target: detectron2.data.get_detection_dataset_dicts, names: maize_train} mapper: target: detectron2.data.DatasetMapper augmentations: - target: detectron2.data.transforms.ResizeShortestEdge max_size: 1333 sample_style: choice short_edge_length: [640, 672, 704, 736, 768, 800] - {target: detectron2.data.transforms.RandomFlip, horizontal: true} image_format: BGR is_train: true use_instance_mask: false num_workers: 4 total_batch_size: 2 lr_multiplier: target: detectron2.solver.WarmupParamScheduler scheduler: target: fvcore.common.param_scheduler.MultiStepParamScheduler milestones: [60000, 80000, 90000] values: [1.0, 0.1, 0.01] warmup_factor: 0.001 warmup_length: 0.011111111111111112 warmup_method: linear model: target: detectron2.modeling.FCOS backbone: target: detectron2.modeling.FPN bottom_up: target: detectron2.modeling.ResNet freeze_at: 2 out_features: [res3, res4, res5] stages: {target: detectron2.modeling.ResNet.make_default_stages, depth: 50, norm: FrozenBN, stride_in_1x1: true} stem: {target: detectron2.modeling.backbone.BasicStem, in_channels: 3, norm: FrozenBN, out_channels: 64} in_features: [res3, res4, res5] out_channels: 256 top_block: {target: detectron2.modeling.backbone.fpn.LastLevelP6P7, in_channels: 256, in_feature: p5, out_channels: '${..out_channels}'} focal_loss_alpha: 0.25 focal_loss_gamma: 2.0 head: target: detectron2.modeling.meta_arch.fcos.FCOSHead conv_dims: [256, 256, 256, 256] input_shape: - &id001 !!python/object/new:detectron2.layers.shape_spec.ShapeSpec [256, null, null, null] - id001 - id001 - id001 - id001 norm: GN num_classes: ${..num_classes} prior_prob: 0.01 head_in_features: [p3, p4, p5, p6, p7] num_classes: 10 pixel_mean: [103.53, 116.28, 123.675] pixel_std: [1.0, 1.0, 1.0] test_nms_thresh: 0.6 test_score_thresh: 0.2 optimizer: target: torch.optim.SGD lr: 0.01 momentum: 0.9 params: {target: detectron2.solver.get_default_optimizer_params, weight_decay_norm: 0.0} weight_decay: 0.0001 train: amp: {enabled: false} checkpointer: {max_to_keep: 100, period: 5000} ddp: {broadcast_buffers: false, find_unused_parameters: false, fp16_compression: false} device: cuda eval_period: 5000 init_checkpoint: '' log_period: 20 max_iter: 1500 output_dir: /media/naeem/T7/trainers/fcos_R_50_FPN_1x.py/output/ [05/24 17:31:06 detectron2]: Full config saved to /media/naeem/T7/trainers/fcos_R_50_FPN_1x.py/output/config.yaml [05/24 17:31:06 detectron2]: Full config saved to /media/naeem/T7/trainers/fcos_R_50_FPN_1x.py/output/config.yaml [05/24 17:31:06 d2.utils.env]: Using a generated random seed 6677598 [05/24 17:31:06 d2.utils.env]: Using a generated random seed 6677598 [05/24 17:31:06 fvcore.common.checkpoint]: No checkpoint found. Initializing model from scratch WARNING [05/24 17:31:07 d2.data.datasets.coco]: Category ids in annotations are not in [1, #categories]! We'll apply a mapping for you. WARNING [05/24 17:31:07 d2.data.datasets.coco]: Category ids in annotations are not in [1, #categories]! We'll apply a mapping for you. [05/24 17:31:07 d2.data.datasets.coco]: Loaded 5000 images in COCO format from /../../../../Scientific Project/dataset/annotations/instancesAnimal_val2017.json [05/24 17:31:07 d2.data.datasets.coco]: Loaded 5000 images in COCO format from /../../../../Scientific Project/dataset/annotations/instancesAnimal_val2017.json [05/24 17:31:07 d2.data.build]: Distribution of instances among all 10 categories: | category | #instances | category | #instances | category | #instances | |:----------:|:-------------|:----------:|:-------------|:----------:|:-------------| | bird | 427 | cat | 202 | dog | 218 | | horse | 272 | sheep | 354 | cow | 372 | | elephant | 252 | bear | 71 | zebra | 266 | | giraffe | 232 | | | | | | total | 2666 | | | | | [05/24 17:31:07 d2.data.build]: Distribution of instances among all 10 categories: | category | #instances | category | #instances | category | #instances | |:----------:|:-------------|:----------:|:-------------|:----------:|:-------------| | bird | 427 | cat | 202 | dog | 218 | | horse | 272 | sheep | 354 | cow | 372 | | elephant | 252 | bear | 71 | zebra | 266 | | giraffe | 232 | | | | | | total | 2666 | | | | | [05/24 17:31:07 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [ResizeShortestEdge(short_edge_length=(800, 800), max_size=1333)] [05/24 17:31:07 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [ResizeShortestEdge(short_edge_length=(800, 800), max_size=1333)] [05/24 17:31:07 d2.data.common]: Serializing 5000 elements to byte tensors and concatenating them all ... [05/24 17:31:07 d2.data.common]: Serializing 5000 elements to byte tensors and concatenating them all ... [05/24 17:31:07 d2.data.common]: Serialized dataset takes 2.98 MiB [05/24 17:31:07 d2.data.common]: Serialized dataset takes 2.98 MiB Traceback (most recent call last): File "D:\detectron2-master\detectron2-master\tools\deploy\export_lazy_config.py", line 257, in exported_model = export_scripting(torch_model) File "D:\detectron2-master\detectron2-master\tools\deploy\export_lazy_config.py", line 103, in export_scripting ts_model = scripting_with_instances(ScriptableAdapter(), fields) File "c:\users\inaki\detectron2\detectron2\export\torchscript.py", line 55, in scripting_with_instances scripted_model = torch.jit.script(model) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_script.py", line 1265, in script return torch.jit._recursive.create_script_module( File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 454, in create_script_module return create_script_module_impl(nn_module, concrete_type, stubs_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 516, in create_script_module_impl script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_script.py", line 594, in _construct init_fn(script_module) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 494, in init_fn scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 516, in create_script_module_impl script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_script.py", line 594, in _construct init_fn(script_module) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 494, in init_fn scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 516, in create_script_module_impl script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_script.py", line 594, in _construct init_fn(script_module) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 494, in init_fn scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 520, in create_script_module_impl create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 371, in create_methods_and_properties_from_stubs concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults) RuntimeError: Module 'ResNet' has no attribute '_out_features' (This attribute exists on the Python module, but we failed to convert Python type: 'omegaconf.listconfig.ListConfig' to a TorchScript type. Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type ListConfig.. Its type was inferred; try adding a type annotation for the attribute.): File "c:\users\inaki\detectron2\detectron2\modeling\backbone\resnet.py", line 446 outputs = {} x = self.stem(x) if "stem" in self._out_features: ~~~~~~ <--- HERE outputs["stem"] = x for name, stage in zip(self.stage_names, self.stages):

Your Environment

sys.platform win32 Python 3.10.3 | packaged by conda-forge | (main, Mar 28 2022, 05:19:17) [MSC v.1916 64 bit (AMD64)] numpy 1.22.3 detectron2 0.6 @c:\users\inaki\detectron2\detectron2 Compiler MSVC 192930141 CUDA compiler not available DETECTRON2_ENV_MODULE PyTorch 1.11.0 @C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch PyTorch debug build False GPU available Yes GPU 0 NVIDIA GeForce RTX 3050 Laptop GPU (arch=8.6) Driver version 512.59 CUDA_HOME C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6 Pillow 9.0.1 torchvision 0.12.0 @C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torchvision torchvision arch flags C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torchvision_C.pyd; cannot find cuobjdump fvcore 0.1.5.post20220305 iopath 0.1.9 cv2 4.5.5

github-actions[bot] commented 2 years ago

You've chosen to report an unexpected problem or bug. Unless you already know the root cause of it, please include details about it by filling the issue template. The following information is missing: "Instructions To Reproduce the Issue and Full Logs"; "Your Environment";

ppwwyyxx commented 2 years ago

Don't have a way to reproduce this. To guess from the error message I think this line: https://github.com/facebookresearch/detectron2/blob/45b3fcea6e76bf7a351e54e01c7d6e1a3a0100a5/detectron2/modeling/backbone/resnet.py#L428 should be changed to

self._out_features = list(out_features)
inakivelasco commented 2 years ago

Don't have a way to reproduce this. To guess from the error message I think this line:

https://github.com/facebookresearch/detectron2/blob/45b3fcea6e76bf7a351e54e01c7d6e1a3a0100a5/detectron2/modeling/backbone/resnet.py#L428

should be changed to

self._out_features = list(out_features)

After changing that and executing again the same command I get the following error:

Traceback (most recent call last): File "D:\detectron2-master\detectron2-master\tools\deploy\export_lazy_config.py", line 257, in exported_model = export_scripting(torch_model) File "D:\detectron2-master\detectron2-master\tools\deploy\export_lazy_config.py", line 103, in export_scripting ts_model = scripting_with_instances(ScriptableAdapter(), fields) File "c:\users\inaki\detectron2\detectron2\export\torchscript.py", line 55, in scripting_with_instances scripted_model = torch.jit.script(model) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_script.py", line 1265, in script return torch.jit._recursive.create_script_module( File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 454, in create_script_module return create_script_module_impl(nn_module, concrete_type, stubs_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 516, in create_script_module_impl script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_script.py", line 594, in _construct init_fn(script_module) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 494, in init_fn scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 520, in create_script_module_impl create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 371, in create_methods_and_properties_from_stubs concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults) RuntimeError: Module 'FCOS' has no attribute 'head_in_features' (This attribute exists on the Python module, but we failed to convert Python type: 'omegaconf.listconfig.ListConfig' to a TorchScript type. Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type ListConfig.. Its type was inferred; try adding a type annotation for the attribute.):

     File "c:\users\inaki\detectron2\detectron2\modeling\meta_arch\dense_detector.py", line 94
             images = self.preprocess_image(batched_inputs) 
             features = self.backbone(images.tensor) 
             features = [features[f] for f in self.head_in_features]                                                                                                                                                                                       
                                           ~~~~~~~~~~~~~~~~~~~~~ <--- HERE                                                                                                                                             
            predictions = self.head(features)
ppwwyyxx commented 2 years ago

self.head_in_features should be changed to a list as well. Also, FCOS is not in the export support list in https://detectron2.readthedocs.io/en/latest/tutorials/deployment.html , so there are probably other changes needed

inakivelasco commented 2 years ago

self.head_in_features should be changed to a list as well. Also, FCOS is not in the export support list in https://detectron2.readthedocs.io/en/latest/tutorials/deployment.html , so there are probably other changes needed

I changed this in the dense_detector.py (i added the final line):

        self.backbone = backbone
        self.head = head
        if head_in_features is None:
            shapes = self.backbone.output_shape()
            self.head_in_features = sorted(shapes.keys(), key=lambda x: shapes[x].stride)
        else:
            self.head_in_features = head_in_features
        self.head_in_features = list(head_in_features)

Now I get the following error:

Traceback (most recent call last): File "D:\detectron2-master\detectron2-master\tools\deploy\export_lazy_config.py", line 257, in exported_model = export_scripting(torch_model) File "D:\detectron2-master\detectron2-master\tools\deploy\export_lazy_config.py", line 103, in export_scripting ts_model = scripting_with_instances(ScriptableAdapter(), fields) File "c:\users\inaki\detectron2\detectron2\export\torchscript.py", line 55, in scripting_with_instances scripted_model = torch.jit.script(model) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_script.py", line 1265, in script return torch.jit._recursive.create_script_module( File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 454, in create_script_module return create_script_module_impl(nn_module, concrete_type, stubs_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 516, in create_script_module_impl script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_script.py", line 594, in _construct init_fn(script_module) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 494, in init_fn scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 520, in create_script_module_impl create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs) File "C:\Users\Inaki.conda\envs\ScientificProject\lib\site-packages\torch\jit_recursive.py", line 371, in create_methods_and_properties_from_stubs concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults) RuntimeError: forward(torch.detectron2.modeling.meta_arch.fcos.FCOSHead self, Tensor features) -> ((Tensor[], Tensor[], Tensor[])): Expected a value of type 'Tensor (inferred)' for argument 'features' but instead found type 'List[Tensor]'. Inferred 'features' to be of type 'Tensor' because it was not annotated with an explicit type. Empty lists default to List[Tensor]. Add a variable annotation to the assignment to create an empty list of another type (torch.jit.annotate(List[T, []]) where T is the type of elements in the list for Python 2) :

                                                                                                                                                                                                         `File "c:\users\inaki\detectron2\detectron2\modeling\meta_arch\dense_detector.py", line 96
 features = self.backbone(images.tensor)
 features = [features[f] for f in self.head_in_features]
 predictions = self.head(features)
                   ~~~~~~~~~ <--- HERE
 if self.training:`

Thanks for your response

ppwwyyxx commented 2 years ago

FCOSHead.forward needs to have type annotation just like RetinaNetHead.forward.

dcy0577 commented 2 years ago

Hi, any new updates on this? I got exactly the same error when trying to deploy the new mask-rcnn baseline with lazy config.