Issue with using images of different sizes for inference in pose estimation

[Ody-trek]

Hello,

I encountered a problem while running the lite/scripts/demo/torchscript/pose_keypoints17.sh script. When I use a folder containing images of different sizes as input, I get the following runtime error during inference: RuntimeError: Trying to resize storage that is not resizable However, when I use a folder containing only one image or images that are all the same size, the inference runs successfully.

My questions: How should I handle input folders containing images of different sizes for inference?

1. Is there any pre-processing step (like resizing or padding) that I need to apply to the images before using them as input?
2. Could the issue be related to the DataLoader or the pipeline configuration in test_pipeline? If so, how can I modify it to avoid such errors?

Error details: The complete error message is as follows:

Distributing 29 image paths into 1 jobs.
/home/chou/deep/sapiens/engine/mmengine/optim/optimizer/zero_optimizer.py:16: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
  from torch.distributed.optim import \
Loads checkpoint by local backend from path: /home/chou/deep/sapiens/lite/torchscript/detector/checkpoints/rtmpose/rtmdet_m_8xb32-100e_coco-obj365-person-235e8209.pth on server: mymelo
/home/chou/deep/sapiens/engine/mmengine/runner/checkpoint.py:354: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(filename, map_location=map_location)
Done: Loaded checkpoint from /home/chou/deep/sapiens/lite/torchscript/detector/checkpoints/rtmpose/rtmdet_m_8xb32-100e_coco-obj365-person-235e8209.pth on server: mymelo
  0%|                                                                                                         | 0/4 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/chou/deep/sapiens/lite/demo/vis_pose.py", line 485, in <module>
    main()
  File "/home/chou/deep/sapiens/lite/demo/vis_pose.py", line 378, in main
    for batch_idx, (batch_image_name, batch_orig_imgs, batch_imgs) in tqdm(
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 701, in __next__
    data = self._next_data()
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1465, in _next_data
    return self._process_data(data)
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1491, in _process_data
    data.reraise()
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/_utils.py", line 715, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 351, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 55, in fetch
    return self.collate_fn(data)
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 398, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 211, in collate
    return [
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 212, in <listcomp>
    collate(samples, collate_fn_map=collate_fn_map)
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 155, in collate
    return collate_fn_map[elem_type](batch, collate_fn_map=collate_fn_map)
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 285, in collate_numpy_array_fn
    return collate([torch.as_tensor(b) for b in batch], collate_fn_map=collate_fn_map)
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 155, in collate
    return collate_fn_map[elem_type](batch, collate_fn_map=collate_fn_map)
  File "/home/chou/miniconda3/envs/sapiens/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py", line 271, in collate_tensor_fn
    out = elem.new(storage).resize_(len(batch), *list(elem.size()))
RuntimeError: Trying to resize storage that is not resizable

/home/chou/deep/sapiens/lite/scripts/demo/torchscript
Processing complete.
Results saved to /home/chou/Desktop/sapiens/pose/Outputs/vis/test_30_val_images/sapiens_0.3b

Thank you in advance for your help!

[FrankWuuu]

In the original AdhocImageDataset, there is a _preprocess function that manages image resizing. You might consider setting the self.shape parameter to define the target dimensions for the images. By doing so, the images in the dataset will be automatically resized to your specified dimensions within the _preprocess function, ensuring they meet your size requirements.

For example:

    inference_dataset = AdhocImageDataset(
        [os.path.join(input_dir, img_name) for img_name in image_names],shape=args.shape
    )

If you’re working with a personal dataset format, you could implement a similar function to handle resizing. This involves adapting the _preprocess logic to fit the structure and requirements of your dataset, ensuring consistent image dimensions throughout your workflow.

[rawalkhirodkar]

@FrankWuuu thank you for answering the question.

@Ody-trek yes, this is by design. We prioritized batch processing over variable size image processing. Setting batch size to 1 in your case will resolve the issue.