SHAMIK-97
commented
10 months ago Sorry, I wrongly mentioned that I didn't perform any modification in mmsegmentation code. But I did some modification. My requirement is the prediction tensor of the Segformer pretrained model. So I modified the function as below:
def inference_model(model: BaseSegmentor, img: Tensor) -> Union[SegDataSample, SampleList]: """Inference image(s) with the segmentor.
Args:
model (nn.Module): The loaded segmentor.
imgs (str/ndarray or list[str/ndarray]): Either image files or loaded
images.
Returns:
:obj:`SegDataSample` or list[:obj:`SegDataSample`]:
If imgs is a list or tuple, the same length list type results
will be returned, otherwise return the segmentation results directly.
"""
# prepare data
#data, is_batch = _preprare_data(img, model)
# forward the model
# with torch.no_grad():
# results = model.test_step(data)
results = model.forward(img)
return results #if is_batch else results[0]This function path is : /mmsegmentation/mmseg/apis/inference.py
Thanks for your error report and we appreciate it a lot.
Checklist
Describe the bug I am trying to do inference with Segformer B4 pretrained model available in mmsegmentation using Cityscape dataset in Colab. GPU ram is getting overflown.
Reproduction
from cityscapesscripts.download import downloader from typing import Any, Callable, Dict, List, Optional, Union, Tuple from torchvision.datasets import Cityscapes import albumentations as A from albumentations.pytorch import ToTensorV2 import numpy as np import os
COLAB_DIR = '/content/data/'
if not os.path.exists(COLAB_DIR): !mkdir /content/data
session = downloader.login() downloader.get_available_packages(session=session)
print('Downloading gtFine and leftImg8bit packages ...\n') package_list =['gtFine_trainvaltest.zip','leftImg8bit_trainvaltest.zip'] downloader.download_packages(session=session, package_names=package_list, destination_path=COLAB_DIR)
!unzip -q /content/data/leftImg8bit_trainvaltest.zip -d /content/data/ !unzip -q /content/data/gtFine_trainvaltest.zip -d /content/data/
from cityscapesscripts.helpers import labels def encode_segmap(mask):
mask_copy = mask.copy() for label in labels.labels: mask_copy[mask==label.id] = label.trainId mask_copy[mask_copy==255] = 19 return mask_copy
class MyClass(Cityscapes): def getitem(self, index: int) -> Tuple[Any, Any]: image = Image.open(self.images[index]).convert('RGB') image = np.array(image) targets: Any = [] for i, t in enumerate(self.target_type): if t == 'polygon': target = self._load_json(self.targets[index][i]) else: target = Image.open(self.targets[index][i]) target = np.array(target) target = encode_segmap(target) target = np.resize(target,(128,128)) targets.append(target) target = tuple(targets) if len(targets) > 1 else targets[0] if self.transforms is not None: pass return image, torch.from_numpy(target)
from PIL import Image data_train= MyClass('/content/data/', split='train', mode='fine', target_type='semantic')
data_val = MyClass('/content/data/', split='val', mode='fine', target_type='semantic')
!git clone https://github.com/open-mmlab/mmsegmentation.git %cd mmsegmentation !git checkout main !pip install -e .
import mmseg from mmseg.apis import inference_model, init_model, show_result_pyplot import mmcv import cv2 import glob import os import time from os.path import join, isdir from os import listdir, rmdir from shutil import move, rmtree, make_archive import pickle print(mmseg.version)
import torch import torchvision from torchvision import datasets, models, transforms from torch.utils.data import DataLoader from torchsummary import summary import torch.nn as nn import numpy as np import os from typing import Callable, Dict, List, Tuple, Union from torch import nn, optim, utils import torch.nn.functional as F from itertools import permutations
!mim download mmsegmentation --config segformer_mit-b4_8xb1-160k_cityscapes-1024x1024 --dest .
config_file_1 = '/content/mmsegmentation/segformer_mit-b4_8xb1-160k_cityscapes-1024x1024.py' checkpoint_file_1 = '/content/mmsegmentation/segformer_mit-b4_8x1_1024x1024_160k_cityscapes_20211207_080709-07f6c333.pth'
model_1 = init_model(config_file_1, checkpoint_file_1, device=device)
idx, (i,l) = next(enumerate(data_train))
res = inference_model(model_1, i)
Did you make any modifications on the code or config? Did you understand what you have modified? I didn't do any modification on the mmsegmentation code.
What dataset did you use? Cityscape
Environment Google Colab Pro.
python mmseg/utils/collect_env.pyto collect necessary environment information and paste it here. sys.platform: linux Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] CUDA available: True numpy_random_seed: 2147483648 GPU 0: Tesla V100-SXM2-16GB CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.2, V12.2.140 GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 PyTorch: 2.1.0+cu121 PyTorch compiling details: PyTorch built with:TorchVision: 0.16.0+cu121 OpenCV: 4.8.0 MMEngine: 0.10.2 MMSegmentation: 1.2.2+c685fe6
$PATH,$LD_LIBRARY_PATH,$PYTHONPATH, etc.)Error traceback
OutOfMemoryError Traceback (most recent call last)
25 frames /content/mmsegmentation/mmseg/apis/inference.py in inference_model(model, img) 114 # forward the model 115 #with torch.no_grad(): --> 116 results = model.test_step(data) 117 118 return results if is_batch else results[0]
/usr/local/lib/python3.10/dist-packages/mmengine/model/base_model/base_model.py in test_step(self, data) 143 """ 144 data = self.data_preprocessor(data, False) --> 145 return self._run_forward(data, mode='predict') # type: ignore 146 147 def parse_losses(
/usr/local/lib/python3.10/dist-packages/mmengine/model/base_model/base_model.py in _run_forward(self, data, mode) 344 """ 345 if isinstance(data, dict): --> 346 results = self(*data, mode=mode) 347 elif isinstance(data, (list, tuple)): 348 results = self(data, mode=mode)
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, kwargs) 1516 return self._compiled_call_impl(*args, *kwargs) # type: ignore[misc] 1517 else: -> 1518 return self._call_impl(args, kwargs) 1519 1520 def _call_impl(self, *args, **kwargs):
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, *kwargs) 1525 or _global_backward_pre_hooks or _global_backward_hooks 1526 or _global_forward_hooks or _global_forward_pre_hooks): -> 1527 return forward_call(args, **kwargs) 1528 1529 try:
/content/mmsegmentation/mmseg/models/segmentors/base.py in forward(self, inputs, data_samples, mode) 94 return self.loss(inputs, data_samples) 95 elif mode == 'predict': ---> 96 return self.predict(inputs, data_samples) 97 elif mode == 'tensor': 98 return self._forward(inputs, data_samples)
/content/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py in predict(self, inputs, data_samples) 218 ] * inputs.shape[0] 219 --> 220 seg_logits = self.inference(inputs, batch_img_metas) 221 222 return self.postprocess_result(seg_logits, data_samples)
/content/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py in inference(self, inputs, batch_img_metas) 339 level=logging.WARN) 340 if self.test_cfg.mode == 'slide': --> 341 seg_logit = self.slide_inference(inputs, batch_img_metas) 342 else: 343 seg_logit = self.whole_inference(inputs, batch_img_metas)
/content/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py in slide_inference(self, inputs, batch_img_metas) 281 # the output of encode_decode is seg logits tensor map 282 # with shape [N, C, H, W] --> 283 crop_seg_logit = self.encode_decode(crop_img, batch_img_metas) 284 preds += F.pad(crop_seg_logit, 285 (int(x1), int(preds.shape[3] - x2), int(y1),
/content/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py in encode_decode(self, inputs, batch_img_metas) 126 """Encode images with backbone and decode into a semantic segmentation 127 map of the same size as input.""" --> 128 x = self.extract_feat(inputs) 129 seg_logits = self.decode_head.predict(x, batch_img_metas, 130 self.test_cfg)
/content/mmsegmentation/mmseg/models/segmentors/encoder_decoder.py in extract_feat(self, inputs) 117 def extract_feat(self, inputs: Tensor) -> List[Tensor]: 118 """Extract features from images.""" --> 119 x = self.backbone(inputs) 120 if self.with_neck: 121 x = self.neck(x)
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, kwargs) 1516 return self._compiled_call_impl(*args, *kwargs) # type: ignore[misc] 1517 else: -> 1518 return self._call_impl(args, kwargs) 1519 1520 def _call_impl(self, *args, **kwargs):
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, *kwargs) 1525 or _global_backward_pre_hooks or _global_backward_hooks 1526 or _global_forward_hooks or _global_forward_pre_hooks): -> 1527 return forward_call(args, **kwargs) 1528 1529 try:
/content/mmsegmentation/mmseg/models/backbones/mit.py in forward(self, x) 442 x, hw_shape = layer0 443 for block in layer[1]: --> 444 x = block(x, hw_shape) 445 x = layer2 446 x = nlc_to_nchw(x, hw_shape)
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, kwargs) 1516 return self._compiled_call_impl(*args, *kwargs) # type: ignore[misc] 1517 else: -> 1518 return self._call_impl(args, kwargs) 1519 1520 def _call_impl(self, *args, **kwargs):
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, *kwargs) 1525 or _global_backward_pre_hooks or _global_backward_hooks 1526 or _global_forward_hooks or _global_forward_pre_hooks): -> 1527 return forward_call(args, **kwargs) 1528 1529 try:
/content/mmsegmentation/mmseg/models/backbones/mit.py in forward(self, x, hw_shape) 292 x = cp.checkpoint(_inner_forward, x) 293 else: --> 294 x = _inner_forward(x) 295 return x 296
/content/mmsegmentation/mmseg/models/backbones/mit.py in _inner_forward(x) 285 286 def _inner_forward(x): --> 287 x = self.attn(self.norm1(x), hw_shape, identity=x) 288 x = self.ffn(self.norm2(x), hw_shape, identity=x) 289 return x
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, kwargs) 1516 return self._compiled_call_impl(*args, *kwargs) # type: ignore[misc] 1517 else: -> 1518 return self._call_impl(args, kwargs) 1519 1520 def _call_impl(self, *args, **kwargs):
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, *kwargs) 1525 or _global_backward_pre_hooks or _global_backward_hooks 1526 or _global_forward_hooks or _global_forward_pre_hooks): -> 1527 return forward_call(args, **kwargs) 1528 1529 try:
/content/mmsegmentation/mmseg/models/backbones/mit.py in forward(self, x, hw_shape, identity) 179 x_kv = x_kv.transpose(0, 1) 180 --> 181 out = self.attn(query=x_q, key=x_kv, value=x_kv)[0] 182 183 if self.batch_first:
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, kwargs) 1516 return self._compiled_call_impl(*args, *kwargs) # type: ignore[misc] 1517 else: -> 1518 return self._call_impl(args, kwargs) 1519 1520 def _call_impl(self, *args, **kwargs):
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, *kwargs) 1525 or _global_backward_pre_hooks or _global_backward_hooks 1526 or _global_forward_hooks or _global_forward_pre_hooks): -> 1527 return forward_call(args, **kwargs) 1528 1529 try:
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py in forward(self, query, key, value, key_padding_mask, need_weights, attn_mask, average_attn_weights, is_causal) 1239 is_causal=is_causal) 1240 else: -> 1241 attn_output, attn_output_weights = F.multi_head_attention_forward( 1242 query, key, value, self.embed_dim, self.num_heads, 1243 self.in_proj_weight, self.in_proj_bias,
/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py in multi_head_attention_forward(query, key, value, embed_dim_to_check, num_heads, in_proj_weight, in_proj_bias, bias_k, bias_v, add_zero_attn, dropout_p, out_proj_weight, out_proj_bias, training, key_padding_mask, need_weights, attn_mask, use_separate_proj_weight, q_proj_weight, k_proj_weight, v_proj_weight, static_k, static_v, average_attn_weights, is_causal) 5404 else: 5405 attn_output_weights = torch.bmm(q_scaled, k.transpose(-2, -1)) -> 5406 attn_output_weights = softmax(attn_output_weights, dim=-1) 5407 if dropout_p > 0.0: 5408 attn_output_weights = dropout(attn_output_weights, p=dropout_p)
/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py in softmax(input, dim, _stacklevel, dtype) 1854 dim = _get_softmax_dim("softmax", input.dim(), _stacklevel) 1855 if dtype is None: -> 1856 ret = input.softmax(dim) 1857 else: 1858 ret = input.softmax(dim, dtype=dtype)
OutOfMemoryError: CUDA out of memory. Tried to allocate 80.00 MiB. GPU 0 has a total capacty of 15.77 GiB of which 32.38 MiB is free. Process 16217 has 15.74 GiB memory in use. Of the allocated memory 15.24 GiB is allocated by PyTorch, and 127.57 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!