OOM during prediction

[patcharees]

Hi,

I used HrSegNetB48 model and got an out-of-memory issue when I ran prediction on 16GB GPU. Do you have any suggestions in case I want to use this HrSegNetB48, but no bigger RAM?

Thanks!

   pred, _ = infer.inference(
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/paddleseg/core/infer.py", line 160, in inference
    logits = model(im)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/paddle/nn/layer/layers.py", line 1254, in __call__
    return self.forward(*inputs, **kwargs)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/paddleseg/models/hrsegnet_b48.py", line 91, in forward
    hrseg3_out = self.seg3(hrseg2_out)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/paddle/nn/layer/layers.py", line 1254, in __call__
    return self.forward(*inputs, **kwargs)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/paddleseg/models/hrsegnet_b48.py", line 197, in forward
    out_l3_i = F.interpolate(out_l3, size=size, mode='bilinear', align_corners=True)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/paddle/nn/functional/common.py", line 622, in interpolate
    out = _C_ops.bilinear_interp(
MemoryError: 

---

C++ Traceback (most recent call last):

0 bilinear_interp_ad_func(paddle::Tensor const&, paddle::optional const&, paddle::optional<std::vector<paddle::Tensor, std::allocator > > const&, paddle::optional const&, std::string, int, int, int, std::vector<float, std::allocator >, std::string, bool, int) 1 paddle::experimental::bilinear_interp(paddle::Tensor const&, paddle::optional const&, paddle::optional<std::vector<paddle::Tensor, std::allocator > > const&, paddle::optional const&, std::string const&, int, int, int, std::vector<float, std::allocator > const&, std::string const&, bool, int) 2 void phi::BilinearInterpKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, paddle::optional const&, paddle::optional<std::vector<phi::DenseTensor const, std::allocator<phi::DenseTensor const> > > const&, paddle::optional const&, std::string const&, int, int, int, std::vector<float, std::allocator > const&, std::string const&, bool, int, phi::DenseTensor) 3 float phi::DeviceContext::Alloc(phi::TensorBase, unsigned long, bool) const 4 phi::DeviceContext::Impl::Alloc(phi::TensorBase, phi::Place const&, phi::DataType, unsigned long, bool, bool) const 5 phi::DenseTensor::AllocateFrom(phi::Allocator, phi::DataType, unsigned long, bool) 6 paddle::memory::allocation::Allocator::Allocate(unsigned long) 7 paddle::memory::allocation::StatAllocator::AllocateImpl(unsigned long) 8 paddle::memory::allocation::Allocator::Allocate(unsigned long) 9 paddle::memory::allocation::Allocator::Allocate(unsigned long) 10 paddle::memory::allocation::Allocator::Allocate(unsigned long) 11 paddle::memory::allocation::CUDAAllocator::AllocateImpl(unsigned long) 12 std::string phi::enforce::GetCompleteTraceBackString(std::string&&, char const, int) 13 phi::enforce::GetCurrentTraceBackStringabi:cxx11

---

Error Message Summary:

ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 3.999023GB memory on GPU 0, 15.524414GB memory has been allocated and available memory is only 263.500000MB.

Please check whether there is any other process using GPU 0.

1. If yes, please stop them, or start PaddlePaddle on another GPU.
2. If no, please decrease the batch size of your model.

[patcharees]

I found that is_slide=True can fix the problem