A memory allocation error is not handled

[nhiroki09]

Decoding large image consumes large size of memory, so insufficient memory tends to occur on 32bit process. In an out of memory situation, de265_image::alloc_image() may fail to allocate memory and returns DE265_ERROR_OUT_OF_MEMORY. But decoded_picture_buffer::new_image() ignores DE265_ERROR_OUT_OF_MEMORY from de265_image::alloc_image(), so a NULL pointer access occurs in de265_image::set_SliceAddrRS().

I think that decoded_picture::new_image() should return -1 when de265_image::alloc_image() returned DE265_ERROR_OUT_OF_MEMORY.

• Test Version: 1.0.8

• Test Environment:

  • Compiler: Visual Studio 2017
    • Platform: Win32
  • OS: Windows 10 Enterprise 64bit
  • CPU: Intel(R) Xeon(R) E-2124G CPU @ 3.40GHz 3.41 GHz
  • RAM: 32GB
  • Using libde265 from libheif.

• Image:

  • Format: HEIF
  • Width : 32768
  • Height: 32768
  • Channels: YCbCr
  • Bits/Sample: 8

[farindk]

See the above change. Can you test this or attach a test image?