a received signal SIGABRT via a large malloc in pnm_load_image

[YourButterfly]

Description of problem:

a received signal SIGABRT via a large malloc in function pnm_load_image，line 285 ，file input-pnm.ci

Version-Release number of selected component (if applicable):

<= latest version

The output information is as follows:

$ ./sam2p /home/pwd/Desktop/crashes-2018-06-19-19-02/sam2p000_id_000000,sig_06,src_000000,op_havoc,rep_128 try.bmp
This is sam2p 0.49.4.
Available Loaders: PS PDF JAI PNG JPEG TIFF PNM BMP LBM XPM PCX TGA.
Available Appliers: XWD Meta Empty BMP PNG TIFF6 TIFF6-JAI JPEG-JAI JPEG PNM XPM PSL1C PSL23+PDF PSL2+PDF-JAI P-TrOpBb.
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

The gdb debugging information is listed below:(with asan):

This is sam2p 0.49.4.
Available Loaders: PS PDF JAI PNG JPEG TIFF PNM BMP LBM XPM PCX TGA.
Available Appliers: XWD Meta Empty BMP PNG TIFF6 TIFF6-JAI JPEG-JAI JPEG PNM XPM PSL1C PSL23+PDF PSL2+PDF-JAI P-TrOpBb.
==56626==ERROR: AddressSanitizer failed to allocate 0x55d20d8000 (368596320256) bytes of LargeMmapAllocator (error code: 12)
==56626==Process memory map follows:

...

==56626==End of process memory map.
==56626==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_common.cc:118 "((0 && "unable to mmap")) != (0)" (0x0, 0x0)
    #0 0x7fb579165c02  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe9c02)
    #1 0x7fb579184595 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x108595)
    #2 0x7fb57916f492  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xf3492)
    #3 0x7fb57917b8a5  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xff8a5)
    #4 0x7fb5790a5a51  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x29a51)
    #5 0x7fb57915c5de in operator new[](unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe05de)
    #6 0x5622718ad25c in pnm_load_image(GenBuffer::Readable*) /home/pwd/fuzz/fuzz-sam2p/sam2p/input-pnm.ci:285
    #7 0x5622718ada99 in in_pnm_reader /home/pwd/fuzz/fuzz-sam2p/sam2p/in_pnm.cpp:29
    #8 0x56227192ddc9 in Image::load(Image::Loader::UFD*, SimBuffer::Flat const&, char const*) /home/pwd/fuzz/fuzz-sam2p/sam2p/image.cpp:1428
    #9 0x562271881d16 in run_sam2p_engine(Files::FILEW&, Files::FILEW&, char const* const*, bool) /home/pwd/fuzz/fuzz-sam2p/sam2p/sam2p_main.cpp:1055
    #10 0x56227187dd50 in main /home/pwd/fuzz/fuzz-sam2p/sam2p/sam2p_main.cpp:1148
    #11 0x7fb57891eb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #12 0x56227187e9d9 in _start (/home/pwd/fuzz/fuzz-sam2p/sam2p/sam2p+0x269d9)

More Information position： function pnm_load_image，line 285 ，file input-pnm.ci

  BITMAP_PLANES (bitmap) = (pnminfo->np)?(pnminfo->np):1;
  /* BITMAP_BITS (bitmap) = (unsigned char *) malloc ((slen_t)pnminfo->yres * pnminfo->xres * BITMAP_PLANES (bitmap)); */
  XMALLOCT(BITMAP_BITS (bitmap), unsigned char *, multiply_check(pnminfo->yres, pnminfo->xres, BITMAP_PLANES (bitmap)));

some debug info:

pwndbg> p pnminfo->yres 
$1 = 3071635911
pwndbg> p pnminfo->xres 
$2 = 120
pwndbg> p bitmap.np
There is no member named np.
pwndbg> p pnminfo->np
$3 = 0

and in multiply_check

static slen_t multiply_check(slen_t a, slen_t b) {
  slen_t result;
  if (a == 0) return 0;
  /* Check for overflow. Works only if everything is unsigned. */
  if ((result = a * b) / a != b) FATALP("PNM: can't open file\n");
  return result;
}

In here, slen_t` size is 8 bytes,

pwndbg> p sizeof(slen_t)
$6 = 8

so , in Macro Define XMALLOCT ,the size is 0x55d20d5548

#define XMALLOCT(var,typep,size) var=(typep)new char[size]

found by pwd@360TeamSerions

[YourButterfly]

POC

[pts]

Thank you for reporting this!

In this case sam2p runs out of memory, because the input image is too large. sam2p reports this by showing terminate called after throwing an instance of 'std::bad_alloc', and fails with SIGABRT. I can't see a vulnerability or bug here.

What should sam2p do instead? It could fail with a friendlier error message, e.g. Input image is too large, doesn't fit in memory, aborting. This would be too much work to implement, because we'd have to override all new operators in the code. What other improvement do you recommend?

Closing this issue for now. I'll reopen it as soon as I learn about an improvement which is feasible to implement.

[YourButterfly]

All right, it just a large flag x , flag y in file pbm, and over the actual size of the input.

AFAICS, the ImageMagic, They use disk space instead of memory sometimes. I `m not sure which one is better.

[pts]

It would be possible to reduce the memory usage of sam2p to just 21 * image_width + constant bytes by making it use temporary files instead of loading the entire image to memory. (Thus the image_height wouldn't matter.) However, this would need a full rewrite of the image transformation and compression code in sam2p, which is a huge piece of work, and it becomes feasible only when someone volunteers for that.