complement() and crop() crash on some inputs

[qrmt]

Let me begin by complementing you on this library, it feels almost magical how fast it's able to process complex masks with so many different functions.

To the bug report: I've observed crashes with complement() and crop() with certain inputs. I haven't been able to debug it further due to my limited C skills. The error message shown is either realloc(): invalid next size or corrupted size vs. prev_size.

This does not occur on ARM-based macs but does occur reliably on x86_64 Ubuntu machines. The library is installed from commit 2e7964e, python version is 3.10.13.

Reproduction

# complement()
>>> import rlemasklib
>>> data = {'size': [150, 306], 'counts': b'0hgS18f\\lNa0_Oa0_O`0@a0_Oa0_O`0@a0_Oa0_OPd7'}
>>> rlemasklib.complement(data)
realloc(): invalid next size
Fatal Python error: Aborted

# crop
>>> import rlemasklib
>>> import numpy as np
>>> data = {'size': [4048, 30], 'counts': b'4`0nm3b0UUL^OkNN`e3f6M>CS1mN;RbLeHQT3m7^jLiIkT3ba0ckL_[Ock2Te0QSMj[Ojl2We0PnLbYOb0\\1`0d0dP3ej0J0000000000000000X6gQMUmN]d2_[1PKf0ZOj1VNQ3oLe0[O4a<[CYi0gVORg6'}
>>> crop_box = np.array([0,0,28,1598], dtype=np.uint32)
>>> rlemasklib.crop(data, crop_box)
realloc(): invalid next size
Fatal Python error: Aborted

Samples

For complement()

{'size': [150, 306], 'counts': b'0hgS18f\\lNa0_Oa0_O`0@a0_Oa0_O`0@a0_Oa0_OPd7'}
{'size': [84, 163], 'counts': b'0Pd28\\^Ma0_Oa0_Oa0_O`0@]X:'}
{'size': [192, 448], 'counts': b';e5P1[Oe0[Of0ZOe0[Oe0[Of0ZOe0[Oe0[O[Tb2'}
{'size': [780, 252], 'counts': b'0ank5S6bWTJV<bN^100kNU1eM[2'}
{'size': [149, 376], 'counts': b'0fdk08g_TOa0_O`0@a0_O`0@a0_Oa0_O`0@a0_Ohci0'}

For crop()

data = {'size': [15044, 7957], 'counts': b'l[`Sa3Ze0QR>jNV1jNV1iNW1iNV1jNW1iNW1iNW1iNW1iNW1iNW1iNW1iNW1iNW1iNW1iNW1iNW1iNjQZi0'}
crop_box = np.array([7883,12304,18,684], dtype=np.uint32)

data = {'size': [1999, 3426], 'counts': b'Rln_6i0fm1`1iTNGXe1P5H9k]NRJVLgN_]1c7TeNQK`LjM[^1d7YdNh1UMkFiW1R8ogNeG6]?eT1fIUkNk6n0a@]Q1^k0E^2bM[1nPOgQOXk0SR1eNY1dUOljNkh0]U1O00001L'}
crop_box = np.array([3409,3,17,1201], dtype=np.uint32)

data = {'size': [15044, 7957], 'counts': b'l[`Sa3Ze0QR>jNV1jNV1iNW1iNV1jNW1iNW1iNW1iNW1iNW1iNW1iNW1iNW1iNW1iNW1iNW1iNW1iNjQZi0'}
crop_box = np.array([7883,12304,18,684], dtype=np.uint32)

data = {'size': [11736, 347], 'counts': b']Sib3g0f];d0ZOh0YOg0POR1QOk0]T[7'}
crop_box = np.array([ 320,5666,6,151], dtype=np.uint32)

data = {'size': [4048, 30], 'counts': b'4`0nm3b0UUL^OkNN`e3f6M>CS1mN;RbLeHQT3m7^jLiIkT3ba0ckL_[Ock2Te0QSMj[Ojl2We0PnLbYOb0\\1`0d0dP3ej0J0000000000000000X6gQMUmN]d2_[1PKf0ZOj1VNQ3oLe0[O4a<[CYi0gVORg6'}
crop_box = np.array([0,0,28,1598], dtype=np.uint32)

I have tried to check if e.g. the rle size is wrong or if pycocotools.mask.encode(pycocotools.mask.decode(data)) changes the RLE, but this does not seem to be the case.

I appreciate any insights you might have on this issue.

[qrmt]

After a valgrind / gdb session, I was able to isolate and fix the bug. See pull request 5.

[isarandi]

Great, thanks a lot!!