Grumpy panics when it can't fulfill an allocation

[alanjds]

google/grumpy#166 opened by @S-YOU on 19 Jan 2017

This python code

x = [True] * sys.maxint

cause

resultElems := make([]*Object, newNumElems)

in seq.go to be panicked. I can't think of a way to handle that.

[alanjds]

Comment by trotterdylan Thursday Jan 19, 2017 at 15:23 GMT

---

I think CPython would raise MemoryError in this case but I'm not sure that we can do the same here: there doesn't appear to be a way to catch failed memory allocations in Go.

[alanjds]

Comment by meadori Friday Jan 27, 2017 at 15:22 GMT

---

CPython does raise a memory error, but not by catching a failed malloc. The check in list_repeat is preemptive:

if (n > 0 && Py_SIZE(a) > PY_SSIZE_T_MAX / n)
        return PyErr_NoMemory();

We can have similar checks in Grumpy.

[alanjds]

Comment by S-YOU Friday Jan 27, 2017 at 15:32 GMT

---

x = [True] * (sys.maxint / 1000000) still panic, so PY_SSIZE_T_MAX is not a ideal solution I guess.

[alanjds]

Comment by meadori Friday Jan 27, 2017 at 16:01 GMT

---

Hmmm, so the panic is from makeslice: panic: runtime error: makeslice: len out of range. We attempt to check for this in Grumpy. However, the check is different than that of the check used for slices in the Go runtime. Maybe we should implement a check more similar to the Go one?

[alanjds]

Comment by trotterdylan Friday Jan 27, 2017 at 17:38 GMT

---

Oh that's interesting. So CPython does not raise MemoryError when malloc() fails?

[alanjds]

Comment by meadori Friday Jan 27, 2017 at 18:10 GMT

---

It does raise MemoryErrors for failed allocation too. For the list repeat case, however, it will try the check I posted in a previous comment before even attempting to allocate memory. CPython does these kinds of preemptive checks in other places too.

Go also does checks like these, hence the panic from makeslice. I think we should model our check off the makeslice one and create a MemoryError if it fails.