[Native Image] Max memory pool size is exceeded before GC

Describe the Issue

Garbage collection is currently done on the allocation slow path after the max eden size was already exceeded by a previous allocation. For example, if new TLAB allocationA causes the max size to be exceeded, the pool max size will remain exceeded until new TLAB allocationB happens. It would be better to do GC when the allocation that actually causes the threshold to be exceeded happens.

This is because, on the slow path, maybeCollectOnAllocation is called too early. maybeCollectOnAllocation calls shouldCollectOnAllocation which checks whether the eden size threshold is exceeded. The problem is that maybeCollectOnAllocation is called before increaseEdenUsedBytes is called, so shouldCollectOnAllocation doesn't take into account the size of the current allocation. Is there a reason for this, or is this a bug?

Using the latest version of GraalVM can resolve many issues.

[X] I tried with the latest version of GraalVM.

GraalVM Version

Latest. All versions of GraalVM are affected.

Operating System and Version

Fedora Linux amd 64

Diagnostic Flag Confirmation

[ ] I tried the -H:ThrowMissingRegistrationErrors= flag.

Run Command

Run any native image executable using the serial GC.

Expected Behavior

Garbage collection should happen as soon as the max memory pool sizes have been exceeded, not at an arbitrary point in the future.

Actual Behavior

When the max pool sizes have been exceeded a GC does not happen. A GC happens on the next TLAB allocation instead.

Steps to Reproduce

Build a Native Image executable with the latest GraalVM. Set a small heap size. Run the executable.

Additional Context

No response

Run-Time Log Output and Error Messages

No response

oracle / graal