Feature request: low level memory API

[pekd]

Some language interpreters have to deal with native memory (e.g. Sulong, truffle86, and every language which implements or uses the Truffle NFI). The only way to access native memory right now is the Unsafe, but it's quite limited (some operations are not supported at all) and it shouldn't be used since it's not part of the public Java API.

There should be a Truffle API for low level memory operations similar to this:

public interface NativeMemory {
    // read access
    public byte  getI8(long address) throws SegfaultException, BusException;
    public short getI16(long address) throws SegfaultException, BusException;
    public int   getI32(long address) throws SegfaultException, BusException;
    public long  getI64(long address) throws SegfaultException, BusException;

    // write access
    public void setI8(long address, byte value) throws SegfaultException, BusException;
    public void setI16(long address, short value) throws SegfaultException, BusException;
    public void setI32(long address, int value) throws SegfaultException, BusException;
    public void setI64(long address, long value) throws SegfaultException, BusException;

    // atomic operations: cmpxchg
    public byte  cmpxchgI8(long address, byte expected, byte value) throws SegfaultException, BusException;
    public short cmpxchgI16(long address, short expected, int value) throws SegfaultException, BusException;
    public int   cmpxchgI32(long address, int expected, int value) throws SegfaultException, BusException;
    public long  cmpxchgI64(long address, long expected, long value) throws SegfaultException, BusException;

    // atomic operations: add
    public byte  lockaddI8(long address, byte value) throws SegfaultException, BusException;
    public short lockaddI16(long address, short value) throws SegfaultException, BusException;
    public int   lockaddI32(long address, int value) throws SegfaultException, BusException;
    public long  lockaddI64(long address, long value) throws SegfaultException, BusException;

    // memory management: low level
    public long mmap(long address, long size, int prot, int flags) throws InvalidArgumentException;
    public void munmap(long address, long size) throws InvalidArgumentException;
    public void mprotect(long address, long size, int prot) throws InvalidArgumentException;
    public long mremap(long old, long oldsize, long newsize, long flags) throws InvalidArgumentException;

    // memory management: high level
    public long malloc(long size) throws OutOfMemoryError;
    public long realloc(long address, long size) throws OutOfMemoryError;
    public void free(long ptr);
}

The read/write operations should be self-explaining; note that the exceptions here are essential. They provide the ability to catch segfaults (SIGSEGV) and bus errors (SIGBUS) in the program. In Sulong this means a crash of the guest program can print a useful message like e.g. a stack trace (instead of a VM crash), in truffle86 this means it's possible to implement segfault handlers for the guest program which is necessary for some programs. I know that these functions probably require changes in HotSpot/OpenJDK to be able to catch the signal and throw exceptions to the Java code, but that's exactly the reason why it's almost impossible to do this efficiently in a language interpreter without having this in the Truffle API.

The atomic operations are necessary to implement low level locking mechanisms when using native memory; e.g. the atomic addition on native memory is not available through any standard JDK class. Maybe more atomic operations like atomic xchg might be useful, but I'm not sure which atomic operations are available on all architectures. However, I know that it's a major pain to implement everything solely with cmpxchg (e.g. using the Unsafe right now).

Of course the memory management functions could be invoked using JNI/SystemJava, but the point of this API is to have all necessary memory related operations in one place and make such JNI/SystemJava code unnecessary for most memory operations.

[remkop]

This looks vaguely related to JEP 370 and memaccess, but more low level and more extensive. Maybe the goals are too different, but perhaps this proposal could be fed back into panama as a potential use case?

[boris-spas]

Tracking internally as Issue GR-20935.