Native Interface (super issue)

[wks]

This is an outline of issues related to the native interface, that is, interacting with the native world. This topic includes but is not limited to object layout, pointer types, object pinning and foreign function calls. We should open other issues to discuss concrete problems.

• Make a platform-specific Mu specification
  • Address some of the following issues, including object layout, calling convention, ...
  • Draft for AMD64: https://github.com/microvm/microvm-spec/wiki/native-interface-x64-unix
• Type system: (https://github.com/microvm/microvm-meta/issues/34)
  • Raw pointer types
  • Structure types with native/explicit object layout
  • Union types (unlikely to have in Mu)
  • Mapping Mu types to C types and native object layout: (in the Native Interface chapter in the spec)
• Memory space beyond heap/stack/global
  • Memory spaces with various constraints
  • Is it movable, pinnable, has reference, can be referenced to, GC-traced, GC-collected, ...?
  • Object pinning: https://github.com/microvm/microvm-meta/issues/28
  • If object pinning is allowed, what does "pin" mean?
• Foreign function interfaces
  • Calling foreign functions from Mu: (The CCALL instruction. See the spec)
  • Calling C functions
  • System calls
  • Calling back to Mu from foreign functions: https://github.com/microvm/microvm-meta/issues/39
  • From C functions
  • Signal handling

The following should be addressed by a higher-level abstraction:

• Loading native libraries
  • The client loads libc and finds the address of dlopen, dlsym, dlclose and dlerror. Then the CCALL instruction takes care of the rest by calling them.
• Loading "heavier-weight" Mu bundles (currently called MuLF): https://github.com/microvm/microvm-meta/issues/30

The following are not related to the native interface, but are related to raw memory:

• How to expose the address of objects so that the user can analyse the memory behaviour? (This involves profiling, too. We may open a dedicated issue for profiling.)

[john5f35]

Hi Kungshan, I remembered that we talked about redefining the reference types into 3: ref, iref and pointer (which refers to any untracted memory location). Has the documentation been updated? Is the pointer type 'weakref' in the documentation?

[wks]

No. Pointer will be a separate type.

A weak reference is still a traced object reference. But when an object is not referred from any strong references (both ref and iref are strong), the GC is permitted (but not forced) to recycle that object and set all weak references to that object to NULL. It is the same as Java's WeakReference and Python's weakref https://docs.python.org/3/library/weakref.html

The document has not been updated to reflect the pointer type. The real bottleneck is to understand “object pinning”. It is not as simple as “telling the GC not to move an object” though it is the case for many GC algorithms.

Kunshan

John Zhang notifications@github.com于2015年1月5日星期一写道：

Hi Kungshan, I remembered that we talked about redefining the reference types into 3: ref, iref and pointer (which refers to any untracted memory location). Has the documentation been updated? Is the pointer type 'weakref' in the documentation?

— Reply to this email directly or view it on GitHub https://github.com/microvm/microvm-meta/issues/24#issuecomment-68655498.

[wks]

Some drafts are added in the spec. Links:

• https://github.com/microvm/microvm-spec/wiki/native-interface
• https://github.com/microvm/microvm-spec/wiki/native-interface-x64

[wks]

Note on variadic functions: The native interface is not designed to call variadic functions. Reasons are:

1. The unsafe native interface is mainly designed to make system calls, not calling arbitrary C functions. i.e. It is more likely Mu will call write rather than fprintf.
2. In necessary cases where this interface is used to inter-operate between languages (e.g. Mu calling CPython, or Mu calling JNI of another JVM not upon Mu), the API should use "well-defined" non-variadic functions (such as Call<type>MethodA rather than Call<type>Method). In rare cases where non-variadic functions are not available, the client has to write some wrappers in C.
   • Alternatively, use LibFFI. Let Mu call LibFFI, and let LibFFI call printf. This may not be efficient, but no variadic functions are.

The reasons why Mu itself did not adopt variadic functions are:

1. C-style variadic functions are unsafe. As for C, variadic functions interpret the argument types at the callee site, usually via additional information from other arguments, such as the formater in printf. Mis-interpreting an argument has undefined behaviour. This is especially true when interpreting an reference as a plain value, or vice versa.
2. Java-style variadic functions is actually passed as an array argument.

Platform-specific details: On x86_64, when calling a variadic function, the AL register contains the number of vector (SSE) registers used. www.x86-64.org/documentation/abi.pdf

Here is a comment on this decision in the ABI: https://gcc.gnu.org/ml/gcc-patches/2010-07/msg01505.html

THe orginal problem was the fact that early K8 chips had no way of effectively storing SSE register to memory whithout knowing its type. So the stores in prologue executed very slow when reformating happent. Same reason was for not having callee saved/restored SSE regs.

On current chips this is not big issue, so I do not care what way we output. In fact I used to have patch for doing the jz but lost it. I think we might keep supporting both to get some checking that ABI is not terribly broken (i.e. that no other copmilers just feeds rax with random value, but always by number of args).

Strangely the AMD64 SysV ABI also says XMMx registers are caller-saved.

There is no such register required on ARM.

For implementations, if it wants to support calling variardic functions directly, it can always call variardic functions as if they were not, and conservatively set AL to the number of FP arguments for all native functions.