ronsaldo
commented
6 years ago Hi Luis,
All of this sounds really interesting, however these things are not easy to implement. In fact, one of the reasons for developing Lowtalk is to experiment with some of these things. Let me go through each one of your points.Allow native objects, that are outside the Smalltalk object memory and are not saved on the image. The Spur object headers for them have bit 54 set to 1. This will require adding a primitive to test this bit.
Allow native objects, that are outside the Smalltalk object memory and are
not saved on the image. The Spur object headers for them have bit 54 set to
- This will require adding a primitive to test this bit.
Why not just use a pointer for this? ExternalAddress on the FFI. In Pharo there is also the UFFI, with FFIExternalObject
Allow pointer mixes, that are objects that are made of managed (tracked by
the garbage collector) and unmanaged (not tracked by the garbage collector). This will require adding two new primitives, Behavior>>basicNew:unmanagedPointers: and Object>>unmanagedPointerCount.
This part is really tricky. With Clemént we had a discussion about this. The easiest way of achieving is to just reuse the format for a CompiledMethod.
In Lowtalk I am also experimenting with this a bit, but it is also tricky. The problem is not in the object format itself, but it is on subclassing. How should the layout of slots organized? what if we have subclasses? In Lowtalk I can at least abuse of the type system, which helps a lot on doing code generation.
Allow native contexts, that have the following instance variables: 'sender
contextData method v4 v5 v6 v7 v8'. contextData is a ByteArray containing the bytes of a CONTEXT structure (716 bytes in x86). method is a pointer to a NativeMethod, or nil if not yet initialised or queried. sender is the sender of the native context (See ContextPart>>sender). v4 through v8 are reserved for extensibility, and native contexts MUST have 8 instance variables to fit nicely with byte arrays containing x86 CONTEXT structures in native memory.
This is also tricky, but it is partially implemented in the Lowcode VM variant. The problem of changing the layout of the context metadata can have an impact in different parts of the VM code. With Lowcode I preferred to use the first four temporary variables to store the metadata for tracking a native context (native context presence/serialization, previous native context stack pointer, native context stack pointer, and metadata for shadow stack for doing inline FFI callouts). Using temporary variables has also the advantage of being able to access to its data from the image side easily.
Allow native methods, that have the following instance variables: 'address
size selector methodClass callingConvention numArgs portions v8 v9 v10 v11 v12 v13 v14'. v8 through v14 are reserved for extensibility, and native methods MUST have 14 instance variables to occupy exactly 64 bytes in 32 bit environments
How is this different of FFI? Do you want to store native code in the image? native code relocations can be tricky.
Support for two-way throwing and resuming of exceptions (at least in
Win32). A new exception class called NativeError will be added, that has the following instance variables in addition to the inherited variables: 'ntstatus flags returnAddress inner parameters translatedError nestedContext'. translatedError was used in internal experiments, please feel free to rename it. ntstatus is the NTSTATUS value of the exception (same as the exception code in Win32, in other operating systems you can autogenerate from other error code systems based on the NTSTATUS values table at MSDN https://msdn.microsoft.com/en-us/library/cc704588.aspx. flags is the exception flags (1 for unresumable exception, 2 for unwind, 4 for exit unwind, 8 for stack invalid, 16 for nested exception). returnAddress is the native return address, that can be nil for exceptions thrown from Smalltalk code. inner is the inner exception, a feature taken from C# and Java. parameters is an array of parameters. nestedContext can be used for nested exceptions, a feature of Structured Exception Handling.
- NTSTATUS values for common exceptions go here: 0xE0234902 for MessageNotUnderstood, 0xE0234903 for PrimitiveFailed (a new exception class that is to be added to make primitiveFailed resumable) and 0xE0234901 for insufficient object memory (you can use the Microsoft-defined code 0xC0000017 or this one, it's your choice). These three codes are custom NTSTATUS values.
This is really, really tricky. This also could introduce a performance overhead on each FFI callout. What about other platforms? what about different exceptions handling models? BTW, usually Win32 is the last platform to be supported, mostly because the build environment is based around Unix style configuration and building scripts, so Win32 is the hardest platform to support.
Numbered primitive support for reading and writing to native memory
(primitives 1010 and 1011)
This is already supported by the FFI primitives. For better performance, it is also possible to Lowcode which implements several inline primitives for doing this.
I am leaving a link to the Lowcode paper: https://hal.inria.fr/hal-01353884/file/Salg16a-IWST16-Lowcode.pdf
Soon I will have a Lowcode based backend for Lowtalk, which should expose more of the features provided by Lowcode to the image.
NativeObjectStub is a stub for a native object that is replaced by the
object when a message is sent to it. When an image is saved, pointers to native objects are replaced with pointers to stubs representing them. Identity must be preserved, such that the results of == and identityHash are not changed by the replacing operation. These stubs point to a string referencing the object, that can be passed to the native module handling the stub to resolve the stub, that is, retrieving a pointer to the native object corresponding to the stub.
This sounds really hard. What about dynamic allocation of objects?
NativeInstructionStream is similar to InstructionStream, except it works with native instructions. Clients of NativeInstructionStream implement a message, #receiveInstruction:, that takes as argument a NativeInstruction, whose protocol is platform-specific.
Do you know about NativeBoost and ASM-JIT? Are you planning on replicating a JIT on the image side? This sounds like a bad idea. It is better to go the full bytecode route with a FFI (the current VM), or the completely native route (Lowtalk, by replicating a full compiler pipeline modeled after LLVM)
What about the interactions with multi-threading?
Primitive 1010 is the readFromNativeMemory: primitive, that replaces the
bytes of the receiver with bytes from native memory starting at the address which is the argument, as an Integer. The primitive fails if the receiver is not a ByteArray or if the argument is not an Integer.
Primitive 1011 is the writeFromNativeMemory: primitive, that writes to native memory starting at the address which is the argument, as an Integer, the bytes of the receiver.
This seems to be already implemented by the FFI. For better performance we have Lowcode.
Primitive 1032 (Only in Win32 VM!) returns the segment of the Thread
Environment Block as a SmallInteger. In Windows 8.1 WOW64 today this should be 0x0053 (83).
Linux (and probably OS X too), does also use a segment register for thead local storage. The format is probably different (%fs vs %gs ).
Best regards, Ronie
2017-10-21 17:52 GMT-03:00 Luís Câmara notifications@github.com:
I'd like to add the following features to newer Spur VMs:
- Allow native objects, that are outside the Smalltalk object memory and are not saved on the image. The Spur object headers for them have bit 54 set to 1. This will require adding a primitive to test this bit.
- Allow pointer mixes, that are objects that are made of managed (tracked by the garbage collector) and unmanaged (not tracked by the garbage collector). This will require adding two new primitives, Behavior>>basicNew:unmanagedPointers: and Object>>unmanagedPointerCount.
- Allow native contexts, that have the following instance variables: 'sender contextData method v4 v5 v6 v7 v8'. contextData is a ByteArray containing the bytes of a CONTEXT structure (716 bytes in x86). method is a pointer to a NativeMethod, or nil if not yet initialised or queried. sender is the sender of the native context (See ContextPart>>sender). v4 through v8 are reserved for extensibility, and native contexts MUST have 8 instance variables to fit nicely with byte arrays containing x86 CONTEXT structures in native memory.
- Allow native methods, that have the following instance variables: 'address size selector methodClass callingConvention numArgs portions v8 v9 v10 v11 v12 v13 v14'. v8 through v14 are reserved for extensibility, and native methods MUST have 14 instance variables to occupy exactly 64 bytes in 32 bit environments.
- A new class (NativeCallGate) with only one instance and that contains methods for gating native methods.
- Support for two-way throwing and resuming of exceptions (at least in Win32). A new exception class called NativeError will be added, that has the following instance variables in addition to the inherited variables: 'ntstatus flags returnAddress inner parameters translatedError nestedContext'. translatedError was used in internal experiments, please feel free to rename it. ntstatus is the NTSTATUS value of the exception (same as the exception code in Win32, in other operating systems you can autogenerate from other error code systems based on the NTSTATUS values table at MSDN https://msdn.microsoft.com/en-us/library/cc704588.aspx. flags is the exception flags (1 for unresumable exception, 2 for unwind, 4 for exit unwind, 8 for stack invalid, 16 for nested exception). returnAddress is the native return address, that can be nil for exceptions thrown from Smalltalk code. inner is the inner exception, a feature taken from C# and Java. parameters is an array of parameters. nestedContext can be used for nested exceptions, a feature of Structured Exception Handling.
- NTSTATUS values for common exceptions go here: 0xE0234902 for MessageNotUnderstood, 0xE0234903 for PrimitiveFailed (a new exception class that is to be added to make primitiveFailed resumable) and 0xE0234901 for insufficient object memory (you can use the Microsoft-defined code 0xC0000017 or this one, it's your choice). These three codes are custom NTSTATUS values.
- Numbered primitive support for reading and writing to native memory (primitives 1010 and 1011).
SPECIFICATION FOR THE NEW SPECIAL OBJECTS ARRAY SLOTS
5 new special objects array slots will be allocated (numbers are based on Squeak 5.1 final):
- 61: NativeCallGate
- 62: NativeContext (subclass of NativeInstructionStream)
- 63: NativeError (subclass of Exception/Error)
- 64: NativeMethod (subclass of SequenceableCollection)
- 65: NativeObjectStub, or nil if you choose not to implement it
SPECIFICATION FOR ANOTHER CLASSES
NativeObjectStub is a stub for a native object that is replaced by the object when a message is sent to it. When an image is saved, pointers to native objects are replaced with pointers to stubs representing them. Identity must be preserved, such that the results of == and identityHash are not changed by the replacing operation. These stubs point to a string referencing the object, that can be passed to the native module handling the stub to resolve the stub, that is, retrieving a pointer to the native object corresponding to the stub.
NativeInstructionStream is similar to InstructionStream, except it works with native instructions. Clients of NativeInstructionStream implement a message, #receiveInstruction:, that takes as argument a NativeInstruction, whose protocol is platform-specific.
SPECIFICATION FOR THE NEW PRIMITIVES
Primitives 1000-1199 are reserved for my proposal and future versions of it.
Primitive 1010 is the readFromNativeMemory: primitive, that replaces the bytes of the receiver with bytes from native memory starting at the address which is the argument, as an Integer. The primitive fails if the receiver is not a ByteArray or if the argument is not an Integer.
Primitive 1011 is the writeFromNativeMemory: primitive, that writes to native memory starting at the address which is the argument, as an Integer, the bytes of the receiver.
Primitive 1013 is the unwind primitive, that unwinds the receiver, which is a NativeContext. The primitive fails if the receiver is not a NativeContext.
Primitive 1032 (Only in Win32 VM!) returns the segment of the Thread Environment Block as a SmallInteger. In Windows 8.1 WOW64 today this should be 0x0053 (83).
Primitive 1033 is the unmanagedPointerCount primitive, that answers the unmanaged pointer count of the receiver. The primitive fails if the receiver is not a pointer mix.
Primitive 1034 is the basicNew:unmanagedPointers: primitive, that answers a new instance of the receiver (which is a behaviour) with the number of indexable managed pointer fields specified by the first argument (sizeRequested) and with the number of unmanaged indexable pointer fields specified by the second argument (unmanagedPointers). The primitive fails if the receiver's format is not a format for pointer mixes; if either of the arguments are negative or not a SmallInteger.
Primitive 1043 is the supportsNativeContexts primitive. It should return true if the VM fully supports native contexts. A fallback may be written that returns false.
Primitive 1045 is the fullParse primitive, that fully parses (fills instance variables according to the bytes of the instruction) the receiver, which is a NativeInstruction. The primitive fails if the receiver is not a NativeInstruction. This primitive is optional.
Primitive 1075 is the isNativeObject primitive, that determines whether an object is a native object. A fallback may be written that returns false.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/157, or mute the thread https://github.com/notifications/unsubscribe-auth/ABepP46RejJ64T-ZYkTHteev7CzfskFcks5sulmIgaJpZM4QBtkU .
ghost
I'd like to add the following features to newer Spur VMs:
SPECIFICATION FOR THE NEW SPECIAL OBJECTS ARRAY SLOTS
5 new special objects array slots will be allocated (numbers are based on Squeak 5.1 final):
SPECIFICATION FOR ANOTHER CLASSES
NativeObjectStub is a stub for a native object that is replaced by the object when a message is sent to it. When an image is saved, pointers to native objects are replaced with pointers to stubs representing them. Identity must be preserved, such that the results of == and identityHash are not changed by the replacing operation. These stubs point to a string referencing the object, that can be passed to the native module handling the stub to resolve the stub, that is, retrieving a pointer to the native object corresponding to the stub.
NativeInstructionStream is similar to InstructionStream, except it works with native instructions. Clients of NativeInstructionStream implement a message, #receiveInstruction:, that takes as argument a NativeInstruction, whose protocol is platform-specific.
SPECIFICATION FOR THE NEW PRIMITIVES
Primitives 1000-1199 are reserved for my proposal and future versions of it.
Primitive 1010 is the readFromNativeMemory: primitive, that replaces the bytes of the receiver with bytes from native memory starting at the address which is the argument, as an Integer. The primitive fails if the receiver is not a ByteArray or if the argument is not an Integer.
Primitive 1011 is the writeFromNativeMemory: primitive, that writes to native memory starting at the address which is the argument, as an Integer, the bytes of the receiver.
Primitive 1013 is the unwind primitive, that unwinds the receiver, which is a NativeContext. The primitive fails if the receiver is not a NativeContext.
Primitive 1032 (Only in Win32 VM!) returns the segment of the Thread Environment Block as a SmallInteger. In Windows 8.1 WOW64 today this should be 0x0053 (83).
Primitive 1033 is the unmanagedPointerCount primitive, that answers the unmanaged pointer count of the receiver. The primitive fails if the receiver is not a pointer mix.
Primitive 1034 is the basicNew:unmanagedPointers: primitive, that answers a new instance of the receiver (which is a behaviour) with the number of indexable managed pointer fields specified by the first argument (sizeRequested) and with the number of unmanaged indexable pointer fields specified by the second argument (unmanagedPointers). The primitive fails if the receiver's format is not a format for pointer mixes; if either of the arguments are negative or not a SmallInteger.
Primitive 1043 is the supportsNativeContexts primitive. It should return true if the VM fully supports native contexts. A fallback may be written that returns false.
Primitive 1045 is the fullParse primitive, that fully parses (fills instance variables according to the bytes of the instruction) the receiver, which is a NativeInstruction. The primitive fails if the receiver is not a NativeInstruction. This primitive is optional.
Primitive 1075 is the isNativeObject primitive, that determines whether an object is a native object. A fallback may be written that returns false.