Language features for FFI

[sjindel-google]

This is a meta-issue for the language features which would simplify the FFI API.

/cc @dcharkes @mkustermann @mraleph @leafpetersen @lrhn

Type-level correspondences

The FFI has as set of types which correspond to C types: Int8, Int64, Double, Float, Pointer, etc. Each of these types has a related Dart type, e.g. int, double or Pointer. These types cannot be instantiated in Dart (exception as annotations), but are used as type variables and in function signatures to specify additional behavior. For example, the Pointer class has two methods load and store:

class Pointer<T extends NativeType> {
  U load<U>();
  store(Object value)
}

Unfortunately, this API can't enforce the relationship between the NativeTypes and corresponding Dart types (this is done via a custom pass in the front-end). So the following illegal code will type-check:

Pointer<Int64> x = ...;
x.load<String>();

Access to function return type

Enforcing the correspondence between these two sets of types is further complicated by the use of function types to represent native function signatures. The following API method creates a native function address which calls a Dart method:

external Pointer<NativeFunction<T>> fromFunction<T extends Function>(
    Function f, {dynamic exceptionalReturn: 0});

Here, we want to enforce that the type of exceptionalReturn corresponds to the return type of f, and that all the types mentioned in the signature of f correspond piece-wise to the native types mentioned in the function type T.

The same holds for asFunction, although it's slightly simpler.

Constant type arguments

We require that arguments to certain API functions are compile-time constants to ensure that we can compile all the required trampolines in AOT. For example:

class Pointer<T extends NativeType> {
    external R asFunction<R extends Function>();
}

In this case we need to ensure that at any call-site: p.asFunction<R>, both R and the receiver type T need to be known at compile-time. The same applies to fromFunction.

Sealed classes and virtual statics

We need to seal some classes including NativeType and all its children (Int8, Double, etc.) except Struct. In addition, we need to allow classes to extend Struct but not implement Struct (we only generate code for classes which extend Struct and the VM relies on having the generated code for any subtype of Struct).

Moreover, we need to prohibit extending a class which extends Struct (because we cannot generate meaningful field offsets), even though Struct itself can be extended.

Part of the reason we need to seal the hierarchy this way is that we cannot express the all the operations the VM needs to perform on one of these types in its signature. For example, the VM needs to manufacture Dart objects corresponding to a native-type in the return value of a native method or the argument to a callback:

typedef NativeFn = Int8 Function();
typedef DartFn = int Function();

DartFn nativeMethod = Pointer<NativeFunction<NativeFn>>.fromAddress(0).asFunction<DartFn>();

If we had virtual statics, we could express the needed functionality in the interface of NativeFunction:

class NativeType<D> {
  virtual static int get size;
  virtual static NativeType<D> convertWord(int word);
  virtual static NativeType<D> convertDoubleWord(int word1, int word2);
  // ...
}

then:

class Int64 extends NativeType<int> {
  external virtual static int get size;
  // ...
}

The same goes for structs. Currently we disallow implementing Struct because we generate code like this:

class Coord extends Struct<Coord> {
  @Double()
  double x;

  @Double()
  double y;

  @Pointer()
  Coord next;
}

Gets elaborated to:

class Coord extends Struct<Coord> {
  Coord.#fromPointer(Pointer<Coord> coord) : super._(coord);

  Pointer<Double> get _xPtr => addressOf.cast();
  set x(double v) => _xPtr.store(v);
  double get x => _xPtr.load();

  Pointer<Double> get _yPtr => addressOf.offsetBy(...).cast();
  set y(double v) => _yPtr.store(v);
  double get y => _yPtr.load();

  ffi.Pointer<Coordinate> get _nextPtr => addressof.offsetBy(...).cast();
  set next(Coordinate v) => _nextPtr.store(v);
  Coordinate get next => _nextPtr.load();

  static final int #sizeOf = 24;
}

We prohibit implementing Struct because the VM needs to lookup the #sizeOf and #fromPointer members for any class which is a subtype of Struct, and we only generate them in classes which extend Struct. If we had virtual statics, we could express these in the interface:

abstract class Struct<S> {
  virtual static Struct<S> fromPointer(Pointer<S> ptr);
  virtual static int get sizeOf;
}

@dcharkes Are there others?

[dcharkes]

For reference, all the error messages we throw in our VM transformer:

FfiTypeMismatch:
  # Used by dart:ffi
  template: "Expected type '#type' to be '#type2', which is the Dart type corresponding to '#type3'."
  external: test/ffi_test.dart

FfiTypeInvalid:
  # Used by dart:ffi
  template: "Expected type '#type' to be a valid and instantiated subtype of 'NativeType'."
  external: test/ffi_test.dart

FfiTypeUnsized:
  # Used by dart:ffi
  template: "Method '#name' cannot be called on something of type '#type' as this type is unsized."
  external: test/ffi_test.dart

FfiFieldAnnotation:
  # Used by dart:ffi
  template: "Field '#name' requires exactly one annotation to declare its C++ type, which cannot be Void. dart:ffi Structs cannot have regular Dart fields."
  external: test/ffi_test.dart

FfiNotStatic:
  # Used by dart:ffi
  template: "#name expects a static function as parameter. dart:ffi only supports calling static Dart functions from c."
  external: test/ffi_test.dart

FfiFieldInitializer:
  # Used by dart:ffi
  template: "Field '#name' is a dart:ffi Pointer to a struct field and therefore cannot be initialized before constructor execution."
  external: test/ffi_test.dart

FfiExtendsSealedClass:
  # Used by dart:ffi
  template: "Class '#name' cannot be extended."
  external: test/ffi_test.dart

FfiStructGeneric:
  # Used by dart:ffi
  template: "Struct '#name' should not be generic."
  external: test/ffi_test.dart

FfiWrongStructInheritance:
  # Used by dart:ffi
  template: "Struct '#name' must inherit from 'Struct<#name>'."
  external: test/ffi_test.dart

For every one of these being able to express them in the type system would remove a ffi-specific check. Edit: Or rather a feature that would prevent us having to check this manually.

[lrhn]

We now have sealed and final classes.

We don't have virtual statics, and have no plan to add such.

Most of the other features mentioned here are ones we wouldn't consider adding for any other reason than to support dart:ffi, which seems to be in the perfect position to give those errors itself (after all, nobody else can compile code importing dart:ffi). I don't see a general use for requiring type arguments to be constant (outside of constant invocations), like I also don't see a general use for requiring actual arguments to be constant expressions. It's generally something that is suggested in order to achieve something else, but rarely actually achieves that in practice.

The one thing you don't get is warnings from the analyzer, unless the analyzer cooperates and recognized the VM requirements (maybe by using annotations like @pragma('vm:must_be_constant')).

Also, a part of the problems comes from using Dart function types to represent native function types. Maybe extension types will be a way to go in the future:

extension type Int32(int _) implements int {}

This creates a subtype of int, which means that Int32 Function(int32, int32) is a subtype of int Function(Int32, Int32), so you can see that it returns int. Still won't allow you to abstract over parameter lists. We'd have to allow something like int Function(...) to be a supertype of all (non-generic) functions returning an int. (That one might actually be occasionally useful, but nobody will look for such a request in this issue.)