how to express "pointer to array of n elements" when n is known only in runtime

[tatumizer]

Right now, pointer supports [] and []= operators without bounds check. It would be very nice to have a variant of pointer with bounds check, where the size is provided dynamically. Such pointer will be essentially equivalent to what some languages call "slice". Further, we could create slices of slices, represented by the same kind of pointer.

Some background: I'm trying to experiment with direct interface to ZIG programming language (totally bypassing C). It supports ffi via C ABI (not clear what they mean by C ABI, because there's no such thing as C ABI, but I'm pretty sure it's the same so called "C ABI" as dart supports in its ffi).

ZIG is a very nice language, designed to compete with C. Performance is in the same ballpark as C, sometimes even better. It's a much safer language than C. It comes with all batteries included: you can download ZIG alone (even of Windows), and it just works with no strings attached. There's a good standard library. In short, literally a piece of gold.

Zig supports slices with bounds check (in "Safe" mode, which is marginally slower than "Fast" mode"). Slices in zig are recommended as a standard way of working with arrays. You rarely need a concept of pointer there (except when you use a pointer to a single item, with no pointer arithmetic - they really differentiate between pointer to one and pointer to many!). The problem is how to represent slices while communicating with dart over ABI ("slices" are not supported by ABI). I got in touch with ZIG people, they said I have to create a struct and put a pointer and length there. When dart calls ZIG passing a slice, encode it as struct; in the ZIG function, immediately convert it to slice, and that's it. But I'm thinking of how dart can do the same when ZIG returns a slice encoded as struct. Unfortunately, this can't be done b/c "pointer to array of n elements" is not supported.

Please put me out of my misery. :smile:

Unrelated question: how does dart ffi know how to call malloc/free in the native library? dll doesn't export malloc/free by default, and there are no standard conventions on how to make these functions available via ABI. Example: native function returns a pointer to some data, ownership is on dart's side, so how to free this memory after use?

[dcharkes]

Hi!

C ABI

Yes, the C ABI is the calling convention on the operating system that is used on the boundaries between shared objects (.so on Linux/Android, .dylib on MacOS/iOS, and .dll on Windows.). And yes, this is exactly what dart:ffi is targeting.

I'm trying to experiment with direct interface to ZIG programming language

Cool stuff! 😎

I got in touch with ZIG people, they said I have to create a struct and put a pointer and length there.

The C language regards pointer arguments, fixed-length array arguments, and unsized array arguments all as the same.

https://www.tutorialspoint.com/cprogramming/c_passing_arrays_to_functions.htm

However, a struct nesting a fixed size array is not the same in the C ABI. See the this example.

dart:ffi is a C FFI, so if we were to support the array syntax for function calls, it would be the same as pointers. However, as you pointed out the ZIG compiler actually uses structs underneath, so that is not going to be compatible.

But I'm thinking of how dart can do the same when ZIG returns a slice encoded as struct. Unfortunately, this can't be done b/c "pointer to array of n elements" is not supported.

Does it return a struct by value? or a pointer to a struct? Both cases should be supported in dart:ffi. (On the dev channel, not yet on stable, inline arrays landed recently: https://github.com/dart-lang/sdk/issues/45122).

class MyStruct {
  @Array(8)
  Array<Uint8> fixedLengthArray;
}

You can return MyStruct both as Pointer<MyStruct> (by reference) and MyStruct (by value) from ffi calls.

Please put me out of my misery. 😄

I hope this reply does. 😄

how does dart ffi know how to call malloc/free in the native library

Please use package:ffi for this. It contains the calloc allocator which you can use to allocate and free memory.

Example: native function returns a pointer to some data, ownership is on dart's side, so how to free this memory after use?

It is best practise to free in the same language as you allocate. So if you allocate in C, and then pass ownership to Dart. Write a C function that does the freeing, and call that function from Dart through FFI when you're done with the data.

[dcharkes]

You can use asTypedList, giving you a TypedData with bounds checking. You cannot get back to a Pointer from that.

Also, you can could .cast<MyStruct>().fixedLengthArray on your pointer, giving you an Array with bounds checking. You can also not get back to a Pointer from that.

Both the the TypedData and Array do not give you access to the Pointer because they can be either backed by Dart or by C memory.

Alternatively, you could write your own wrapper class for Pointer which forwards the [] and []= operators.

class Slice<T extends NativeType> {
  final Pointer<T> _pointer;

  final int bound;

  Slice(this._pointer, this.bound);

  Pointer<T> get unbounded => _pointer;

  void _checkBound(int bound) {
    if (bound < 0 || bound >= this.bound) {
      throw RangeError.range(bound, 0, this.bound - 1);
    }
  }
}

extension Uint8Slice on Slice<Uint8> {
  int operator [](index) {
    _checkBound(index);
    return _pointer[index];
  }

  void operator []=(int index, int value) {
    _checkBound(index);
    _pointer[index] = value;
  }
}

You cannot have Slice implement Pointer, because dart:ffi would not know what to do with these when passed to FFI calls to C.