Platform Specific Architecture Specification with @Native

[tsavo-at-pieces]

Hey all!

I have been working on a package (runtime_native_semaphores) and ran into an interesting scenario:

Not sure if this is a bug or not but while I was implementing sem_open bindings on Unix (specifically MacOS arm64 and x86_64 architectures) I discovered that mode_t had variable types.

Notably, on arm64 mode_t needs to be passed/bound as an UnsignedLong and on x86_64 architectures it needs to be passed/bound as a UnsignedShort

This was my workaround:

typedef sem_open_function_type<M> = Pointer<sem_t> Function(Pointer<Char>, Int, VarArgs<(mode_t<M>, UnsignedInt)>);
typedef dart_sem_open_function_type = Pointer<sem_t> Function(Pointer<Char> name, int oflag, int mode, int value);

Pointer<sem_t> sem_open(Pointer<Char> name, int oflags, [int? mode, int? value]) =>
    sem_open_callable(name, oflags, mode ?? MODE_T_PERMISSIONS.RECOMMENDED, value ?? 1);

// if we are on MacOS Arm64, then our mode_t is UnsignedLong otherwise it is UnsignedShort
final sem_open_pointer = Abi.current() == Abi.macosArm64
    ? DynamicLibrary.process().lookup<NativeFunction<sem_open_function_type<UnsignedLong>>>('sem_open')
    : DynamicLibrary.process().lookup<NativeFunction<sem_open_function_type<UnsignedShort>>>('sem_open');

final dart_sem_open_function_type sem_open_callable = Abi.current() == Abi.macosArm64
    ? sem_open_pointer.cast<NativeFunction<sem_open_function_type<UnsignedLong>>>().asFunction()
    : sem_open_pointer.cast<NativeFunction<sem_open_function_type<UnsignedShort>>>().asFunction();

where the above code came from this file

This was the only exception that needed to be made as my other bindings i.e. sem_unlink, sem_post, sem_wait, sem_trywait and sem_close were all successfully implemented leveraging the new @Native decorator like so:

/// [sem_post] function shall unlock the semaphore referenced by sem by
/// performing a semaphore unlock operation on that semaphore.
/// If the semaphore value resulting from this operation is positive,
/// then no threads were blocked waiting for the semaphore to become unlocked;
/// the semaphore value is simply incremented.
///
/// If the value of the semaphore resulting from this operation is zero,
/// then one of the threads blocked waiting for the semaphore shall be
/// allowed to return successfully from its call to [sem_wait].
@Native<Int Function(Pointer<sem_t>)>()
external int sem_post(Pointer<sem_t> sem_t);

/// [sem_close] closes the named semaphore referred to by sem,
/// allowing any resources that the system has allocated to the
/// calling process for this semaphore to be freed.

/// On success [sem_close] returns 0; on error, -1 is returned, with
/// [errno] set to indicate the error.
@Native<Int Function(Pointer<sem_t>)>()
external int sem_close(Pointer<sem_t> sem_t);

/// [sem_unlink] removes the named semaphore referred to by name.
///  The semaphore name is removed immediately.  The semaphore is
///  destroyed once all other processes that have the semaphore open
///  close it.
@Native<Int Function(Pointer<Char>)>()
external int sem_unlink(Pointer<Char> name);

Is there a world where one might be able to leverage something like @Native<T>(abi: Abi.macosArm64) or @Native<T>(on: Abi.macosArm64) to help determine which of two architecture specific bindings should be activated at runtime/compilation time?

It could perhaps look something like this:

typedef SemOpenNativeFunctionType<M> = Pointer<sem_t> Function(Pointer<Char>, Int, VarArgs<(mode_t<M>, UnsignedInt)>);

@Native<SemOpenNativeFunctionType< /* Passing in type M as UnsignedLong */ UnsignedLong>>(on: Abi.macosArm64)
@Native<SemOpenNativeFunctionType</* Passing in type M as UnsignedShort */ UnsignedShort>>(on: Abi.macosX64)
external Pointer<sem_t> sem_open(Pointer<Char> name, int oflag, int mode, int value);

or perhaps (although likely harder to do in dart as one might implement in other languages):

typedef SemOpenNativeFunctionType = Pointer<sem_t> Function(Pointer<Char>, Int, VarArgs<(mode_t, UnsignedInt)>);

@Native<SemOpenNativeFunctionType>(typeMappings: [
AbiTypeMapping(on: Abi.macosArm64, argument: 'mode_t', type: UnsignedLong), 
AbiTypeMapping(on: Abi.macosX64, argument: 'mode_t', type: UnsignedShort),
])
external Pointer<sem_t> sem_open(Pointer<Char> name, int oflag, int mode, int value);

Maybe I'm missing something obvious on my side - any thoughts/help would be greatly appreciated!

Nevertheless, great work! It's lovely to use so far 🙌

[tsavo-at-pieces]

@mraleph @dcharkes perhaps this is something that one of you might know about?

Regardless, just something I noticed and it's not broken at the moment although did trip me up pretty hard. I was writing some pretty funny C code to emulate and debug this with ffigen 😅

Thanks! -T

[dcharkes]

@tsavo-at-pieces nice to meet you!

If mode_t is an UnsignedLong or UnsignedShort, we should generate an AbiSpecificInteger.

/// The C `mode_t` type.
@AbiSpecificIntegerMapping({
  Abi.macosArm64: Uint64(),
  Abi.macosX64: Uint16(),
})
final class Mode extends AbiSpecificInteger {
  const Mode();
}

See the documentation on type-map in the readme on how to make FFIgen generate Mode instead of UnsignedLong or UnsignedShort.

type-map:
  'native-types':
    'mode_t':
      'lib': 'your_lib_where_you_define_Mode'
      'c-type': 'Mode'
      'dart-type': 'int'

[tsavo-at-pieces]

Nice to meet you as well @dcharkes wonderful work being done over in these parts!

I did make it that far which was how I was able to determine the nuance for sem_open 🙌

That said I was just using ffigen in development and ended up removing it from the final dependency once I knew the types thanks to the elegant @Native decorators ✨

I guess my question was more around how I could achieve a similar effect only using the decorators when I do know the type discrepancies between architectures.

Lmk thoughts and thanks again for the quick ping back here!

[dcharkes]

I guess my question was more around how I could achieve a similar effect only using the decorators when I do know the type discrepancies between architectures.

I'm not sure I understand the question. I was thinking to use AbiSpecificInteger for both sem_t and mode_t.

/// mode_t
@AbiSpecificIntegerMapping({
  Abi.macosArm64: Uint64(),
  Abi.macosX64: Uint16(),
})
final class Mode extends AbiSpecificInteger {
  const Mode();
}

/// sem_t
@AbiSpecificIntegerMapping({
  // ...
})
final class Sem extends AbiSpecificInteger {
  const Sem();
}

@Native<Pointer<Sem> Function(Pointer<Char>, Int, VarArgs<(Mode, UnsignedInt)>)>()
external Pointer<Sem> sem_open(Pointer<Char> name, int oflag, int mode, int value);

Is there a reason you cannot use AbiSpecificInteger for mode_t?

That said I was just using ffigen in development and ended up removing it from the final dependency once I knew the types thanks to the elegant https://github.com/Native decorators ✨

FFIgen can generate @Native annotations:

# ffigen.yaml
ffi-native:
  assetId: 'myasset' # Optional.

[tsavo-at-pieces]

Apologies @dcharkes - I meant specifically without ffigen at all i.e. I'm am hand-writing the @Native decorators similar to native_synchronization package

I was unsure how to specify the Abi specific type alongside the hand written decorators and you gave me exactly what I was missing - I completely overlooked extending AbiSpecificInteger and was trying to use a typedef 🫠

I have updated the code and this all makes a lot more sense now 😅

I have updated the package and we are good to go 🙌 feel free to check it out on pub.dev | runtime_native_semaphores

[tsavo-at-pieces]

Cheers!