[ffi] Add support for abi-specific structs

[dcharkes]

Sometimes structs are defined differently on different platforms. (Some fields might be commented out on some platforms.)

A workaround is to have a different struct for the different platforms, but that cascades into everything that is using that struct (function signatures, struct fields, and pointers).

We could introduce an AbiSpecificStruct that exposes all the different fields on all platforms in one api.

@AbiSpecificStructMapping({
  Abi.linuxX64: MyStructLinux(),
  Abi.windowsX64: MyStructWindows(),
  // ...
})
class MyStruct extends AbiSpecificStruct {
  external int a;
  external int b;
}

This would be the struct/union counterpart of https://github.com/dart-lang/sdk/issues/42563.

edit: We would probably make an API similar to https://api.dart.dev/stable/2.17.0/dart-ffi/AbiSpecificInteger-class.html with annotations instead of types.

[artob]

@dcharkes That's pretty neat. What happens if accessing field external int b on Windows, in this example? A compiler error, I trust?

[LeoSandbox]

Hey guys, I'm looking to use this package https://pub.dev/packages/steamworks on both Windows and MacOS. This issue is blocking the package to work properly.

I've been making a breakthrough RPG with Flutter for the past 2 years and about to start the last stretch of production to ship a demo and then release. This package will be used to send Steam Achievement to Steam API.

Is there any way I can help on this issue? Can you point me to the right resource so I can try to work on it?

[dcharkes]

Hi @LeoSandbox nice to meet you!

We welcome contributions! ❤️

The first step is to build a Dart SDK locally:

• https://github.com/dart-lang/sdk/blob/main/docs/Building.md
• https://github.com/dart-lang/sdk/blob/main/CONTRIBUTING.md

The best starting point would be to look at the commits that introduced ABI-specific integers:

• https://dart-review.googlesource.com/c/sdk/+/221501
• https://dart-review.googlesource.com/c/sdk/+/221821

The analyzer is a different implementation than the common compiler frontend (CFE).

To run the analyzer and CFE from source to be able to run the debugger in vscode:

            {
                "name": "dart analyzer.dart",
                "type": "dart",
                "request": "launch",
                "program": "pkg/analyzer_cli/bin/analyzer.dart",
                "args": [
                    "tests/ffi/static_checks/vmspecific_static_checks_array_test.dart",
                ],
                "toolArgs": [],
                "enableAsserts": true,
                "cwd": "${workspaceFolder}",
            },
            {
                "name": "dart gen_kernel.dart",
                "type": "dart",
                "request": "launch",
                "program": "pkg/vm/bin/gen_kernel.dart",
                "args": [
                    "--enable-experiment=records",
                    "--platform=${workspaceFolder}/xcodebuild/ReleaseARM64/vm_platform_strong.dill",
                    "pkg/vm/testcases/transformations/ffi/address_of_struct_element.dart",
                ],
                "toolArgs": [],
                "enableAsserts": true,
                "cwd": "${workspaceFolder}",
            },

You might need to adjust the build folder to out/ instead of xcodebuild/ if you're not on MacOS.

The use of the API could look something like the following:

@AbiSpecificStructMapping({
  Abi.linuxX64: MyStructLinux(),
  Abi.windowsX64: MyStructWindows(),
  // ...
})
final class MyStruct extends AbiSpecificStruct {
  // No annotation, the native types are per ABI
  external int a;

  // The ABI-specific struct should have the superset of all fields of the struct per ABI.
  external int b;

  // The ABI-specific struct should have the superset of all fields of the struct per ABI.
  external int c;
}

final class MyStructLinux extends Struct {
  @Int8()
  external int a;

  @Int8()
  external int b;
}

final class MyStructWindows extends Struct {
  @Int64()
  external int a;

  @Int64()
  external int c;
}

I expect a PR to implement this feature to roughly have the following changes:

• [ ] A new class AbiSpecificStruct (and AbiSpecificUnion) in sdk/lib/ffi/abi_specific.dart
  • [ ] With a long doc-comment including examples
• [ ] An implementation for static errors
  • [ ] The ABI-specific struct must have the superset of all fields of the structs mentioned in the annotation
  • [ ] The ABI-specific struct must have a Dart type compatible with the fields of the structs mentioned in the annotation (e.g. Windows cannot have @Double() external double a; while Linux has @Int8() external int a;).
  • [ ] The annotation must only mention other structs if it's a struct.
  • [ ] The error messages are defined in pkg/analyzer/messages.yaml and pkg/front_end/messages.yaml. You'll need to run the code generators mentioned in those files to get the Dart methods to use in the analyzer and CFE to report the errors.
• [ ] Tests for static errors (both for CFE and analyzer) in tests/ffi/static_checks/vmspecific_static_checks_abi_specific_struct_test.dart
• [ ] An implementation in the CFE in pkg/vm/lib/transformations/ffi/*.dart. I believe AbiSpecificNativeTypeCfe can be mostly reused to deal with structs/unions besides only integers. Basically the idea is that the external Foo x is transformed to a getter and setter with the real body that has a big switch based on ABI in it.
• [ ] Unit test cases for the CFE similar to pkg/vm/testcases/transformations/ffi/abi_specific_int.dart. They show the transformed code before it is send to the backend.
• [ ] Integration test cases that exercise the code in tests/ffi/*_test.dart
  • [ ] The best way to do this is probably to extend the test generator in tests/ffi/function_callbacks_structs_by_value_generated_test.dart to be able to generate tests for AbiSpecificStructs. Then you don't have to write tests by hand.
• [ ] The most tricky part will most likely be the backend implementation for passing AbiSpecificStructs by value in FFI calls and callbacks (Pointer.fromFunction and NativeCallables). The implementation there would be in runtime/vm/compiler/frontend/* See the linked PR for what was changed for AbiSpecificInteger.

You'll have to touch many parts of the Dart compiler for this feature (similar to https://dart-review.googlesource.com/c/sdk/+/221501). However, the feature should be somewhat straightforward to implement due to its similarity to the AbiSpecificIntegers. Please feel free to ping me or this issue with any questions!