rolfbjarne
commented
3 days ago I like this idea, it can make the bindings a lot simpler, both for us, and particularly for customers. The current binding procedure/API is rather unintuitive.
One potential place where things would get uglier is availability attributes, and how we use them to exclude APIs on certain platforms. Roslyn isn't able to remove stuff from the compilation, so any of the No* attributes won't work, we'd have to use conditional compilation instead:
[BindingType]
public class CBManager : NSObject {
#if __TVOS__
[SupportedOSPlatform ("tvos13.0")]
[UnsupportedOSPlatform ("ios")]
[UnsupportedOSPlatform ("maccatalyst")]
[UnsupportedOSPlatform ("macos")]
[Export ("authorization", ArgumentSemantic.Assign)]
public partial static CBManagerAuthorization Authorization { get; }
#endif
}Another is the BindAs attribute, we might have to flip it to be BindFrom:
[BindFrom (typeof (NSNumber))]
[Export ("roll", ArgumentSemantic.Strong)]
nfloat? Roll { get; }I think a good plan forward would be:
- Come up with a good name for the roslyn-based generator (rgen? 🙈)
- Add a bare-bones roslyn-based generator, that does the very minimal possible to be even remotely useful (probably just support generating a single NSObject-derived class, no methods, no protocols, no nothing), just to get all the plumbing right.
- Add it to the build system, so that customer projects automatically get it (note that we don't have to limit ourselves to binding projects, we can enable this for all of our projects).
- Add lots of tests & documentation (both end user documentation and technical documentation).
- Look in the current api definitions for something that needs a single (simple at first) feature to be implemented, and port that manually to the roslyn-based generator. We can also look at missing APIs from xtro. Here it's important to add features one by one to the roslyn-generator, because otherwise PRs will snowball and likely stall (there are many things we can improve with the current bindings, so tackling them in tiny pieces makes sure we actually get them done).
- Binding new APIs might be somewhat slower than porting existing APIs, but it has the advantage that we improve our API coverage at the same time.
- On the other hand, binding existing APIs has the advantage that we'll be able to compare the resulting API diff, to make sure the generated API doesn't change.
- Once a decent number of features have been implemented, we could look into creating a tool to automatically convert bgen-based api definitions to roslyn-based ones.
Misc notes:
No intellisense. The compiler has no access to the generated class.
This is fixable with bgen; the problem is that design-time builds are disabled.
Enabling design-time builds is easy, but we don't want to run design-time builds remotely from Windows, so then the problem becomes how to not run design-time builds remotely, which turned out to be a rather bigger problem, which I've been working on on and off for quite a while now (related to the next point).
- Make bgen work from Windows #16611
bgen works fine from Windows already (we run the bgen tests on Windows in fact), the problem is that resources in binding projects don't (which I'm working on fixing). Switching to Roslyn won't make this easier, it's completely unrelated.
mandel-macaque
RFC: Migrate bgen to use roslyn instead of the reflection API.
The following is a proposal to migrate the current bgen implementation, based on the reflection API, to a roslyn generator. A proof of concept can be found at https://github.com/mandel-macaque/macios-generator
Introduction
The Microsoft.iOS/MacOS bindings relies in a custom tool called bgen. bgen generates code by reading a dll that contains all the binding definitions described as a interface. Each interface represents a native class, bgen loads the assembly, reads the metadata that was used to tag the interfaces and generates the final classes and any needed trampoline.
Although the approach was the best possible one when roslyn did not exists, the solution is limited by the reflection API and the fact that an intermediate generation process is needed outside the normal compilation. The following problems come to mind:
For developers:
For the SDK result:
Timing
Up until the release of net9 and C# 13 the language did not support partial properties which limited the possibility of using Roslyn for the binding generations since if would heavily impact the quality of the final API. The new language addition allows to have a Roslyn code generator that is able to cover all the binding cases.
Benefits
A Roslyn Code Generator has access to both the SyntaxTree and the Semantic model of the current compilation, both pieces allow to minimize the need of type annotations, adds better support for nullability and all possible new language features as well as it allows to share knowledge between a code generator and a Roslyn analyzer improving the feedback loop for developers and the quality of the SDK. Some of the direct benefits are:
Known issues with bgen
Due to the nature of the reflection API, there are several issues that the current generator cannot fix easily:
Build complexity
The build of the Microsoft.iOS SDK is very complex because it occurs in several phases. This makes most new developers struggle to understand in which of the available sections under frameworks.sources each of the files has to go. This is specially painful when we are adding structure that need to have the same size and layout in all steps of the build. Moving to Roslyn allows to all files be in the same compilation unit as well as it allows to have a csproj per platform to organize the build.
Having all code in the same build also allows to remove several classes that we have that are used to indentify which frameworks are present per platform. With all files in the same context, Roslyns semantic model can be used to find a symbol and decide if a class is generated or not.
API changes
Moving away from bgen to Roslyn means that we have to change the API to support intellisense on customer projects. The API change is 1:1 meaning that the migration from one version to another can be automated. Moving to Roslyn does not force our customers to move to Roslyn. bgen can be distributed customers as long as it is needed.
The following code snippets show the differences between API
Class definitions
bgen requires all classes to be interfaces, and use the BaseType attribute to define the class hierarchy:
With a move to Roslyn the bindings can be changed to use classes and normal class inheritance allow users to access the class in their current compilation unit.
Protocol implementations in bgen are represented by class inheritance. That is complicated and in some occasions it forced developers to create empty interfaces as with the following example:
With Roslyn this becomes much simpler:
Class visibility is also simplified, while bgen requires classes to be marked with [Abstract] to state that they are, a roslyn code generator does not have that limitation, same with static classes:
Which becomes
Because we will be working with classes, we can simply list the interfaces that are implemented and the code generator will write the needed interface methods.
Notes
Class must ALWAYS be
partiala Roslyn Analyzer can help avoiding common mistakes and Sharpie will usually be the tool used to write bindings.Methods and Properties
The new code generation relies heavily on the fact that properties and methods can be partial. The changes for method and properties are minimun, while for constructors we have to do some extra changes.
The bgen binding of methods and properties look like the following:
With roslyn we can define the method and properties as follows, pay attention at the fact that we added partial and virtual as needed. The Static and Abstract attributes are not longer needed. Nullability is directly supported, that removes the need of the NullAllowed attribute.
The WeakDelegate removal deserves as special mention. Because we do not longer need the WarpAttribute, the weak delegate property can be written in plain C#, which will be generated by objective sharpie.
Notes
All methods in the SDK are virtual, this will have to be respected in the Microsoft.iOS binding by adding the
virtualkeyword in the method definition. A roslyn analyzer can help spot this possible common mistakeConstructor
Constructor are a special case of methods, the biggest problem is that a constructor in C# 13 cannot be partial, nevertheless that does not suppose a major problem, since we can declare the init methods as constructors and the Roslyn code generator can create a constructor that will take the same parameters. This init methods should be marked as private and can me marked to be inlined by the compiler.
In bgen constructors are defined as follows:
which becomes:
The above code can easily generate the following constructors for the class:
Notes
Adding more than on constructor with the same parameters will result in a compilation error, this is an issue present already in bgen with a simple workaround that can be implemented as a code generation in a Roslyn analyzer.
Categories
Categories are probably the easiest bindings to port (see migration). They represent a extension class for a specific type. Currently in bgen, as with all other bindings, they are represented as an interface decorated with the Category attribute:
Easily converted to a class via:
The generator can easily generate the partial method that uses the self pointer.
Protocols
Protocols represent interfaces that are implemented in the native world. bgen does not add
Iin front of protocols which is very confusing to C# developers, but that is done because everything is an interface and later theIwill be added in the generated code. This is not longer the need with Roslyn which can define the interface and then in the implementing class add the missing methods.Old bgen binding:
new code:
More importantly we can support optional methods in protocols with the use of default interface methods.
bgen code
Roslyn approach:
Enums and Smart Enums
Enums perse should be a problem for bgen but they add some compliations to the build:
Moving to a Roslyn based solution solve this problem, there are no two steps in the build process and enums are allways present.
Smart enums are, with categories, the simples biding type to port since they are extension classes. The only work to do is to annotate them with the BindingType attribute.
General
All classed, interfaces and enums that have to be generated will have to be annotated with the BindingType attribute. This annotations is to allow the Roslyn code generator filter the nodes that need to be generated. bgen does not need that since it assumes that is interested in all types, the code generator on the other hand needs a way to filter.
Multi language support
Because Roslyn allow use to separate the understanding of the syntaxt tree and the semantic model from the code generation, moving to this approach will allow to have different backends that can write different types of trampolines opening the door for swift bindings once there is support from the runtime team.
Work items
Moving from bgen to any new implementation is a scary task and we need to be able to do it in a way that if we face a set back, priorities might psuh back, people come and go etc. The good thing of moving to Roslyn is that we do not have to port all binding. Since Roslyn is part of the last step, we can add the code generator to the last compilation, after bgen has generated code and just focus on smaller tasks. A tentative road map would be:
Migration
The migration of the SDK does not need to be done by hand. We can have 3 different tools that will share a lot of knowledge that will help with the migration of the SDK and that can later be a product for customers: