Open generic interceptors do not compare marshalling behaviors

[monkey0506]

Describe the issue

Open generic interceptors compare the interface generic type arguments to select an appropriate implementation class, but do not compare any marshalling behaviors (except when there is no static calling convention). This issue does not apply to closed generic interceptors, because they are guaranteed to have the same marshalling for every intercepted method.

var action = INativeAction<string>.FromFunctionPointer<TMarshaller>(functionPtr);

Here, TMarshaller is a generic type parameter of some containing type or method, constrained by where TMarshaller : IMarshaller<TMarshaller>, new(). For example, if this line was contained in a method such as:

public static INativeAction<string> GetNativeAction<TMarshaller>(nint functionPtr) where TMarshaller : IMarshaller<TMarshaller>, new()
{
    return INativeAction<string>.FromFunctionPointer<TMarshaller>(functionPtr);
}

And then this method was invoked by GetNativeAction<CdeclMarshaller>(functionPtr) and GetNativeAction<StdCallMarshaller>(functionPtr), then we will generate an interceptor that performs checks like:

if (typeof(X) == typeof(string))
{
    return (INativeAction<X>)(object)(new NativeAction_U10384200(functionPtr)); // marshaller was `CdeclMarshaller`
}
if (typeof(X) == typeof(string))
{
    return (INativeAction<X>)(object)(new NativeAction_S1348320298(functionPtr)); // marshaller was `StdCallMarshaller`
}

We have distinct implementation classes for each marshaller, but because the interface type arguments are the same, the second branch of our interceptor will always be unreachable.

We cannot split this into separate interceptors, because the two calls refer to a single line in the source (inside the body of GetNativeAction).

Proposed solution

Given this example it may seem that the solution is to carry the generic type argument for marshaller, so we can perform a similar Type check. Rather, this approach would still fail if the marshaller is not supplied at all (by the non-generic factory methods), yet the marshalling behaviors are the same.

Here, we must perform runtime checks of the marshalling behaviors. This was previously done in the incremental-generator branch by caching an equatable representation of the MarshalAsAttributes so that the factory methods were not recreating these objects every time. We can take the same approach here.

When creating the open generic interceptor source text, we have a collection of ImplementationClasses, which each in turn carry a MarshalInfo describing the marshalling behaviors for the implementation. From the implementation classes, we could generate an additional set to represent each MarshalAsAttribute as it would be represented at runtime, and a collection (referencing that set) of each MarshalParamsAs collection. These can then be used by the interceptor to perform the necessary runtime checking to select the correct implementation class.

Additional considerations

See RuntimeMarshalAsAttributeCollection from the incremental-generator branch. This was never represented in the interceptors branch because it was thought to be unnecessary due to the way MethodReferences were compared; #33 shows that this was misguided.

[monkey0506]

Actually, because the MarshalAsAttributes can now only be supplied by the marshaller type, this can be done more simply by checking the typeof the marshaller. Handling the CallingConvention is already a supported case if it is not a static value.

[monkey0506]

Fixed by #36.