[Feature] Would it be possible to declare fields?

[Sergio0694]

Hey, just discovered this package, looks amazing! Thank you for your time building this! 😄

I have a question/proposal, would it be possible to support declaring fields? This would be useful when the type of such fields is not accessible from C# (eg. with internal types from other assemblies). Accessing and using those fields from IL would be easy, but from the readme it doesn't look like it's currently possible to declare the first right now? 🤔

Rationale

In the Microsoft.Toolkit.HighPerformance package (source here, API browser here) we have a number of types that basically act equivalent for the System.ByReference<T> type from CoreCLR, which is unfortunately internal (made a proposal about making it public a while back (here), but that's not really planned). The workaround I came up with is to just use a Span<T> as proxy for the ByReference<T> type, using MemoryMarshal.CreateSpan<T>(ref T, int). This is kinda ok when we can reuse the Span<T>.Length property for other purposes (basically as if it was just another int field in the parent type), but it's just unnecessary overhead when that's not needed. Specifically, in the Ref<T> and ReadOnlyRef<T> types.

I was toying with the idea of just writing a part of that type directly in IL, declaring a ByReference<T> field and simply using that directly from the various available APIs. Of course, this would only be for .NET Core 2.1, .NET Core 3.1 and .NET 5, and not for the other targets. Would something like this be possible? I'm actually curious in general even outside of this specific case 😄

Something like (just the idea):

[LocalField("System.ByReference`1", "reference")]
public readonly ref struct Ref<T>
{
    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public Ref(ref T value)
    {
        Type
            byRefType = Type.GetType("System.ByReference`1").MakeGenericType(typeof(T)),
            parameterType = typeof(T).MakeByRefType();
        MethodRef cctor = MethodRef.Constructor(byRefType, parameterType);

        Ldarg_0();
        Ldarg_1();
        Newobj(cctor);
        Stfld(FieldRef.Field(byRefType, "reference"));
    }

    public ref T Value
    {
        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        get
        {
            Type byRefType = Type.GetType("System.ByReference`1").MakeGenericType(typeof(T));

            Ldarg_0();
            Ldflda(FieldRef.Field(byRefType, "reference"));
            Call(MethodRef.PropertyGet(byRefType, "Value"));

            return ref IL.ReturnRef<T>();
        }
    }
}

Again congrats again for building this, I've been looking for something like this for a while! 🚀

[ltrzesniewski]

Hi, and thanks, I'm glad you like my lib! 😄

You're right: currently InlineIL only lets you insert IL instructions. I suppose that a feature like you describe could be added, but I'm not sure the CLR would let you load a type which violates visibility constraints. I never tried to build a type like this, so I don't really know. I guess I'll write a test to find out. 😉

[john-h-k]

iirc violating internal is allowed and it is part of how InternalsVisibleTo works, but would need to verify

[ltrzesniewski]

Hmm... you just reminded me of IgnoresAccessChecksToAttribute, which can take care of this problem if it occurs.

[Sergio0694]

Hey @ltrzesniewski, thank you for chiming in! I'm happy to hear you think this could be interesting 😄

Just to try this out and make myself more familiar with your lib, I tried the following:

// Dummy type just as a test
internal readonly unsafe ref struct ByReference<T>
{
    private readonly IntPtr _value;

    public ByReference(ref int x)
    {
        _value = (IntPtr)Unsafe.AsPointer(ref x);
    }

    public ref T Value => ref Unsafe.AsRef<T>((void*)_value);
}

// Some test method
public static void CreateAndTest<T>(ref T x, T test)
{
    IL.DeclareLocals(typeof(ByReference<>).MakeGenericType(typeof(T)));

    // var byRef = new ByReference(ref x);
    Ldarg_0();
    Newobj(MethodRef.Constructor(
        typeof(ByReference<>).MakeGenericType(typeof(T)),
        typeof(T).MakeByRefType()));
    Stloc_0();

    // byRef.Value = test;
    Ldloca_S(0);
    Call(MethodRef.PropertyGet(
        typeof(ByReference<>).MakeGenericType(typeof(T)), 
        "Value"));
    Ldarg_1();
    Stobj<T>();

    Ret();
}

This works absolutely fine (which is great!) Trying to replace that typeof(ByReference<>) with Type.GetType("System.ByReference``1") (the double backtick is just for GitHub's markdown) though fails to build with the following error:

Unexpected instruction, expected a type reference but was: IL_000e: call System.Type System.Type::GetType(System.String) - InlineIL requires that arguments to IL-emitting methods be constructed in place. (in System.Void FodyIL.Program::Create(T&,T) at instruction IL_000e: call System.Type System.Type::GetType(System.String))

I assume we'd need a new API to do this kind of "dynamic type loading" within an IL-emitting method? Thanks again for looking into this! 😊

[ltrzesniewski]

You're right: Type.GetType is not supported by InlineIL, and there are a few reasons for that:

• If you want a type from your own assembly, just use typeof, it's better in every way.
• If you want a type from a different assembly, TypeRef has a constructor which takes an assembly name and a type name as a string

I guess I could support calls to Type.GetType with an argument like "TypeName, AssemblyName" (or the full assembly-qualified name), but it did not strike me as a very good API to expose. The TypeRef(string, string) constructor just feels better.

[Sergio0694]

Oh I see, I completely missed the TypeRef(string, string) constructor, that's perfectly fine, yeah!

I'm definitely still missing something (probably obvious) though, when I try:

new TypeRef("System.Private.CoreLib", "System.ByReference`1").MakeGenericType(typeof(T))

I get this error:

Could not find type 'System.ByReference`1' in assembly 'System' (in System.Void FodyIL.Program::Create(T&,T) at instruction IL_003b: call System.Void InlineIL.IL::DeclareLocals(InlineIL.LocalVar[]))

Also tried:

new TypeRef(TypeRef.CoreLibrary, "System.ByReference`1").MakeGenericType(typeof(T)))

But this resulted in:

Could not find type 'System.ByReference`1' in assembly 'System.Runtime' (in System.Void FodyIL.Program::Create(T&,T) at instruction IL_003b: call System.Void InlineIL.IL::DeclareLocals(InlineIL.LocalVar[]))

Once again I might just be missing something obvious here 😅

Anyway since this still wouldn't work in my case as I'd still need support for declaring fields, I'll just want until you eventually feel like giving this a try instead of just posting you and pinging you in your notifications for now real reason 😄

[ltrzesniewski]

Don't worry, I'm happy to help and it's interesting to know what you tried and didn't work.

I guess the reason why the ByReference type is not found is that your assembly doesn't reference System.Private.CoreLib.

.NET Core assemblies reference the System.Runtime assembly (that's what TypeRef.CoreLibrary returns), which is a reference assembly that only exposes public types. You'd need to add an actual reference to System.Private.CoreLib for this to work.

[ltrzesniewski]

@Sergio0694 actually you don't need InlineIL to access System.ByReference.

I made a hack you may be interested in: ByRefTest.zip

Here's how it works:

• The CoreLib project builds an assembly with the following full name: System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e. It contains a public System.ByReference type with a dummy implementation.
• The ByRefTest project references CoreLib with Private="false", which prevents it from being copied to the build output. The real System.Private.CoreLib will be used at runtime.
• It also contains an [assembly: IgnoresAccessChecksTo("System.Private.CoreLib")] attribute. It turns out it's necessary, otherwise the CLR refuses to load the type.

And that seems to work just fine! 😄

I initially tried to use the real System.Private.CoreLib along with IgnoresAccessChecksToGenerator for this, but it turns out Roslyn doesn't like having two core libraries, even if one of them is behind an alias.

[Sergio0694]

Hey @ltrzesniewski - I'm just blown away here and I had no idea this was even remotely possible, thank you! I'm sure that IgnoresAccessChecksTo attribute might also come in handy in the future in other scenarios too!

I've opened a draft PR integrating that trick you discovered, let's see how that goes! 😄

Thank you again for your time and help!

[ltrzesniewski]

You're welcome! 😄

This could also have been implemented in a slightly different way, for example by using a Fody in-solution weaver that would inject the necessary code and assembly reference, but it would probably be harder to read and maintain than the way I made it.

One more thing I thought of after posting this: System.Private.CoreLib will be referenced in the .deps.json file. I don't know if it's an issue, hopefully not.

I guess I won't be adding the feature you asked for InlineIL in the end, since the way I've shown here solves the problem much better anyways.

[Sergio0694]

Yeah getting Fody involved would make the whole thing even harder to maintain, you're right. After all, this is basically just a temporary workaround, ideally I'd just want to switch to proper ref T fields if/when they will become available (maybe C# 10 and .NET 6? Who knows!). This is an incredibly powerful workaround for the moment though! 😄 And it's already miles better than just hacking around with a fake Span<T> with unused length.

About that .deps.json file, not sure, but I guess someone reviewing that PR will be able to chime in on that. Yeah I'd agree with you, this solution is effectively much easier to implement than actually writing new support for declaring fields with InlineIL. That said, I'll still definitely be using your lib in the future in other projects, as it's just great! 🚀

[ENikS]

@ltrzesniewski @Sergio0694

Guys, you are my heroes! I been looking for something like this for last two years. You have no idea how much time I killed looking for ref solution. You just made my life SO MUCH EASIER!!!

The trick is so brilliantly simple 😄

[ENikS]

Well, it works in core apps, but I can't make it work in framework. Any ideas?

[ltrzesniewski]

Sorry, but this trick won't work in .NET Framework, as its runtime does not support ref fields. There's no System.ByReference<T> type in mscorlib at all. Spans are implemented differently on .NET Framework - you get the so called "slow spans" that contain an object field, an offset and a count, instead of a ByReference and a count.

But from what I understand from the other thread, you need references to data on the stack only, which means the GC does not need to move these references. If you can use InlineIL or Reflection.Emit, I suppose you could use an IntPtr field in your struct, that you then reinterpret-cast to a ref T with custom IL code similar to what Unsafe.AsRef does. I guess it may require some adjustments to work though (at least change the signature from void* to IntPtr). Be careful, as any pointer to data on the managed heap will probably break after a GC.

You should be also able to just use a Span<T> instead of all this mess I guess.

[ENikS]

It is all on the stack, no concern for GC at all. Perhaps I could even live with strait Unsafe in the framework. If I want to use InlineIL, how would I do it properly?

[ltrzesniewski]

If you can use unsafe code (as I suppose you can on .NET Framework), then storing a void* field in your struct and using Unsafe.AsRef<T> and Unsafe.AsPointer to convert between the pointer and a ref would be by far the easiest option.

If you'd like to use InlineIL, you can do the same thing but copy/paste the AsRef<T> and AsPointer implementations from the example. I guess you could change void* to IntPtr and it should still work. But I don't really see the point, using Unsafe would be the better solution.

[ENikS]

How would you suggest I deal with this on netstandard2.1; netstandard2.0; netstandard1.6 ? I would hate to duplicate all of these structures with classes.

[ltrzesniewski]

The System.Runtime.CompilerServices.Unsafe NuGet package supports all of these frameworks, so as long as you can use unsafe code (because of void*), the same implementation as in .NET Framework should work for all of these as well.

[ENikS]

Well, this is where it gets you! Office 365 sandbox prefers netstandard and prohibits Unsafe. Some for other environments I have to support. Any other ideas?

[ltrzesniewski]

I'm not familiar with the Office 365 sandbox, so I won't be able to give you a definite solution. But you may try to use InlineIL and see what happens.

I don't know if the following will work, but I suppose it could:

[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static ref T AsRef<T>(IntPtr source)
{
    IL.DeclareLocals(
        false,
        new LocalVar("local", typeof(int).MakeByRefType())
    );

    IL.Push(source);
    Stloc("local");
    Ldloc("local");
    return ref IL.ReturnRef<T>();
}

[MethodImpl(MethodImplOptions.AggressiveInlining)]
public static IntPtr AsPointer<T>(ref T value)
{
    Ldarg(nameof(value));
    Conv_U();
    return IL.Return<IntPtr>();
}

Use an IntPtr field in your struct, and round-trip between that and a ref to a struct with these two methods.

[ENikS]

Will try it, thank you!

[ENikS]

Lucas,

I am contemplating integration of Fody (or one of the libraries like Fody) into static code generation tool for Unity Container DI. I wander if you would be interested to discuss this idea? Could you email me unitycontainer at eniks dot com to discuss the possibility?

[ltrzesniewski]

@ENikS if you'd like to integrate Fody into your library, I think you should rather talk with @SimonCropp.

I suggest you open an issue in the Fody repository to discuss what you have in mind.