Expose top-level nullability information from reflection

[terrajobst]

With C# 8, developers will be able to express whether a given reference type can be null:

public void M(string? nullable, string notNull, IEnumerable<string?> nonNullCollectionOfPotentiallyNullEntries);

(Please note that existing code that wasn't compiled using C# 8 and nullable turned on is considered to be unknown.)

This information isn't only useful for the compiler but also attractive for reflection-based tools to provide a better experience. For example:

• MVC

  • Provides a way to automatically deserialize inputs to controller methods ("model binding")
  • Would like to provide model validation so that the existing pattern would allow code to bail early
  • Without it, customers would have to apply a custom attribute, such as [Required], or resort to additional null-checks
  • Only needs top-level annotations, i.e. string? but not nested, such as IEnumerable<string?>

• EF

  • Provides a way to generate database schemas from user classes ("code first")
  • Would like use nullable information to infer whether columns should be null or non-null (they already do that for nullable value types).
  • Without it, customers would have to apply a custom attribute to repeat that information.
  • Also only needs top-level annotations

The nullable information is persisted in metadata using custom attributes. In principle, any interested party can already read the custom attributes without additional work from the BCL. However, this is not ideal because the encoding is somewhat non-trivial:

• Custom attribute might be generated. The custom attribute might have been generated (meaning is embedded in the user's assembly) or might use the to-be-provided attribute.
• Encoded as a byte array. The tri-state is encoded as a linearized version of the constructed generic type.
• Compressed. Right now, each member will have the attribute when nullability is turned on but this causes metadata bloat. We're working on a proposal that allows the containing member, type, or assembly to state a default to reduce the number of attribute applications.

It's tempting to think of nullable information as additional information on System.Type. However, we can't just expose an additional property on Type because at runtime there is no difference between string (unknown), string? (nullable), and string (non-null). So we'd have to expose some sort of API that allows consumers to walk the type structure and getting information.

Unifying nullable-value types and nullable-reference types

It was suggested that these APIs also return NullableState.MaybeNull for nullable value types, which seems desirable indeed. Boxing a nullable value type causes the non-nullable representation to be boxed. Which also means you can always cast a boxed non-nullable value type to its nullable representation. Since the reflection API surface is exclusively around object it seems logical to unify these two models. For customers that want to differentiate the two, they can trivially check the top-level type to see whether it's a reference type or not.

API proposal

namespace System.Reflection
{
    public sealed class NullabilityInfoContext
    {
        public NullabilityInfo Create(ParameterInfo parameterInfo);
        public NullabilityInfo Create(PropertyInfo propertyInfo);
        public NullabilityInfo Create(EventInfo eventInfo);
        public NullabilityInfo Create(FieldInfo parameterInfo);
    }

    public sealed class NullabilityInfo
    {
        public Type Type { get; }
        public NullableState ReadState { get; }
        public NullableState WriteState { get; }
        public NullabilityInfo? ElementType { get; }
        public ReadOnlyCollection<NullabilityInfo>? GenericTypeArguments { get; }
    }

    public enum NullableState
    {
        Unknown,
        NotNull,
        MaybeNull
    }
}

Sample usage

Getting top-level nullability information

private NullabilityInfoContext _nullabilityContext = new NullabilityInfoContext();

private void DeserializePropertyValue(PropertyInfo p, object instance, object? value)
{
    if (value == null)
    {
        var nullabilityInfo = _nullabilityContext.Create(p);
        var allowsNull = nullabilityInfo.WriteState != NullableState.NotNull;
        if (!allowsNull)
            throw new MySerializerException($"Property '{p.GetType().Name}.{p.Name}'' cannot be set to null.");
    }

    p.SetValue(instance, value);
}

Getting nested nullability information

class Data
{
    public string?[] ArrayField;
    public (string?, object) TupleField;
}
private void Print()
{
    Type type = typeof(Data);
    FieldInfo arrayField = type.GetField("ArrayField");
    FieldInfo tupleField = type.GetField("TupleField");

    NullabilityInfoContext context = new ();

    NullabilityInfo arrayInfo = context.Create(arrayField);
    Console.WriteLine(arrayInfo.ReadState);         // NotNull
    Console.WriteLine(arrayInfo.Element.ReadState); // MayBeNull

    NullabilityInfo tupleInfo = context.Create(tupleField);
    Console.WriteLine(tupleInfo.ReadState);                        // NotNull
    Console.WriteLine(tupleInfo.GenericTypeArgument[0].ReadState); // MayBeNull
    Console.WriteLine(tupleInfo.GenericTypeArgument[1].ReadState); // NotNull
}

Custom Attributes

The following custom attributes in System.Diagnostics.CodeAnalysis are processed and combined with type information:

• [AllowNull]
• [DisallowNull]
• [MaybeNull]
• [NotNull]

The following attributes aren't processed because they don't annotate static state but information related to dataflow:

• [DoesNotReturn]
• [DoesNotReturnIf]
• [MaybeNullWhen]
• [MemberNotNull]
• [MemberNotNullWhen]
• [NotNullIfNotNull]
• [NotNullWhen]

@dotnet/nullablefc @dotnet/ldm @dotnet/fxdc @rynowak @divega @ajcvickers @roji @steveharter

[steveharter]

Due to 5.0 schedule constraints, this is being moved to Future.

[steveharter]

Here's a strawman2 proposal (updated from strawman1 proposal)

• Add extension methods to the existing System.Reflection.Extensions assembly.
  • This avoids runtime knowledge and pay-to-play for consumers.
• These lookup the nullable attributes and return a class that returns the nullable information and possibly other attribute-based information such as [DoesNotReturn].
  • This is somewhat slow and no extra caching is performed (like similar cases elsewhere). Caching should be done at a higher level, if necessary.
  • First a check is done to see if the type is in a nullable context or not. If the type is not in a nullable context, return "CLR semantics" of NotNull or MayBeNull depending on if the type is a reference type, value type, or Nullable<T>.
  • The more I look into this, the more complexity I find:
  • The [NullableAttribute] semantics are non-trivial
  • The nullable context attribute can be in several locations
  • It is non-obvious whether the current nullability attributes affect in or out nullability. For example, the names "AllowNull" and "MaybeNull" don't provide that context . This design tries to make that easier by having both "in" and "out" properties each with their own enum.
  • We could add additional properties such as "bool AllowsNull {get;}" that can be used for simple scenarios that don't need to differentiate between in and out (especially true for properties; parameters differ more than properties here).
• The extensions should work for both runtime-based and designtime-based reflection types (via MetadataLoadContext).
  • This means we can't return System.Attribute instances since MLC can't create attributes since that could execute code.

Sample usage

PropertyInfo propertyInfo = typeof(TestClass).GetProperty("MyProperty")!;
AttributedInfo attributeInfo = propertyInfo.GetAttributedInfo();
Assert.Equal(NullableInCondition.AllowNull, attributeInfo.NullableIn);
Assert.Equal(NullableOutCondition.MaybeNull, attributeInfo.NullableOut);

Proposed API (in System.Reflection namespace):

+public static class FieldInfoExtensions
{
+    public static AttributedInfo GetAttributedInfo(this FieldInfo fieldInfo);
}

public static class MethodInfoExtensions
{
+    public static AttributedInfo GetAttributedInfo(this MethodInfo method);
}

+public static class ParameterInfoExtensions
{
+    public static AttributedInfo GetAttributedInfo(this ParameterInfo parameter);
}

public static class PropertyInfoExtensions
{
+    public static AttributedInfo GetAttributedInfo(this PropertyInfo property);
}

+public sealed partial class AttributedInfo
+{
+    public bool HasNullableContext { get; } }

+    // The two common methods:
+    public NullableInCondition NullableIn { get; } }
+    public NullableOutCondition NullableOut { get; } }

+    // Possibly simple helper that ignores "in"\"out" attributes and uses [Nullable] and return type characteristics:
+    // public bool AllowNull { get; }

+    // We can expose non-nullability state as well:
+    public DoesNotReturnCondition DoesNotReturn { get { throw null; } }

+    // Set for [NotNullIfNotNull] and [DoesNotReturnIf]
+    public string[]? MemberReferences { get; }

// todo: expose info for generic parameter types and array element type
+}

+ public enum NullableInCondition
+{
+    NotApplicable,
+    AllowNull,
+    DisallowNull
+}

+public enum NullableOutCondition
+{
+    NotApplicable,
+    MaybeNull,
+    MaybeNullWhenFalse,
+    MaybeNullWhenTrue,
+    NotNull,
+    NotNullWhenTrue,
+    NotNullWhenFalse,
+    NotNullIfNotNull
+}

+public enum DoesNotReturnCondition
+{
+    NotApplicable,
+    Returns,
+    DoesNotReturn,
+    DoesNotReturnWhenTrue,
+    DoesNotReturnWhenFalse
+}

Prototype at https://github.com/dotnet/runtime/compare/master...steveharter:ReflectionExt

[roji]

This is somewhat slow and no extra caching is performed (like similar cases elsewhere). Caching should be done at a higher level, if necessary.

For caching to be done effectively outside the API, wouldn't this have to expose [NullableContext] information (which it really shouldn't, since that's a metadata encoding detail)?

For example, say I ask for nullability info on method Foo. Internally, this would potentially walk all the way up to the assembly to find the nearest [NullableContext]. If I then ask for info on method Bar on that same type, that same walk up to the assembly would have to happen again - the user has no way of caching anything.

Given the complexity of nullability metadata, it seems important for NullableContext to be calculated and cached internally, e.g. at the type level. I'm also not sure why this can't be pay-per-play by doing things lazily...

[jaredpar]

The approach I'd originally envisioned here is that the API would track only the annotation state of the types. Essentially is the type oblivious, annotated or not annotated. That is the key information that is missing from the reflection API today. The other values listed in the proposal here include attributes which are readily available today.

The other issue is that the attributes are being represented as enum values. It is reasonable to expect that the langauge will add more attributes in the future. The evidence being that we've done so already and had rejected a few other attributes under the "lets see if the need is really there before doing this.". Generally I believe the BCL has shied away from using enum to represent values that can expand in the future.

[MichalStrehovsky]

Cc @vitek-karas @marek-safar for linker

We have form factors (Blazor-WASM) where nullability information is dropped on deployment by default because it's a size concern.

Linker is not aware of these attributes right now - this information comes from an XML embedded into CoreLib. If APIs like this are added, it will likely require hardcoding the knowledge of the attributes and the new APIs into linker.

[jkotas]

We may need new efficient and linker-friendly custom attribute querying APIs to build this feature on top of: https://github.com/dotnet/runtime/issues/44319#issuecomment-722588772 . The linker would need to understand these new custom attribute querying APIs, but it would not need to have hardcoded knowledge of every place that queries specific attributes.

[GSPP]

Caching could be performed even if this API is part of the System.Reflection.Extensions assembly. There could be a NullableReflectionContext of sorts. It would be a class with query methods. An instance of that class would maintain a cache.

I personally like that the main reflection APIs are not polluted with C# concepts. This has been done in the past and it was, in my opinion, a design mistake. It is not true to the spirit of having a language-independent CLR.

[krwq]

How would properties with [AllowNull] and [DisallowNull] be expressed? I feel like there is different nullability for setter and getter. Similar question for ref arguments? Also should the nullable custom attributes be just exposed through IEnumerable<Attribute> NullableAttributes (or similar)? That should be more future-proof rather than trying to express that with enum

[roji]

Caching could be performed even if this API is part of the System.Reflection.Extensions assembly. There could be a NullableReflectionContext of sorts.

That's certainly possible, but it would require consuming applications to manage the cache instance themselves, making it harder to reuse the information across different parts of the program etc. Re C# concepts, I am not sure to what extent nullability is (at least in theory) supposed to be a C#-only thing - it makes sense for F#/VB to be able to express and consume the same concepts at some point in the future.

[jzabroski]

To me, it's a combination of "how hard is the decoding" and "how likely is it that someone needs to answer that question using reflection APIs". -- @terrajobst

@terrajobst How do you map out what is hard to decode and how likely it is that someone needs to answer that question? Is there a list of user stories? For example, c# now has language design documents available: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/proposals/csharp-7.1/async-main A lot of these have specific motivations for why something was introduced, beyond "totes brill upvotes".

I am curious, because I personally can think of several user stories (and large in circulation projects) for various aspects of reflection that are common to applications I write. I also know how to rewrite some of these reflection calls to use more efficient expression trees, etc. But how common is this knowledge? It seems socially unjust and bad for .NET popularity for it to be kept in the knowledge of someone with 18+ years experience with .NET

[jzabroski]

The approach I'd originally envisioned here is that the API would track only the annotation state of the types.

@jaredpar What do you mean by the "annotation state of the types"? I could not tell if you were responding to Immo or Steve's strawman API proposal.

I mostly just want to know if the properties can be nullable, non-nullable or unknowable. I did not really understand what value Steve's proposal brought to the table, other than forcing me to understand the literal difference between AllowNullAttribute and MaybeNullAttribute. If I were to leave .NET for 5 years and come back and read this API on docs.microsoft.com, I would just go back to Google and search StackOverflow for "how to use reflection to find nullable reference types" and I would find this post , briefly lament that I am not using the "canonical API" (and perhaps think about how many times code will call the reflection helper function), and then probably go eat a long lunch with the time I saved.

[steveharter]

@krwq

How would properties with [AllowNull] and [DisallowNull] be expressed? I feel like there is different nullability for setter and getter

I've updated the proposal (strawman2) to have separate "in" and "out" state, each with a separate enum. For a property getter, the "out" enum is appropriate. For a setter, the "in" enum is appropriate.

Also should the nullable custom attributes be just exposed through IEnumerable NullableAttributes (or similar)? That should be more future-proof rather than trying to express that with enum

Since the API is also intended to work with MetadataLoadContext, System.Attribute can't be returned since MLC doesn't support attributes since creating an attribute executes code (the constructor of the attribute). We could return CustomAttributeData but that would be hard to use.

[steveharter]

@jzabroski

I mostly just want to know if the properties can be nullable, non-nullable or unknowable.

Are your scenarios related to run-time, or design-build-compile-time? If there are real scenarios for run-time then we have to be concerned about perf, plus any tooling that may strip out these attributes for run-time optimizations of assembly size.

If we do not have run-time scenarios, I wonder if using Roslyn makes more sense. The more I look at these APIs, the more complexity I am finding. Also because the metadata produced by C# has changed over releases, that is also a factor in producing a reliable API.

It used to be simpler when there was just "?" which caused the compiler to add "[Nullable]". This applied to the whole property and affected both the getter and the setter. Now with the new nullability attributes there is more complexity including the ability to have the getter and setter have different nullability.

Here's an example of different ways to specify nullability on properties:

#nullable enable

using System;
using System.Diagnostics.CodeAnalysis;

namespace ConsoleApp87
{
    public class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");

            var obj = new MyClass();

            object? nullableValue;
            object nonNullableValue;

            nullableValue = obj.MyProperty0; // ok
            nonNullableValue = obj.MyProperty0; // warning CS8600: Converting null literal or possible null value to non-nullable type.
            obj.MyProperty0 = nullableValue; // ok
            obj.MyProperty0 = nonNullableValue; // ok
            obj.MyProperty0 = null; // ok
            obj.MyProperty0 = 1; // ok

            nullableValue = obj.MyProperty1; // ok
            nonNullableValue = obj.MyProperty1; // warning cs8600: converting null literal or possible null value to non-nullable type.
            obj.MyProperty1 = nullableValue; // warning cs8601: possible null reference assignment.
            obj.MyProperty1 = nonNullableValue; // warning cs8601: possible null reference assignment.
            obj.MyProperty1 = null; // warning cs8625: cannot convert null literal to non-nullable reference type.
            obj.MyProperty1 = 1; // ok

            nullableValue = obj.MyProperty2; // ok
            nonNullableValue = obj.MyProperty2; // ok
            obj.MyProperty2 = nullableValue; // ok
            obj.MyProperty2 = nonNullableValue; // ok
            obj.MyProperty2 = null; // ok
            obj.MyProperty2 = 1; // ok

            nullableValue = obj.MyProperty3; // ok
            nonNullableValue = obj.MyProperty3; // ok
            obj.MyProperty3 = nullableValue; // ok
            obj.MyProperty3 = nonNullableValue; // ok
            obj.MyProperty3 = null; // ok
            obj.MyProperty3 = 1; // ok

            nullableValue = obj.MyProperty4; // ok
            nonNullableValue = obj.MyProperty4; // ok
            obj.MyProperty4 = nullableValue; // ok
            obj.MyProperty4 = nonNullableValue; // ok
            obj.MyProperty4 = null; // warning CS8625: Cannot convert null literal to non-nullable reference type.
            obj.MyProperty4 = 1; // ok
        }
    }

    class MyClass
    {
        public object? MyProperty0 { get; set; } // property has [Nullable]
        [DisallowNull] public object? MyProperty1 { get; set; } // setter has [DisallowNull]; property has [Nullable]
        [AllowNull] public object MyProperty2 { get; set; } // setter has [AllowNull]
        [AllowNull, NotNull] public object MyProperty3 { get; set; } // getter has [NotNull]; setter has [AllowNull]
        public object MyProperty4 { get; set; } = 1; // no attributes
    }

[roji]

Are your scenarios related to run-time, or design-build-compile-time? If there are real scenarios for run-time then we have to be concerned about perf, plus any tooling that may strip out these attributes for run-time optimizations of assembly size.

There are definitely important runtime scenarios here, such as EF Core or other serialization frameworks needing to inspect CLR types for nullability.

The more I look at these APIs, the more complexity I am finding. Also because the metadata produced by C# has changed over releases, that is also a factor in producing a reliable API.

Indeed... That's also why such a (runtime) API is needed...

[MichalStrehovsky]

The more I look at these APIs, the more complexity I am finding. Also because the metadata produced by C# has changed over releases, that is also a factor in producing a reliable API.

Indeed... That's also why such a (runtime) API is needed...

Runtime doesn't version with the C# compiler though so if this is in flux, is placing this logic into the runtime really the right spot?

[roji]

I'm hoping that by now things have stabilized somewhat, no? If the encoding keeps shifting in meaningful ways, doesn't this break older compilers looking at newer assemblies?

[RicoSuter]

Just wanted to give you a heads-up that I already implemented a library to do that: https://github.com/RicoSuter/Namotion.Reflection

Still hope that .NET has a native library for this... but I couldn't wait for that :-)

And here I wrote an article about how this stuff works internally: https://blog.rsuter.com/the-output-of-nullable-reference-types-and-how-to-reflect-it/

I'm hoping that by now things have stabilized somewhat, no? If the encoding keeps shifting in meaningful ways, doesn't this break older compilers looking at newer assemblies?

From what I saw with the library and the test is that the output is quite stable - but hard to parse...

[chkn]

I created an API that handles nullability of generic type parameters, as well as reified nullable types like System.Nullable and F# Option and ValueOption: https://github.com/chkn/Xamarin.SwiftUI/blob/master/src/SwiftUI/Swift/Interop/Nullability.cs

It would be nice to have a reflection API, but it needs to at least handle generic parameter nullability.

[buyaa-n]

Based on the previous proposals I wrote another version of API proposal, pretty much tried to express the nullability info how we see in the source (nullable info and optional attributes) and i think this should be enough:

• I have no strong opinion about where the APIs should be added (Reflection or Reflection.Extensions), if we decide the APIs added in extensions the API will be changed to a static method.
• NullableInfo type will have main nullability info expressed with NullableState and optional AttributeInfo which will include the NullableAttribute enum and optional boolean or string[] Arguments
• Did not add nullable context info as it can be changed anywhere in the code with #nullable enable/disable, the NullableState will express that for the member
• Events do not support any nullability Attribute so it proposed GetNullableState()API which will return NullableState without AttributeInfo
• If we want to support generic type parameters we might need to add an API into Type type
• The extensions should work for both runtime-based and designtime-based reflection types (via MetadataLoadContext).
  • This means we can't return System.Attribute instances since MLC can't create attributes since that could execute code.

namespace System.Reflection
{
    public partial class FieldInfo
    {
+        public virtual NullableInfo GetNullableInfo() { return null!; }
    }

    public partial class PropertyInfo
    {
+        public virtual NullableInfo GetNullableInfo() { return null!; }
    }

    public partial class EventInfo // Does not support any attribute
    {
+        public virtual NullableState GetNullableState() { return NullableState.Unknown; }
    }

    public partial class MethodBase
    {
+        public virtual NullableInfo GetReturnTypeNullableInfo() { return null!; }
    }

    public partial class ParameterInfo
    {
+        public virtual NullableInfo GetNullableInfo() { return null!; }
    }

+    public sealed class NullableInfo
+    {
        // The value comes directly from NullableAttribute or nullable context
+        public NullableState Nullability { get; }

        // Optional attributes applied to the member: AllowNull, DisallowNull, NotNull, NotNullIfNotNull, NotNullWhen
        // DoesNotReturn, DoesNotReturnIf, MaybeNull, MaybeNullWhen, MemberNotNull, MemberNotNullWhen
+        public List<AttributeInfo>? Attributes { get; }
        // Or just return IList<CustomAttributeData>
+    }

     // EDIT: updated NotNull => NotNullable, MaybeNull => Nullable to not mistake with nullable attributes
+    public enum NullableState
+    {
+        Unknown, // value type or ref type when nullable disabled
+        NotNullable, // non nullable ref type, nullable context enabled
+        Nulllable // nullable ref or value type, nullable context enabled
+    }

+    public enum AttributeType
+    {
+        AllowNull,
+        DisallowNull,
+        DoesNotReturn,
+        DoesNotReturnIf,
+        MaybeNull,
+        MaybeNullWhen,
+        MemberNotNull,
+        MemberNotNullWhen,
+        NotNull,
+        NotNullIfNotNull,
+        NotNullWhen
+    }

+    public sealed class AttributeInfo
+    {
+        public AttributeInfo(AttributeType attributeType)
+        {
+            AttributeType = attributeType;
+        }
+        public AttributeType AttributeType{ get; }

          // or IList<CustomAttributeTypedArgument>? ConstructorArguments
+        public IEnumerable<object>? ConstructorArguments{ get; set;} // NotNullWhen(Boolean), MemberNotWhen(String[]), MemberNotNullWhen(Boolean, String[])
+    }
}

CC @steveharter @terrajobst

[steveharter]

@buyaa-n

Did not add nullable context info as it can be changed anywhere in the code with #nullable enable/disable, the NullableState will express that for the member

Does NullableState.Unknown have anything to do whether or not there is a nullable context, or does it mean a conditional like MemberNotNullWhen is being used, or both?

Hmm there must be a way to determine nullability since Roslyn does it. I believe the attribute can exist at assembly level + base class + type + member. I do think if we can correctly detect nullability and return the correct values for members with a non-nullable context, that will make this API more useful since consumers can ask "can this be null" without having to understand nullable contexts. If we don't provide it, consumers may need to write code to answer that themselves by determining if the type is a reference type vs. value type and also whether it is the special Nullable<T>).

Is the NullableState bitwise flags? If not, can the consumer differentiate between in and out modifiers on a method parameter, for example, that has both in and out nullable modifiers?

public enum NullableState { Unknown, // value type or ref type when nullable disabled NotNull, // non nullable ref type, nullable context enabled MaybeNull // nullable ref or value type, nullable context enabled }

In order to prevent name collisions and possible confusion with the in attributes of the same name I suggest: Unknown, Nullable, NotNullable.

[buyaa-n]

In order to prevent name collisions and possible confusion with the in attributes of the same name I suggest: Unknown, Nullable, NotNullable.

Updated enum values NotNull => NotNullable and MaybeNull => Nullable of the NullableState enum, thanks

Does NullableState.Unknown have anything to do whether or not there is a nullable context, or does it mean a conditional like MemberNotNullWhen is being used or both?

Yes, it can express whether or not there is a nullable context for ref type member, if this state found for ref type it means nullable is disabled (or nullable context is not set). If any other state (NotNullable, Nullable) is set for ref type that means nullable is enabled (or nullable context is set). Example:

#nullable enable
public class MyClass
{
    public string MyProperty { get; set; } // NullableState.NotNullable

    // ...

#nullable disable
    public string AnotherProperty { get; set; } // NullableState.Unknown

    // ...

#nullable enable
    public string? NullableProperty { get; set; } // NullableState.Nullable 

    // ...
}

Hmm there must be a way to determine nullability since Roslyn does it. I believe the attribute can exist at assembly level + base class + type + member. I do think if we can correctly detect nullability and return the correct values for members with a non-nullable context, that will make this API more useful since consumers can ask "can this be null" without having to understand nullable contexts. If we don't provide it, consumers may need to write code to answer that themselves by determining if the type is a reference type vs. value type and also whether it is the special Nullable).

Yes, there is a way to determine the nullability context, i didn't mean we cannot determine nullability context, i meant we don't need a separate field to express the context, the NullableState could give that info, also consumers cannot rely on nullable context because it can be changed anywhere within the type/code as shown in the above example.

If we don't provide it, consumers may need to write code to answer that themselves by determining if the type is a reference type vs. value type and also whether it is the special Nullable

Not sure if i understood your point, but i think they would need to check the type of the member for that as nullable value type does not depend on the nullability context

[steveharter]

If we don't provide it, consumers may need to write code to answer that themselves by determining if the type is a reference type vs. value type and also whether it is the special Nullable

Not sure if i understood your point, but i think they would need to check the type of the member for that as nullable value type does not depend on the nullability context

If there is no nullability context, consumers may still want to know whether a given parameter etc can be null, so if this API can return the correct value in that case I think that would be beneficial as it would avoid having to write that code manually each time. Here's the code I had in for this in my previous prototype.

Also on having AttributeType being "bitwise" here's a couple examples that have both in and out modifiers\attributes:

static bool MyMethod([NotNullWhen(false)] ref object? ret); // Has both NotNullWhenAttribute (out) and "object&" (in)
[AllowNull, NotNull] public object MyProperty { get; set; } // getter has [NotNull]; setter has [AllowNull]

[buyaa-n]

If there is no nullability context, consumers may still want to know whether a given parameter etc can be null

If there is no nullability context we can't add nullable annotation (i.e ?), and I don't think the compiler would add any nullability info, i mean there is no way to say if a given parameter can be null or not in case the nullability context is not set/enabled. In that case, any parameter or member will have NullableState.Unknown

[jeffhandley]

I'm wondering if we should have EventInfo also contain NullableInfo even though it can't have attributes.

1. Is there a possibility they could gain attributes in the future?
2. It would keep the APIs consistent, reducing one-off logic required for consuming these infos
3. We could just have the Attributes be null or empty

Is there any downside to doing that?

Should we name AttributeType and AttributeInfo something with "Nullable" in the name, like NullableAttributeType and NullableAttributeInfo?

[buyaa-n]

Also on having AttributeType being "bitwise" here's a couple examples that have both in and out modifiers\attributes:

By this desing there will be no separate nulablility info for inandoutmodifiers\attributes, because we will return nullability info withNullableState` and list of AttributeInfo if there is any attribute available, i don't think the API should process the attributes to give more info

static bool MyMethod([NotNullWhen(false)] ref object? ret); // Has both NotNullWhenAttribute (out) and "object&" (in)
// in this case the NullableInfo of the `ret` parameter will look like:
NullableInfo = {
    Nullability = NullableState.Nullable,
    Attributes  = [ { 
        AttributeType = AttributeType.NotNullWhen,
        ConstructorArguments = [
            false
        ]
    }]
}

[AllowNull, NotNull] public object MyProperty { get; set; } // getter has [NotNull]; setter has [AllowNull]
/// and in this case the NullableInfo of the `MyProperty` property will have:
NullableInfo = {
    Nullability = NullableState.NotNullable,
    Attributes = [ 
    { AttributeType = AttributeType.AllowNull },
    { AttributeType = AttributeType.NotNull }
    ]
}
// in this case NotNull attribute has no effect with nullability as the main state is already not null, 
// but as the compiler allows these attributes added together we will return them as is

[jkotas]

I have no strong opinion about where the APIs should be added (Reflection or Reflection.Extensions), if we decide the APIs added in extensions the API will be changed to a static method.

I would strongly prefer that this API is separate type, not tighly coupled with reflection. Something like:

public class NullabilityParser
{
    public NullabilityParser();

    // MemberInfo covers MethodBase, FieldInfo, PropertyInfo, EventInfo, TypeInfo, System.Type.
    // We can have specialized overloads too or insted if we wish.
    public NullableInfo GetNullableInfo(MemberInfo member); 

    public NullableInfo GetNullableInfo(ParameterInfo member);
}

The efficient parsing of the nullability information will require caching of a bunch of state. The separate type will make this caching easy to control and pay-for-play.

[buyaa-n]

If we decide to use a separate type as @jkotas suggested the type can be added in System.Diagnostics.CodeAnalysis namespace. In this case, we can populate the original nullability attribute and use collection of the Attribute type as we do not need to restricted by MetadataLoadContext support. NullableInfo type would look like this:

    public sealed class NullableInfo
    {
        // The value comes directly from NullableAttribute or nullable context
        public NullableState NullableState { get; }

        // Optional attributes applied to the member: can be instances of AllowNull, DisallowNull, NotNull, NotNullIfNotNull, NotNullWhen
        // DoesNotReturn, DoesNotReturnIf, MaybeNull, MaybeNullWhen, MemberNotNull, MemberNotNullWhen
        public IEnumerable<Attribute>? Attributes { get; }
    }

A sample usage could look like:

   private void DeserializePropertyValue(PropertyInfo property, object instance, object? value, NullabilityParser parser)
   {
        if (value == null)
        {
            NullableInfo nullabeInfo = parser.GetNullableInfo(property);
            bool disAllowNull = nullabeInfo.NullableState == NullableState.NotNullable; // assuming null allowed if NullableState.Unknown (nullable not enabled)

            if (nullabeInfo.Attributes != null)
            {
                if (nullabeInfo.NullableState == NullableState.Nulllable)
                {
                    disAllowNull = nullabeInfo.Attributes.FirstOrDefault(a => a is DisallowNullAttribute) != null;
                }
                else if (nullabeInfo.NullableState == NullableState.NotNullable)
                {
                    disAllowNull = nullabeInfo.Attributes.FirstOrDefault(a => a is AllowNullAttribute) == null;
                }
            }

            if (disAllowNull)
                throw new Exception($"Property '{property.GetType().Name}.{property.Name}'' cannot be set to null.");
        }

        property.SetValue(instance, value);
    }

[jkotas]

Optional attributes applied to the member: can be instances of AllowNull, DisallowNull, NotNull, NotNullIfNotNull, NotNullWhen

How is the API going to work for local definitions of attributes in netstandard2.0 and similar builds (see https://github.com/dotnet/runtime/blob/7ee226d0a634f1491f4050a798015a6587f7076d/src/libraries/Directory.Build.targets#L223)? Is it going to remap the locally defined attribute types to the public canonical ones in CoreLib?

[jkotas]

restricted by MetadataLoadContext support

It would be nice if the parser works on System.Type instances created by MetadataLoadContext too.

[buyaa-n]

It would be nice if the parser works on System.Type instances created by MetadataLoadContext too.

it would work, i meant the resulting nullability info does not need to restricted by MetadataLoadContext limitation

How is the API going to work for local definitions of attributes in netstandard2.0 and similar builds (see https://github.com/dotnet/runtime/blob/7ee226d0a634f1491f4050a798015a6587f7076d/src/libraries/Directory.Build.targets#L223)? Is it going to remap the locally defined attribute types to the public canonical ones in CoreLib?

I have no plan for that on top of my head, or might just use the existing CustomAttrbuteData instead of recreating each attribute:

    public sealed class NullableInfo
    {
        // The value comes directly from NullableAttribute or nullable context
        public NullableState NullableState { get; }

        // Optional attributes applied to the member: representing the AllowNull, DisallowNull, NotNull, NotNullIfNotNull, NotNullWhen
        // DoesNotReturn, DoesNotReturnIf, MaybeNull, MaybeNullWhen, MemberNotNull, MemberNotNullWhen attributes
        public IEnumerable<CustomAttrbuteData>? Attributes { get; }
    }

A sample usage could look like:

   private void DeserializePropertyValue(PropertyInfo property, object instance, object? value, NullabilityParser parser)
   {
        if (value == null)
        {
            NullableInfo nullabeInfo = parser.GetNullableInfo(property);
            bool disAllowNull = nullabeInfo.NullableState == NullableState.NotNullable; // assuming null allowed if NullableState.Unknown (nullable not enabled)

            if (nullabeInfo.Attributes != null)
            {
                if (nullabeInfo.NullableState == NullableState.Nulllable)
                {
                    disAllowNull = nullabeInfo.Attributes.FirstOrDefault(a =>  a.AttributeType.Equals(typeof(DisallowNullAttribute))) != null;
                }
                else if (nullabeInfo.NullableState == NullableState.NotNullable)
                {
                    disAllowNull = nullabeInfo.Attributes.FirstOrDefault(a =>  a.AttributeType.Equals(typeof(AllowNullAttribute)) ) == null;
                }
            }

            if (disAllowNull)
                throw new Exception($"Property '{property.GetType().Name}.{property.Name}'' cannot be set to null.");
        }

        property.SetValue(instance, value);
    }

[jkotas]

The implementation of IEnumerable<...> Attributes property is going to take the full list of attributes on the underlying item and apply filter to it. Is there enough value in this filtering? Would it be better to just tell people to get the attributes on the underlying item if they need the details?

[buyaa-n]

The implementation of IEnumerable<...> Attributes property is going to take the full list of attributes on the underlying item and apply filter to it. Is there enough value in this filtering? Would it be better to just tell people to get the attributes on the underlying item if they need the details?

The attributes can be found in different places like for PropertyInfo in its SetMethod.Parameter or GetMethod.ReturnParameter, so i thought except filtering it would be handy to have them available at the same place with NullabilityState. But after looking into it again it might better let the customers find them themselves, or even better if we update the nullability of the ParameterInfo returned SetMethod.Parameter, GetMethod.ReturnParameter based on them so that customers will not need to handle the attributes. Overall you are right we don't need to keep attributes info with the nullability info

[terrajobst]

Here is my proposal that addresses the ability to cache as well as retreiving nullability information for the top level as well as for array elements and generic parameters. If you want to play with it, the prototype with these APIs is in a single file.

Usage

Getting top-level nullability information

private NullabilityInfoContext _nullabilityContext = new NullabilityInfoContext();

private void DeserializePropertyValue(PropertyInfo p, object instance, object? value)
{
    if (value == null)
    {
        var nullabilityInfo = _nullabilityContext.Create(p);
        var allowsNull = nullabilityInfo.State != NullableState.NotNull;
        if (!allowsNull)
            throw new MySerializerException($"Property '{p.GetType().Name}.{p.Name}'' cannot be set to null.");
    }

    p.SetValue(instance, value);
}

Getting nested nullability information

class Data
{
    public string?[] ArrayField;
    public (string?, object) TupleField;
}
private void Print()
{
    Type type = typeof(Data);
    FieldInfo arrayField = type.GetField("ArrayField");
    FieldInfo tupleField = type.GetField("TupleField");

    NullabilityInfoContext context = new ();

    NullabilityInfo arrayInfo = context.Create(arrayField);
    Console.WriteLine(arrayInfo.State);         // NotNull
    Console.WriteLine(arrayInfo.Element.State); // MayBeNull

    NullabilityInfo tupleInfo = context.Create(tupleField);
    Console.WriteLine(tupleInfo.State);                        // NotNull
    Console.WriteLine(tupleInfo.GenericTypeArgument[0].State); // MayBeNull
    Console.WriteLine(tupleInfo.GenericTypeArgument[1].State); // NotNull
}

Proposed API

namespace System.Reflection
{
    public sealed class NullabilityInfoContext
    {
        public NullabilityInfo Create(ParameterInfo parameterInfo);
        public NullabilityInfo Create(PropertyInfo propertyInfo);
        public NullabilityInfo Create(EventInfo eventInfo);
        public NullabilityInfo Create(FieldInfo parameterInfo);
    }

    public sealed class NullabilityInfo
    {
        public Type Type { get; }
        public NullableState State { get; }
        public NullabilityInfo? Element { get; }
        public NullabilityInfo[]? GenericTypeArguments { get; }
    }

    public enum NullableState
    {
        Unknown,
        NotNull,
        MaybeNull
    }
}

[buyaa-n]

For the usage example I would pass the property's SetMethod.Parameter to account the nullabiity attributes applied to the setter

private NullabilityInfoContext _nullabilityContext = new NullabilityInfoContext();

private void DeserializePropertyValue(PropertyInfo p, object instance, object? value)
{
    if (value == null)
    {
        var nullabilityInfo = _nullabilityContext.Create(p.SetMethod.GetParameters()[0]);
        var allowsNull = nullabilityInfo.State != NullableState.NotNull;
        if (!allowsNull)
            throw new MySerializerException($"Property '{p.GetType().Name}.{p.Name}'' cannot be set to null.");
    }

    p.SetValue(instance, value);
}

[terrajobst]

For the usage example I would pass the property's SetMethod.Parameter to account the nullabiity attributes applied to the setter

This makes no difference because the C# compiler emits the correct nullable attributes for both the property as well as for the accessors (albeit using different encodings). However, for the user it's a bit more sensible to ask for the nullability state of the property, so I'd keep that ability.

For example,

class Person
{
    public string? Middle { get; set; }
    public string First { get; set; }
    public string Last { get; set; }
}

becomes

[System.Runtime.CompilerServices.NullableContext(1)]
[System.Runtime.CompilerServices.Nullable(0)]
class Person
{
    [System.Runtime.CompilerServices.Nullable(2)]
    [field: System.Runtime.CompilerServices.Nullable(2)]
    public string Middle
    {
        [System.Runtime.CompilerServices.NullableContext(2)]
        get;
        [System.Runtime.CompilerServices.NullableContext(2)]
        set;
    }

    public string First { get; set; }

    public string Last { get; set; }
}

[safern]

Should we include Oblivious to NullableState?

[terrajobst]

Should we include Oblivious to NullableState?

That's called Unknown in the enum. We could, however, decide to use the Roslyn naming, which is:

public enum NullableAnnotation
{
    None,
    NotAnnotated,
    Annotated,
}

Personally, I don't find those particular descriptive.

[jnm2]

The C# docs use these terms (https://docs.microsoft.com/en-us/dotnet/csharp/nullable-references#nullability-of-types):

Any reference type can have one of four nullabilities, which describes when warnings are generated:

• Nonnullable: Null can't be assigned to variables of this type. Variables of this type don't need to be null-checked before dereferencing.
• Nullable: Null can be assigned to variables of this type. Dereferencing variables of this type without first checking for null causes a warning.
• Oblivious: Oblivious is the pre-C# 8.0 state. Variables of this type can be dereferenced or assigned without warnings.
• Unknown: Unknown is generally for type parameters where constraints don't tell the compiler that the type must be nullable or nonnullable.

[safern]

The C# docs use these terms

Yeah that's why I suggested cause I went to the docs and I saw it includes both, Unknown and Oblivious.

[jaredpar]

@jnm2

Think that doc is still considering our C# 8 semantics and not taking into account the work we did in C# 9 with type parameters and null annotations. The NRT spec is the best place to go for the latest info (BTW edits to this are very welcome)

https://github.com/dotnet/csharplang/blob/main/proposals/csharp-9.0/nullable-reference-types-specification.md

From that though there is the following:

A given type can have one of three nullabilities: oblivious, nonnullable, and nullable.

The Roslyn API refers to these as None, Annotated and NotAnnotated respectively. To me at least it makes sense to use the same terminology in the runtime APIs.

@terrajobst

Personally, I don't find those particular descriptive.

To a compiler dev they are completely descriptive 😄

This is a more general API though where as Roslyn tends to be a very compiler centric API / terminology. So if these don't make sense to the more general audience it may make sense to use different terms.

[buyaa-n]

This makes no difference because the C# compiler emits the correct nullable attributes for both the property as well as for the accessors (albeit using different encodings)

@terrajobst i meant the nullability attributes used in the source, which are added into the property's getter/setter's CustomAttributes by the compiler. For example, if your example had the below attributes, the setter of Middle should not allow null

class Person
{
    [DisallowNull] public string? Middle { get; set; } // setter has DisallowNull
    [AllowNull] public string First { get; set; } // setter has AllowNull
    [MaybeNull] public string Last { get; set; } // getter has MaybeNull
}

[KPixel]

I would like to suggest renaming the enum NullableState to use DisallowNull instead of NotNull. "Disallow" makes more sense because a variable can still be null if we force it. And "NotNull" makes that seem impossible:

string s = null!;

And it would match with the sample above that reads:

        var nullabilityInfo = _nullabilityContext.Create(p);
        var allowsNull = nullabilityInfo.State != NullableState.NotNull;
        if (!allowsNull)
            throw new MySerializerException($"Property '{p.GetType().Name}.{p.Name}'' cannot be set to null.");

Actually, the exact state is ShouldNotBeNull, but that might be too verbose :)

Edit: For me, Annotated and NotAnnotated are the least intuitive options because they make me think that #nullable is enabled or disabled. (They do make sense within the compiler because they are tracking the presence of the "?" token.)

[stephentoub]

A given type can have one of three nullabilities: oblivious, nonnullable, and nullable.

The Roslyn API refers to these as None, Annotated and NotAnnotated respectively. To me at least it makes sense to use the same terminology in the runtime APIs.

FWIW, I find that confusing. If I see string?, I think of that as being annotated (it has a '?' annotation), and if I see string, I think of that as being not annotated (there's no ? annotation), so it's confusing to me that annotated == non-nullable and not annotated == nullable.

[stephentoub]

A given type can have one of three nullabilities: oblivious, nonnullable, and nullable.

The Roslyn API refers to these as None, Annotated and NotAnnotated respectively

Just read the XML comments in the source; I think the earlier comment actually reversed the order, and annotated actually is nullable and not annotated actually is non-nullable, which does make sense :)

https://github.com/dotnet/roslyn/blob/5a012a12b6885373734c508d3ec811a6c9a02ef8/src/Compilers/Core/Portable/Symbols/NullableAnnotation.cs#L30-L37

[stephentoub]

That said, I also prefer the terminology used in the docs: oblivious, non-nullable, and nullable, as it maps to how we've told developers (outside of the compiler team) to think about these concepts.

[bartonjs]

Video

API Review notes:

• Consider splitting State into the "read" and "write" states, to allow for straightforward expression of public string? Foo { get; [DisallowNull] set; }
• Does NullableState need something like "ItsComplicated" for [MaybeNullWhen] and other state-based things?
• Should there be a method to get the nullability related custom attributes associated with the context?
• NullabilityInfo.GenericTypeArguments returns an array, so it's either cloning or is mutable. Consider ReadOnlyCollection, or making it a method

[RikkiGibson]

Consider splitting State into the "read" and "write" states,

This is probably the right thing to do. The scenario I had in mind was void M<T>(T t). When you read from t, we have to assume that it could be null, since T could be a nullable reference type, but when you write to t, we have to assume that null is disallowed, because T could be a non-nullable reference type.

[GrabYourPitchforks]

A reminder: last I checked, the mono linker removes nullability annotations when trimming assemblies. Since the functionality proposed here operates over runtime assemblies rather than reference assemblies, we'll need to figure out how to reconcile this. I don't know if there are any tracking issues for this on the mono side.

[vitek-karas]

The linker will warn if there's a typeof(NullableAttribute) in the code needed by the app (I assume this code here would do that). It's basically to guard against the cases where it wants to remove an attribute which the app might need. A way to "fix" this is that the code would declare an explicit dependency on the NullableAttribute. It definitely works via descriptor.xml but that's not the right tool here. I think it should work through DynamicDependency (and if not we can make it so).

The problem will be that these attributes can be redefined in multiple assemblies.

[jnm2]

The problem will be that these attributes can be redefined in multiple assemblies.

And some of us are doing so with abandon. 😊 (DisallowNullAttribute, etc)