Open MadsTorgersen opened 7 years ago
@Joe4evr
it still won't be the silver bullet until it can be enforced at runtime.
Maybe, but it doesn't have to be the silver bullet in order to be immensely useful. It's like non-nullable types in TypeScript or generics in Java. Both are very easily tricked since the underlying type systems of both languages are incapable of enforcing them, but they still go a very long way in helping developers do the right thing.
@HaloFour I may be missing something, but how could it work for locals? I mean you can set attribute for a method/type/etc, but locals doesn't support them. Or we lose this information after converting to runtime?
@Joe4evr
it still won't be the silver bullet until it can be enforced at runtime.
It can be enforced virtually everywhere. Compile time? Check, you have it. If you set warnings as errors
you just cannot have this situation for known types until you do explicit cast. Want get an exception on explicit call? Use tools like Fody/PostSharp/Custom-roslyn-tool that generate these checks for you. Want to know if return type is nullable or not? No problem, reflection may be extended a bit to read these attributes and treat them accordinly. C# does some dirty hacks for some special types or conditions, so if you didn't have any problems with decimal constants or other amazing hacks then I don't see problem in adding another one. The only problem I see is watch types of locals, but it's based on response on first part of my comment. Anyway, it's a very rare usecase - determine if local variable in method is nullable reference type or not - so it's virtually a sivler buller as I see it.
@Pzixel
I may be missing something, but how could it work for locals? I mean you can set attribute for a method/type/etc, but locals doesn't support them. Or we lose this information after converting to runtime?
That information is lost at runtime. But since locals are confined to their method and the compiler understands how the locals are assigned and used it's not necessary to persist it anywhere. The attributes exist to inform the compiler what to expect at the boundary between different assemblies.
@HaloFour I mean that it looks just like Java's type erasure: we can't inspect method body via reflection to determine if this variable can be null and this one - can't. I don't say it's an important use case, but it still exist.
@Pzixel
It's much more isolated that Java's type erasure which eliminates the type information in nearly all cases including at the boundary. Given an instance of java.collections.ArrayList
it's impossible via reflection to determine what the generic type argument is, if any, and the add
method always only takes java.lang.Object
. Conceptually similar, but the scope is significantly reduced. Tools like Reflector might not be able to discern if a local was originally declared as nullable or not and that might impact some specific scenarios around IL rewriting. But those are pretty narrow concerns and those tools should be able to determine type based on usage, which is more important anyway as the compiler will consider a string?
to be a string
if flow analysis determines that it wouldn't be null
at a given point within a method.
Want get an exception on explicit call? Use tools like Fody/PostSharp/Custom-roslyn-tool that generate these checks for you.
You wouldn't even need those tools because Roslyn itself is gonna have that functionality (at least mentioned that it's considered). See the design doc#Tweaks.
Want to know if return type is nullable or not? No problem, reflection may be extended a bit to read these attributes and treat them accordinly.
Why need reflection extended? They're just normal Attributes.
You wouldn't even need those tools because Roslyn itself is gonna have that functionality (at least mentioned that it's considered). See the design doc#Tweaks.
Think about costs of implementing it.
Why need reflection extended? They're just normal Attributes.
Because currently GetType doesn't return information about if type is nullable or not, like IsClass, IsValueType and so on.
@HaloFour that's what I talking about, yes.
I don't say it's an important use case, but it still exist.
Because currently GetType doesn't return information about if type is nullable or not
if (memberInfo.GetCustomAttribute<CanBeNullAttribute>() != null)
Done.
@Joe4evr typeof(string)
and typeof(string?)
both return System.String
which either has the attribute, or does not. There is no non-nullable-string CLR type under this proposal.
On the other hand, I'd expect string? Foo()
(or string Foo()
) under this proposal to have a new attribute in MethodInfo.ReturnTypeCustomAttributes
.
It would be part of called on the MemberInfo types, obviously.
@Joe4evr I mean that it have to be a property, not an unnamed attribute. Of course you can read it from attributes.
public static class NullableReferenceExtensions {
private const string ATTRIBUTE_NAME = "System.Runtime.CompilerServices.CanBeNullAttribute";
public static bool CanReturnNull(this MethodInfo method) {
switch (method.ReturnParameter) {
case ParameterInfo parameter: return parameter.CanBeNull();
default: return false;
}
}
public static bool CanBeNull(this ParameterInfo parameter) {
return CanBeNull(parameter.GetCustomAttributes(false));
}
public static bool CanBeNull(this FieldInfo field) {
return CanBeNull(field.GetCustomAttributes(false));
}
public static bool CanBeNull(this PropertyInfo property) {
return CanBeNull(property.GetCustomAttributes(false));
}
private static bool CanBeNull(object[] attributes) {
if (attributes == null || attributes.Length == 0) return false;
foreach (var attribute in attributes) {
if (attribute != null && string.Equals(attribute.GetType().FullName, ATTRIBUTE_NAME, StringComparison.Ordinal)) {
return true;
}
}
return false;
}
}
There's more, but you get the gist. That's likely all any property/method added to the BCL would do. I'm going with the assumption that the compiler will resolve the attribute not by a specific type but by name, which is pretty common for it to do.
@HaloFour Those methods for PropertyInfo
, FieldInfo
, and MethodInfo
can be collapsed into one for MemberInfo
. Why the object[]
, though?
@Joe4evr
Those methods for
PropertyInfo
,FieldInfo
, andMethodInfo
can be collapsed into one forMemberInfo
.
Good catch. I was just throwing that together and didn't bother to look up whether or not the method was inherited from a base class.
Why the
object[]
, though?
That's a helper method to deal with the array of attributes returned by GetCustomAttributes(bool)
. The return value of that method is object[]
.
Either way this is all just illustrative.
Reposting here since it's more relevant:
In type inference, if a contributing type is a nullable reference type, the resulting type should be nullable. In other words, nullness is propagated.
Hm, even if the compiler always treated the reference type inferred as nullable the compiler should still be able to determine it's actual nullness based on flow analysis. So it makes sense for var
to always be nullable when it comes to reference types. If you assign it the result of a non-null expression the compiler would treat that as non-null until the point that the variable is assigned something else.
string ReturnsNonNullable() => "Hello";
string? ReturnsNullable() => "World";
var s = ReturnsNonNull(); // is string? but is definitely not null
if (s.Length > 0) { ... } // no warning, s is not null here
s = ReturnsNullable(); // legal, s is nullable
if (s.Length > 0) { ... } // warning, s might be null here
s = null; // legal, s is nullable
I guess it depends on if it's considered more important to guard the value of s
at the time of assignment v. at the time of consumption.
@HaloFour why var
should be always nullable? I'd rather prefer
string ReturnsNonNullable() => "Hello";
string? ReturnsNullable() => "World";
var s = ReturnsNonNull(); // is string
if (s.Length > 0) { ... } // no warning
s = ReturnsNullable(); // illegal, s is not nullable
string? s1 = ReturnsNonNull();
s1 = ReturnsNullable(); // legal, s is nullable;
var s2 = s; // not-null
var s3 = s1; // null
@Pzixel
The type of the local is always string
, it's nullability depends on how flow analysis determines its use. For example:
string ReturnsNonNullable() => "Hello";
string? s = ReturnsNonNullable();
if (s.Length > 0) { ... } // no warning, flow analysis has determined that s won't be null here
As such, in my opinion, the inferred type of the local is a lot less important than the eventual consumption of the local. It obviates the need for having to explicitly type s
as a nullable reference and for the need of any proposed syntax that would be both inferred and explicitly nullable, e.g. var?
.
I'm not particularly invested in either outcome, I'm just interested in having the conversation and seeing what the LDM thinks about it if anything.
Unlike value type nullables, this proposal does not propose for a separate type for nullable reference type. As long as a separate nullable reference type is not available, var?
is not possible as goes my understanding.
@HaloFour in this case I don't see why we even need this new syntax in locals if they just do nothing. And if we introcude them, then we don't have to have this flow analysis. Just check that variable of type non-nullable
is always assigned, show errors when nullable variable is dereferenced without explicit check on null and that's all. Compiler checks that all checks are passing and just compiles the code as always, without persisting information about what types was originally nullable.
On the other hand, we can have a flow analysis, but in this case I don't see much sense in ?
syntax for locals.
For var
, there are two choices: infer the narrowest type you can (which means non-nullable unless the return type of the function is explicitly nullable), or always infer nullable.
If you choose the first option, you break backwards compatibility in some cases:
var s = foo(); // Assume foo() returns non-nullable
...
s = null; // This was legal before, but now will be an error?
We've been talking a lot about generating warnings when dereferencing nullable types, but I don't think there's been much discussion about assigning nulls to non-null references. Sure, you can just throw a warning, but this will result in error behavior later in the application, in code that should be perfectly null-safe.
The second option (always infer nullable) should simply not be chosen. It will result in lots of incorrect warnings, and if the compiler is going to throw lots of warnings at devs for things they know are correct, devs are more likely to just ignore ALL the warnings, and this entire endeavor is worthless.
@MikeyBurkman this proposal don't throw any erros in any situations so
Sure, you can just throw a warning
It's how this proposal works. Thus no backwards compatibility is broken.
@MikeyBurkman
The second option (always infer nullable) should simply not be chosen. It will result in lots of incorrect warnings, and if the compiler is going to throw lots of warnings at devs for things they know are correct, devs are more likely to just ignore ALL the warnings, and this entire endeavor is worthless.
The compiler wouldn't throw warnings all over the place. The nullability of a variable is not set for the duration of its scope. The proposal already states that if you assign a non-nullable expression to a nullable variable that the compiler will treat that variable as non-nullable up until the point that it is potentially reassigned.
By having var
considered nullable you do get the best of both worlds. The compiler will still treat the variable as non-nullable as long as you assign to it the result of a non-nullable expression, and the compiler will still allow you to assign it a nullable value later (including null
) and the nullability warnings would subsequently kick in.
What are the cons of treating nullable reference types T?
as a separate type Nullable<T>
? ?
is being newly introduced anyway. There will be no errors shown, only warnings to prevent breaks. Then why not use a new type Nullable<T>
?
T
is already nullable.
@gulshan The cons are 0) that would have to build on the premise that references are non-nullable by default (which they aren't), 1) the fact that Nullable<T>
already exists, and 2) as the team has already discussed many moons ago, using wrapper types would be detrimental with regards to interop with other types.
@Joe4evr Thanks for pointing to the discussion. It seems at that time there were three versions of nullable references, non-nullable with !
, nullable with ?
and usual types. And structs for both non-nullable and nullable references would make it quite impossible to manage for down level compilers. But now that the non-nullable variant is gone, using struct is quite feasible in my opinion. Then proposed and downlevel versions will be something like-
string? M(string s) { ... } // Proposed
Nullable<string> M(string s) { ... } // Appearance to downlevel compilers
Regarding conversion, with explicitly non-nullable T!
gone, only conversion needed is from T
to T?
. This is already being done with value types and should not be problematic for reference types.
Regarding other two points,
0) Reference types are nullable by default now. But isn't treating reference types as non-nullable by default is the purpose of this proposal?
1) Existence of Nullable<T>
for value types makes it a better candidate to reuse for reference types IMHO.
So, my point is, using structs can be reconsidered now as explicit non-nullable notation is out of table now.
@gulshan It's impossible to make a Nullable<T> where T : class
that will one hundred percent of the time not contain a null value.
And that's before you get into the area of backwards compatibility, particularly for the BCL.
@yaakov-h Sorry, I don't get the point. Would you please elaborate a bit more? With some example maybe.
What I am suggesting now is, BCL already have struct Nullable<T> where T : struct
and we can just remove the type constraint so that it becomes struct Nullable<T>
and covers the reference types also. Then Nullable<T>
will correspond to T?
regardless of being value or reference type. It contains nulls in a boolean property. And BCL does not have T?
for reference types yet. Why it would break backwards compatibility?
@gulshan then all BCL types has to return not-null values because you cannot change signature thus IStructuralEquatable.Equals(object, object)
is never allowed to accept nulls while it's not true and currently you can compare two nulls and get true
.
Again, if you have in BCL method string Foo(string)
and it accepts and returns null you cannot write Nullable<string> Foo(Nullable<string>)
because you change the signature making so it incompatible with previous versions.
@Pzixel I thinks instead of changing the current methods, overloads with nullable versions can be introduced when appropriate. In The current attributes based approach, overloads will not be possible I guess. And nullability will be just striped in previous compiler versions. With separate types, previous compiler versions will have some of the benefits of non-nullable types, without the syntactic sugar of T?
and auto conversions from T
to T?
/Nullable<T>
maybe.
Beside that you have to make an overload to virtually all methods in BCL, what happens if method just can't return a not-null value but its signature makes him to do it? Throw an exception where it never was thrown? Return default(T)? Crash the whole program?
Your idea just leads to a very weird design (all methods in framework will have a not-null counterpart), and requires huge amount of work without any profit. We don't need a separate type except some rare features like reflection on locals.
When null checks were introduced to typescript, it was thought almost all the method signatures of all the definition files have to be updated. But practically, the changes in the signatures were very few in comparison. Because most of the time, nullability is not intended and no new overload is needed. In some cases, nullabilty is intended and only those methods should be overloaded. Simply, with this feature enabled, new compiler will generates warnings while compiling BCL libraries for the cases where a nullable overload is needed.
@gulshan
BCL has to ship with lot of warnings (or just suppress these warnings) in order to be compatible with previous versions of compiler.
BCL knows nothing about warnings, it's up to compiler, you know. C# 8 compiler knows these attributes and show a warning if something is wrong. C# 7.0 ignore these attributes and show no warnings. It's a very smooth idea that works very well without doing weird things. It just like mentioned TypeScript - you won't see any type checks in generated .js file why then you expect that C# compiler should show information about nullability in IL?
Exactly.
I raised this earlier but for using modreq
in the method signature instead of a wrapper struct... but .NET just has too much baggage for runtime enforcement to be a workable solution.
@Pzixel I was wrong in the regard that BCL has to suppress the warnings and thus removed that part. It seems a clean solution is impossible without breaking. Even in current attribute based solution, to be backward compatible, either BCL has to make all references nullable or expect nulls and keep checking for nulls for non-nullable references.
I'm really not thrilled with this implementation of nullable references. I want a stronger type system, not additional compiler warnings. For example, from what I understand, this doesn't allow me to write an interface like this:
interface IYielder<T>
{
T? Yield()
}
This is like an IEnumerator<T>
, but only for non-nullable types and combines MoveNext()
and Current
. If you've got a null, it means that collection has run out!
I'd even be satisfied to be able to write something like this
interface IYielder<T!> { ... } // something like where T: non-nullable
and opt in to using non-nullable references, that seems more useful than just a couple of warnings...
I want a stronger type system
Tough luck. 😛
I'd even be satisfied to be able to write something like this
interface IYielder<T!> { ... }
And put T!
basically everywhere in your code, rather than working on the assumption that like 95~99% of the time, a variable/field/parameter/return value is not going to be null
? That option has already been discussed to death and that's where it should stay, IMO.
On 30 Aug. 2017 12:32 am, Victor Gavrish notifications@github.com wrote:I'm really not thrilled with this implementation of nullable references. I want a stronger type systems, not additional compiler warnings. For example, from what I understand, this doesn't allow me to write an interface like this:
interface IYielder
and opt in to using non-nullable references, that seems more useful than just a couple of warnings...
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.
While I completely appreciate the concern that some teams may not want to deal with an "extreme" enforcement of this functionality, I totally and completely support the "extreme" switch option on a project-by-project basis. The switch would have to be "off" for open source code during transition, while capable of being "on" for in-house code.
There have been numerous times in my career when I spent an intense day or two working my way through a large code base introducing some new construct which provided great benefit like this would provide. In the old days this would be slow due to lack of help from the development environment. With VS, applying such code-wide changes becomes significantly easier.
I am absolutely and totally in support of turning on "extreme" and spending an intense day or two working my way through any and all errors and cleaning them up. The resulting benefit is way beyond worth it.
The only exception is open source / third party code (libraries) do need their own separate switch since in many cases sending the updates back isn't an option, and making changes to third party code just creates a support headache related to new releases of the code. Pressuring the third party developers to update is the best option, but they have their own schedules and sometimes some code bases can be low priority for some supplier.
I'm going to assume we're talking about the compiler also recognizing ReferenceEquals(x,null) as a "null check". I'm pointing this out because none of the discussion seems to make reference to it.
We actually make extensive use of "ReferenceEquals(x,null)" rather than "x == null" since "==" can be overridden whereas "ReferenceEquals(x,null)" cannot. If our goal is to make sure there won't be a subsequent null-reference then we're really not interested in giving the class an opportunity to vote on the subject. It also avoids making the call to a "==" overload when the goal is to check for null and not object equality/inequality.
There seems to be a clear argument for the compiler to flag whether a given project (library, etc) (method?) is a participant adhering to non-nullable or not. As one example, it influences how the compiler could "trust" the extremely important case of a class overriding the "==" and "!=" operators. If the class is not flagged as a participant, then the compiler cannot trust its potential "==" and "!=" overloads to perform a proper null check.
However if the class (method?) has been flagged as a participant, then the compiler can trust the "==" and "!=" overloads since their normal signature would have changed from "nullable" to "non-nullable". Calling the overload for "x == null" (maybe?) becomes meaningless (actually invalid) since the arguments are non-nullable for a participating class. There would need to be alternate "operator ==(Test? x, Test? y)" overloads introduced.
Will there be a warning level (extreme?) which flags any resulting meaningless checks for null on a non-nullable variable? This will obviously result from treating existing "nullable signatures" variables as "non-nullable". The compiler could just no-op such checks -- although there is cases like "Test[] x= new Test[12];". Based on the compiler potentially just no-op'ing, maybe such checks would be treated as info by Roslyn similar to updating variable declarations to inline, etc.
With due recognition of the problems with things like "Test[] x= new Test[12];", the "ultimate" goal of all this should be providing a way to code such that the compiler no longer bothers to generate null checks on references provided "ultra-extreme" compiler checking is performed. If a variable is non-nullable and the developer does everything under the "extreme" restrictions then the goal should be for the compiler to perform all tests necessary such that no null reference checks are required on non-nullables at runtime -- except possibly when being assigned from a nullable if the compiler cannot adequately determine that the developer has ensured the value cannot be null.
Again, some things may stand in the way of this desired "ultimate goal" in the early stages, such as "Test[] x = new Test[12];". But the path being followed during the early implementations of this capability should head in the direction of trying to achieve that goal somehow in the future. (Approaches shouldn't accidentally interfere with that possibility.)
Being an old-hand embedded programmer from the machine/assembly/C days, I've noted "volatile" with C# but I'm unclear as to the extent it actually gets honored. A "volatile" variable throws a very big wrench into much of this stuff. It leads to the question of whether the constructs being proposed would make it impossible for a compiler to properly honor "volatile".
On subsequent consideration, perhaps the further below (which I wrote first) could be addressed by two new statics on Object alongside ReferenceEquals().
public static bool Assignable<T>(T? value, ref T target) where T : class
and
public static bool Assignable<T>(T? value, T @else, out T target) where T : class
The compiler would then inline them like ReferenceEquals().
======
I think some special consideration should be made regarding returned reference values. Conceptually a lot of methods only return "null" as meaning "false" (didn't find it, etc) and not "really" as meaning null to be a legitimate value. For example, List
So in nearly all (not literally all) cases the value returned by List
Conceptually this would be:
if (!(Nonnullable x = Nullable())) ObjectNotFound();
Obvious this specific approach isn't doable, so it's only meant to demonstrate the concept. It also raises the question of "what is x if the conditional is false?" If can't be "default" since x can't be null.
One approach would be x has to have been initialized previously and "false" would mean the assignment never occurred.
Alternately a variant of the concept "Nonnullable x = Nullable() ?? otherValue
" could return true/false instead of the value assigned to x. Something along the lines of "x ??= y : z
" which returns true/false rather than the value assigned to x. This construct would support inline variable declarations.
With the introduction of Nullable reference type there's a plan to statically check for initialization of the non-nullable types within the constructor. My question is: will the initialization of the non-nullable types be allowed also in a method to be called by the constructor, with a recursive static check? In other words, in the following example, will the initilization of _var1
be accepted in the fooInit()
method and not accepted for _var2
?
class Test
{
public void foo() { }
};
class Test2
{
Test _var1; // Non-nullable
Test _var2; // Non-nullable
public Test2(bool condition)
{
fooInit(condition);
}
void fooInit(bool condition)
{
_var1= new Test(); // OK, "_var1" initialized and not null
if (condition)
_var2= new Test(); // Warning/Error: "_var2" may be unitialized within the constructor
}
public void fooUse()
{
_var1.foo();
_var2.foo();
}
};
We actually make extensive use of "ReferenceEquals(x,null)" rather than "x == null" since "==" can be overridden whereas "ReferenceEquals(x,null)" cannot.
The feature currently recognizes the equivalent syntax x is null
.
then the goal should be for the compiler to perform all tests necessary such that no null reference checks are required on non-nullables at runtime
In many cases, the JIT omits null checks altogether. For example, when calling a virtual method on an object, the null check is not necessary because an access violation will occur in the processor. The runtime handles the processor exception and throws one of the .NET Exception
types instead (typically a NullReferenceException
).
... The compiler could just no-op such checks ...
... the goal should be for the compiler to perform all tests necessary such that no null reference checks are required on non-nullables at runtime ...
... question of whether the constructs being proposed would make it impossible for a compiler to properly honor "volatile" ...
Currently there are no changes to semantics or runtime behavior implemented (or even planned as far as I know) as part of this feature. For example, the feature could enable a developer to remove null checks in places, but the compiler will not remove them on its own.
So in nearly all (not literally all) cases the value returned by List.Find() is conceptually non-nullable, but with the caveat that null indicates "not found". Thus some consideration of a construct which deals with such a return value without needing to otherwise superfluously introduce a nullable variable just for the interim check would be worth considering.
The method would return a nullable type. Alternative approaches (TryFind
, FindIndex
) can be used if you need to distinguish this case.
It also raises the question of "what is x if the conditional is false?" If can't be "default" since x can't be null.
My question is: will the initialization of the non-nullable types be allowed also in a method to be called by the constructor, with a recursive static check?
@ceztko My understanding is the feature is designed to support intraprocedural analysis, meaning the analysis would consider the signature of fooInit
and its impact on the local variables of the constructor, but would not actually look at the contents of fooInit
while analyzing the constructor.
@ceztko I expect an error here, because it's just same thing as not all code paths return a value
.
the analysis would consider the signature of fooInit and its impact on the local variables of the constructor
@sharwell it's still not clear at what level non-nullability of member variables will be enforced in the opt-in case, with all non-nullability warnings treated as errors:
Strong enforcement: the use of this
in the constructor will be prevented until all non-nullable members are initialized with non-null values, similarly to what happens today for non-default struct
constructors, where all non-static member must be initialized before using this
. In this case a fooInit
method that is used in the constructor to initialize a non-nullable variable, either by returning value or out
parameter, must be necessarily static
to prevent use of non-nullable members before they are really initialized with non-null values. Example:
class Test
{
object _obj; // Non-nullable
public Test()
{
// _obj.ToString(); // ERROR: _obj is not yet initialized with non-null value
// foo(); // ERROR: "this" can't be used until all non-nullable members are initialized
init(out _obj); // OK
foo(); // OK
}
static void init(out object obj)
{
obj = new object();
}
public void foo()
{
_obj.ToString();
}
};
Weak enforcemenent: the constructor will ensure all non-nullable members are initialized with non-null values. Use of this
in the constructor is allowed but non-nullability can be violated at runtime in the scope of the constructor. Example:
class Test
{
object _obj; // Non-nullable
public Test()
{
init(out _obj); // Without this line, it wouldn't compile. It will fail at runtime
}
void init(out object obj)
{
_obj.ToString(); // This line will fail at runtime with null exception
obj = new object();
}
}
@Pzixel it's my understanding that non-nullable types behavior is an opt-in/opt-out choice. Opt-in: error, Opt-out: warning or no reporting at all (current behavior).
@ceztko That's incorrect. It's Opt-in: warning, Opt-out: no reporting at all (current behavior).
That's incorrect. It's Opt-in: warning, Opt-out: no reporting at all (current behavior).
Hmm...ah, ok! Well, this really a feature where I would consider "stronger" opt-in, like treating all the the non-nullability warnings as errors.
@ceztko All warnings can be treated as errors, warnings, or disabled. You can do this individually (per warning ID), or as a whole.
Note this includes the
object
constraint:where T : object
.LDM:
!
)!
, cast, impact of dereference on null-state, extension to nullable value types,class?
,MaybeNull
)?.
,!
on l-values, pure null tests)default
loophole)MaybeNull
)[DoesNotReturn]
)Task<T>
variance, cast)T??
)