Non-nullable type annotation syntax

[munificent]

This is the tracking bug for the real implementation of the postfix ? syntax to mark a type annotation as nullable as part of supporting non-nullable types (#28619). See the "Syntax" section of the proposal for details.

Since we're still only in the prototype phase, consider this bug to be on hold. If we decide to go ahead and ship a real implementation, I'll update this.

Flag

While this is being implemented, to ensure we don't expose users to inconsistency in our tools, it should be put behind a flag named nnbd.

Tracking issues

• [x] Analyzer (#27232)
• [ ] dart2js (#27232)
• [ ] dev_compiler(#27234)
• [ ] Formatter (https://github.com/dart-lang/dart_style/issues/536)
• [ ] Specification (#27235)
• [ ] VM (#27236)
• [ ] IntelliJ-plugin (#27237)

[crelier]

Just curious, why was '?' chosen? It seems very counter-intuitive to me, since a question mark implies some kind of optionality. In this case, we rather want to express a restriction. A more assertive exclamation mark would be more appropriate, in my opinion, but I did not study possible grammar conflicts, if that is the reason of this choice.

[lrhn]

The question mark means that the type is nullable/optionally null/union of type and Null, and its absence means that the type is just itself.

[donny-dont]

Any chance union types will come along with this? They seem to be natural fits.

[dynaxis]

This seems just an experiment for now. But I'd like to note that there might be needs for addition of APIs to the standard library. For instance, Kotlin has filterNotNull and mapNotNull on nullable sequence or collection, which return one with non-nullable element type. Dart's Stream and collections would be better to have such new APIs. They can definitely be implemented in user codes, but are really clumsy to do so.

[crelier]

@lrhn Oh, I see. The title is misleading and the text does not clarify. It should actually be "Nullable type annotation syntax" instead of "Non-nullable type annotation syntax".

[zoechi]

@tantumizer

"is-not-null" assertion operator.

This is shorter

return x ?? (throw new NullPointerException);

See also #24892

I find x!! a bit cryptic. If the return type for the function that contains this code is a non-nullable type it should throw in checked mode anyway and make the check and throw redundant.

[lrhn]

@tatumizer

Good questions, some not decided yet.

Optional parameters: If nothing else changes, you'll have to make them nullable by adding a ? to the type. After all, that is their type. We may consider allowing you to make them non-nullable if they have a non-null default value. If we do that, we should probably also change how default parameters work by making an explicit null argument be equivalent to not having the argument at all. There is some design-space here to explore.

For a nullable var, it's not really a problem as we are moving towards Strong Mode where var means to infer the type. You can write dynamic? but since null is already assignable to dynamic, I think dynamic? will just mean the same as dynamic (so we may disallow it entirely, there is no reason to have two ways to write the same thing). Same for Object and Null.

Too many question-marks. Likely true, but they are consistently about null (except for the original ?: conditional operator).

A not null assertion (that is: take nullable value, throw if it's null, otherwise we know the type). That could just be x as Foo if the type of x is Foo?. We will have to change as to not allow null for non-nullable types, you would then write x as Foo? to get the current behavior. We are also considering shorthands to quickly coerce a value to an expected type. Maybe x! will check for null if x has type Foo? and it's used in a position where a (non-nullable) Foo is needed. Maybe that's just being too clever by half.

[zoechi]

If you have

String foo = map["foo"];

then ?? throw new NullPointerException(); is redundant because String already is non-nullable and checked mode should throw.

[floitschG]

As a clarification: if we add only ? (meaning "this type is nullable"), then the non-? type must be non-nullable. This means that String foo = map["foo"] would statically not be allowed, unless we have implicit downcasts for nullable types, too. (It wouldn't be that awkward, since A is pretty much a subtype of A|Null).

If we don't allow downcast assignments, then there must be a way to go from nullable to non-nullable. We have the following choices:

• as, conditions and ifs promoting the type: String foo = map["foo"] ?? ... basically falls into that category. This requires no change to the language.
• a special operator: String foo = map["foo"]!! which checks for nullability. I guess this would also include !!. as in map["foo"]!!.bar(). As Lasse suggests, this could potentially be more general, coercing more than just nullable types. For example, it could replace the implicit downcast.

@tatumizer: I'm definitely interested in the reasons for why you had to use !! more often than you thought.

[eernstg]

Currently the nullability experiment is about syntax only.

In Patrice Chalin's proposal ( https://github.com/chalin/DEP-non-null/blob/master/doc/dep-non-null-AUTOGENERATED-DO-NOT-EDIT.md), nullability is a property of a type (T? is essentially a shorthand for T | Null: B.3.1), and Foo<C> would be the generic class Foo instantiated with an actual type argument C, which is a non-null type, assuming that C is the name of a class. The type argument of Foo<E> where E is a type variable of an enclosing generic entity such as a generic class Bar<E> .. could be a nullable type or a non-null type, depending on the instantiation of that generic class of which this is an instance. For instance, if we consider an instance of Bar<int?> and Bar contains an occurrence of Foo<E> then E is a nullable type, namely int?. If you want to make sure that a given type is non-null then you may or may not have an operator for it: For instance, Foo<E!> could be a Foo of non-null E, which would in this case be int (int? with the ? stripped off).

When a type variable can stand for a nullable as well as a non-null type it is necessary to be a little bit smarter in code generation, such that it will work in both cases, with good performance.

Because of complexities like this, there are quite a number of issues that we haven't decided on, so we can't promise anything specific about these design choices. But currently we won't even promise that there will be anything like a ! operator for stripping ? off of a given type, we are just looking for a syntax that will work well for ?.

On Tue, Sep 6, 2016 at 3:14 AM, Tatumizer notifications@github.com wrote:

Another question: consider generic class Foo. What is the meaning of E? Is it a nullable type? Or only non-nullable type? If the latter, we would need Foo<E?> to denote nullable E, but this idea is probably not tenable. So E is nullable. Suppose we want to say that for Foo, we accept generic type parameter E only if NOT nullable? How to write this restriction? Maybe Foo<E!!> :) Or what?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dart-lang/sdk/issues/27231#issuecomment-244828272, or mute the thread https://github.com/notifications/unsubscribe-auth/AJKXUhdiciCSV8iIfIi2pqtBdS5pIByZks5qnL5kgaJpZM4J0Ezv .

Erik Ernst - Google Danmark ApS Skt Petri Passage 5, 2 sal, 1165 København K, Denmark CVR no. 28866984

[munificent]

Replying to a few random things that weren't already covered:

Just curious, why was '?' chosen?

It's the same syntax used to represent nullable types in C#, Ceylon, Fantom, Kotlin, and Swift.

Any chance union types will come along with this?

We are interested in exploring union types too, but they're a big feature with a lot of consequences, so we aren't working on them right now. There's only so many hours in the day. :)

It should actually be "Nullable type annotation syntax" instead of "Non-nullable type annotation syntax".

I considered that, but the obvious response is that Dart 1.0 already has nullable type annotations—all type annotations are nullable. So this is really about adding a way to express non-nullable types. And the way we do that is by adding new syntax for nullable types and changing the existing syntax to mean non-nullable.

I find x!! a bit cryptic.

Me too, but something along these lines might be worth doing. A big part of why we are doing an experiment around non-nullable types is to get answers to usability questions like this. How often do users need to assert that they know something isn't null when the compiler doesn't? We're hoping to implement enough of the static checking to be able to answer that confidently.

String foo = map["foo"]!!;  // I'm sure it's not null, I've just assigned it!

Another part of this experiment is determining how we need to change our core libraries to make it pleasant to work with non-nullable code. In this case, I think Map should support two accessor methods. One returns V? and returns null if the key isn't present. The other returns V and throws if the key isn't found. In this case, you'd use the latter and wouldn't need !!.

There's some interesting API design questions about which of those operations should be [] versus a named method, which is more commonly used, etc. but we need to start trying things out to get a feel for that.

Opened dart sdk code at random place, and in 30 sec got a first example (https://github.com/dart-lang/sdk/blob/master/sdk/lib/collection/linked_list.dart)

That entire class was designed around the idea that E is nullable. Once that assumption is no longer true, there are probably systemic changes you could make to the entire class so that you don't need to sprinkle !! everywhere.

It's also probably true that core low-level collection classes like this will bear the brunt of the manual null checking. They are closer to the metal and need to do things a little more manually. Higher-level application code should hopefully be able to use non-nullable types more easily.

Another question: consider generic class Foo. What is the meaning of E?

I have an answer in mind for this, which I think lines up with Patrice's proposal, but I haven't verified that or written mine down in detail yet. (That's why this issue is about non-nullable syntax. :) ).

The short answer is that here, since you have no constraint, E can be instantiated with either a nullable or non-nullable type. If E was constrained to some non-nullable type, it could only be instantiated with a non-nullable type.

Suppose we want to say that in Foo, E must be NOT nullable. How to write this restriction? Maybe Foo<E!!>

If you want to say E is some specific type, you can give it a constraint and that implicitly constrains it to be non-nullable too, unless the constraint is nullable:

class Point<T extends num> {
  T x, y; // <-- These are non-nullable.
  Point(this.x, this.y);
}

new Point<int>(); // Fine.
new Point<int?>(); // Error! Constraint is non-nullable.

class Pointish<T extends num?> {
  T x, y; // <-- These may or may not be nullable.
  Pointish(this.x, this.y);
}

new Pointish<int>(); // Fine.
new Pointish<int?>(); // Also fine.

I don't currently plan to support a constraint that says, "The type must be non-nullable, but I don't care anything else about the type." I could be wrong, but it doesn't seem very useful to me.

[eernstg]

With the approach where T? stands for T | Null, there is no difference between Object and Object?. This is because the set of objects typeable as Object already includes the object typable as Null.

You could invent an operator computing a "type difference" (think set difference), say \, and then you could express "everything except null" as Object \ Null. You might even claim that the ! operator that we might introduce in order to "strip the ? off" of a given type variable could be defined to compute exactly that (you could say that it stands for the type-level function (Type T) => T \ Null). You could also consider one of the other interpretations of nullability in the first place. But, as mentioned, we're currently focusing on finding a good syntax, so it'll be a while before those other topics come up.

[munificent]

You want to change collection classes to only accept non-nullable type parameters?

No, the collections don't have constraints and should accept nullable type arguments.

My point was that the implementation of LinkedList takes for granted a type system where E is always nullable and non-nullability is not expressible. Once that assumption is no longer true, the implementer of LinkedList, might change how it's implemented. Maybe something like:

class LinkedList<E extends LinkedListEntry<E>>
    extends Iterable<E> {
  E? _first;

  E get first {
    if (_first != null) return _first as E;
    throw new StateError('No such element');
  }

  E get last {
    return first._previous;
  }

  // ...
}

And my claim was that the situation is typical, you need to write '!!' quite often.

Yeah, you may be right, though I'd like to see how it actually works out in practice in the context of Dart and it's core libraries.

[floitschG]

Currently, it would need to be int? index, since it's initialized with null.

There are other similar cases where the null initialization would be annoying:

int x;
if (someBool) {
  x = 499;
} else {
  x = 42;
}

If these cases are too common we will investigate definite-assignment strategies, like in Java, and see if they would fix that problem.

[munificent]

That is, declaration int index; is treated like int? index;

We could do that, but I think it's probably better to be explicit here at the expense of a little verbosity. I think it would be a bad thing if adding an initializer to a variable changed how its type annotation was interpreted.

Otherwise, with these question marks everywhere, program will look like a crossword puzzle or something.

The last time I poked around a corpus, I found that ~90% of variables were non-nullable, so I don't think the ? will be too common. But we won't know for sure until we try, which is what this experiment is about.

[zoechi]

@tatumizer I worked several years in a language that didn't have null (An ERP system - mostly a database frontend - this might be different for code closer to the metal) and I only missed null when I had to deal with outside code (COM components or similar, but there was some special value provided that was converted to null when passed to outside code).

I liked it a lot to not have to deal with null at all.

I think this is the main point of NNBD that null is rarely needed, otherwise we could keep the current NBD.

A way would be (I think it was mentioned somewhere above) to define default values.

int index;

could be treated like

int index = 0;

This would be similar to now where it is treated like

int index = null;

For String it could be the empty string and for classes an instance created by the default constructor or a named constructor MyClass.default(). But I think just now allowing non-nullable types without initialization would be fine as well.

[zoechi]

It seems currently dart is on its path to require fake nullables. In other words, programmer has to lie about the types, just to keep compiler happy.

Why do you think that?

int x;

I think if there is not a defined default value (like 0), this can and should not be allowed.

[DanTup]

I'm not sure why you would expect this code to work?

Foo x;
print(x);

The whole point of having a non-nullable value is to guarantee it has a value; you shouldn't expect to be able to declare one without a value and use it?

And if you take the print out of your example:

Foo x;
x = new Foo(10);

I'm pretty sure the compiler could figure this out; the value is always assigned before it is used and therefore is guaranteed to be valid. I agree that Dart shouldn't just pluck random values out of the air.

I think it's valid to want to be able to declare a non-nullable value and assign it later, but I don't think it's valid to want to use it before it has been. Eg., this code I think should be valid:

int x;
if (something) {
  x = 1;
} else {
  x = 2;
}

Although you could shorten this to int x = something ? 1 : 2; it feels like if the above isn't valid it could be frustrating. However, I would expect an error if you tried to user the value before it was guaranteed to be assigned and I can't think of any obvious ways that wall fall down. If you have a concrete example of where it would, I think it'd be very useful to post :-)

I think even things like try blocks wouldn't be difficult (relatively ;)) for the compiler to understand:

int index;
try {
  thing();
  index = 1;
  callFunc(index); // index is assigned here, this is fine
}
finally {
  callFunc(index); // index is not valid here, it isn't guaranteed to be assigned
}

[dynaxis]

In Dart, even local variables are considered to be null initialized by its semantics. But In the case of Java, local variables may be left uninitialized and the compiler ensures any use-before-assignment is flagged as illegal. On the other hand, Java's instance/static variables are all null or zero initialized by default. If I remember correctly, it was a decision to avoid unnecessary redundant assignments to local variables and wasn't related to nullability at all (Java has no nullability concept builtin).

For the record, I think such a decision to allow local variables to be left uninitialized is no longer useful for optimization point of view since modern compilers are smart enough to eliminate such unnecessary assignments to local variables.

Back to the main point, (I think) Kotlin followed the Java tradition to not decide on a default value used for non-nullable variables. In Kotlin, you may use lateinit modifier to leave even non-nullable instance variables left uninitialized. But in the case, the compiler inserts runtime checks to use sites to ensure the variables are actually initialized before uses.

I'm not sure which way is better. But I think having a default value doesn't defeat the whole point of having non-nullable type. In the end, we don't get in trouble in most cases by uninitialized (local) variables. If that was the problem, we should've been getting NPE all over the places while using Dart.

UPDATE: My original point was that requiring local variables to be initialized are not that much a problem since their scopes are quite narrow (Maybe I'm wrong here). For instance variables, I think it's better to require explicit initialization rather than a default value.

But thinking a little bit more, requiring an explicit initialization at declaration site seems better than a default value. It's less complex and less troublesome in many aspects, IMHO.

[dynaxis]

I've been writing Kotlin in production for the last 3~4 months. For the !! operator in Kotlin, I find myself avoiding uses of it as much as possible, since I know they entail runtime checks. So I think it's ok to have a little bit more verbose syntax. Overloading x as NonNullable seems good to me.

Also for Kotlin, I think its non-nullable type support is quite well designed and seems working for me at least for now. But we don't know if the design is single way of doing it correctly.

[lrhn]

void main() {
  int x;
  print("Hello"); // I'm printing Hello now, not x 
  x = 10;
  // proceed with x=10, it won't ever be null again
}

Is it a valid Dart program (in a variant of Dart with not-null-by-default nullable types)?

Probably not. The declaration int x; declares x as not nullable and doesn't initialize it to a value. In the simplest case, that alone is enough to reject the program. Bob was mentioning one optimization that could be allowed - use a definite assignment analysis which checks that all reads of the variable is guaranteed to occur after a write to the variable, and allow uninitialized nullable variables as long as they satisfy the analysis and are never read before they have been written to at least once. Initialization counts as a write, so initialized variables are always guaranteed to be read-after-write, but a program like the following can then also be allowed:

int x;
if (foobar()) {
  x = 42;
} else {
  x = 10;
}
print(x);  // use of x guaranteed to occur after write to x.

Your program fails to satisfy this - it reads the variable x before it's written, so it would still be invalid with that optimization.

You can think of "definitely assigned" analysis as a way to allow implicit initialization of non nullable variables with a suitable value. It's extremely confusing if we just do that in general and pick some default value, but if it is always assigned before being read, it doesn't matter which value we use since nobody will ever see it.

[lrhn]

main() {
   List<int> list = new List<int>(5);
   print (list[0]);
}

That is a problematic case, and we are aware of it. There are many different possible solutions:

• Have something like class List<E?> ... enforce that the type parameter is always converted to nullable, so all lists have nullable element types. Writing new List<int>() and new List<int?>() would mean the same thing (List<int?>).
• Don't have that, but List uses E? everywhere inside, so the effect is that elements are nullable (but then List<int> and List<int?> are different but equivalent, that's inconvenient).
• Allow different constructors to have different type requirements. Then new List<int>(5) is invalid because that constructor requires a nullable type, but List<int>.filled(5, 0) works.
• Add an optional second parameter of type T that is used for initialization: new List<int>(5, 0) works but new List<int>(5) doesn't because null isn't a valid value for that E. That solves this particular problem, but not any other similar problem. And it makes List.filled partly redundant.
• Something better that I can't think of right now.

It's definitely not trivial to make everything fit together just right and give a good developer experience. We are looking at what other languages do too, but not everything translates to Dart or to what we want Dart to be. That's why we need to try things out, to see if there are other annoying edge cases like the List constructor that we haven't realized yet.

[eernstg]

when we write "int x" .. Is null a valid value? .. But null is not any special among other values.

I agree that it is very useful to discuss the semantics conceptually in order to achieve meaningful and consistent technical definitions. But starting from the technical level we have two clear answers here: In current Dart, null is a valid value for x, and in a future Dart where nullability support has been added, it is not. Conceptually, I think both choices are meaningful, you just need to know which one is in effect, and you need to use that when reasoning about potential NPEs.

The value of null is indeed special, because the spec singles out that value, and also because it is used to model "does not exist", which is different from any particular regular value of, say, an int variable.

When we write "int? x" .. It might mean simply that for technical reasons .. we have to declare it this way

In that case I'd suggest that the developer refactors the code slightly, such that it is possible to use a technical device (the type int rather than int?) which reflects the intended "meaning" of the variable.

It may be a useful language enhancement to use definite assignment analysis, such that int x; can be allowed, given that x cannot possibly be evaluated before some non-null value has been assigned. It may also be a bad choice to do this, because it allows for code which is quite brittle ("I just changed blah-blah which can't possibly make a difference, so why doesn't it compile any more ... Ah maybe this method call can throw!", etc).

[lrhn]

The optional nullability is intended for Dart 2 which will also has strong mode-like type guarantees. That means that you can't assign a List<Object> to a variable expecting List<int>, nor can you cast it using as. The expression objList as List<int> must throw if objList is-not-a List<int>. A List<int?> is-not-a List<int> (it's the other way around due to Dart's covariant generics).

That means that the line int x = intList[0]; will only be reached in situations where intList contains an actual List<int> which also guarantees that its operator[] returns int and not null.

It is true that if your program is actually flawed, adding extra type checks will just make the program throw at a different and earlier point, possibly by completely rejecting the program. Your explicit down-cast in this example can't be caught by the compiler, you are doing everything correct in order to tell the compiler that you know what you are doing, but it will add a check at runtime to ensure that you don't break the type invariants.

The List you get from JSON decoding is hard to handle in general. You might actually know that it is a list of integers, but it will not be a List<int> anyway. You can't cast yourself into that, you have to actually wrap the list or create a new list (new List<int>.from(jsonList) or new DelegatingList.typed<int>(jsonList)) which will both do runtime checks - either at creation time or at access time - to ensure that you will never read a non-int out of the resulting list. With a little luck, Dart 2 will make it even easier to convert to a different type of list - but you will still get runtime checks because the static checks can't see that it's safe, and it must either be safe or it must throw.

[eernstg]

Agreeing completely with Lasse, I have just one more comment: We may do something (e.g., change the placement of the type Null in the type hierarchy) such that Object is no longer the same thing as Object?, which is needed in order to create general data structures whose element type is constrained to be a non-null type (if the type argument bound is Object and that's the same thing as Object? then int? and all other nullable types are acceptable type arguments).

[zoechi]

I find it very cumbersome in Dart (and other languages) to always have to deal with nulls in every method, when there is no use for them at all. All values that I access have to be checked for null before any property or method is accessed.

Non-nullable types are like implicit asserts that free the developer from tedious work. The analyzer already does the checks that otherwise the developer had to do with asserts and is-not-null checks. It's just more of what types already do for us, just with more fine-grained control.

If you actually want allow nulls, then all you have to do is adding ? to the type names. In my experience, situations where one actually wants nullable types are quite rare.

Non-nullable types are not about false safety. They are about being able to expressing intent with less code. Just because there are still other sources for bugs, doesn't mean non-nullable doesn't provide benefits. You could similarly argue that the analyzer is redundant because even when you fix all issues it shows, your code can still have bugs.

In my opinion non-nullability is not about adding a feature but removing a bug that was introduced decades ago without a convincing reason and repeated just because people were already used to it.

[zoechi]

do you start your function with something like this?

and writing this is pure nonsense. Just to ensure I'm null-safe when I never asked for null in the first place

[DanTup]

Seems like this thread is going in circles, but my two cents...

@tatumizer IMO, it already reached optimal level of safety. Adding more safety will add more complexity, and complexity breeds bugs! Instead of trivial bugs related to occasional NPE (very easy to fix)

I'd love to know what the "very easy fix" is for NPEs. In my experience, it's a game of whack-a-mole. Your function fails because it was called with a null; what are your options?

1. Put null checks into your function - this is not a fix, it's too late, it's run-time and you've got invalid data, you can't do what you need to do so you crash in front of the user (or gracefully fail to do what you were supposed to and probably leave something in a bad state)
2. Look at the stack and fix the place that was incorrectly providing a null. What do you do about the other 100 places that called your function? Wait until you get an NPE exception stack trace for each one? Doesn't it seem a bit crap to require a bug to occur before you can fix it? Is it reasonable to expect a developer to manually review all code interactions to ensure no nulls can exist where they're not wanted?

In most cases null is just not valid - your code should not compile and you should not be able to ship it. Failing at runtime is just not a reasonable thing for an application to do.

There is no reason you should not be able to write your code such that it doesn't require nullable fields where they don't exist. If you can't explain it to the compiler, how can you expect the computer to run it?

@zoechi Non-nullable types are like implicit asserts that free the developer from tedious work.

I'd say it goes much further than this; asserts are runtime crashes; non-null types give us these guarantees at dev time!

IMO there is absolutely no reason to force people have to deal with nulls in a modern language. How will we ever write software that doesn't crash in front of a user if we can't do something as simple as write an add(int, int) function knowing it can only be called with ints? nulls are an enormous hole on our type system and there's no reason we shouldn't be able to opt-out of them.

IMO if we want to improve the quality of software (and we should, because it's generally poor) we really need to go much further (give contracts and dependent types!) but I can't see how we'll ever get there while nulls are still kicking around :(

[DanTup]

Maybe dart-misc is a better place to continue this discussion?

@tatumizer

What do you do about the other 100 places that called your function?

Who wrote the code for these 100 places bombarding you with nulls? It's what? denial-of-service attack? Or you copied/pasted same bug in 100 places?

Not sure how people write programs these days.

If the tools don't allow you to forbid null, it's all too easy to introduce this, especially on a huge, evolving codebase. One day something is guaranteed not to be null; the next day it needs to be deletable (eg. customer information). The type system gives us no ability to know all of the places we're using an order/booking/whatever and assuming customer is non-null.

And we don't always have the luxury of writing our own code - the team I lead has almost 1m lines of code in our major project most of which was written by developers that no longer work here and an outsourced firm; the quality is not excellent but it's what we have. Sure, it shouldn't have ended up like this, but I can't change the past. Making changes and improvements is really tough when the foundations are bad. Non-nulls would give us enormous guarantees that would help us improve the quality.

Compiler takes care of everything: checks for nulls and throws NPE; checks array bounds on every access; for each type cast, checks the types; checks the number and types of parameters; etc. etc. You guys already live like Kings, and still keep complaining.

This is poor logic. Just because things were even worse before does not mean we should not make them better. NPEs are not a solution, they are a band-aid. Tell your end user that "at least that error you got didn't crash the whole server" and they don't care; you've failed. Runtime failures are failures; we need to be stopping these bugs before shipping. Sure, some devs might not care about this, but some do and we should have that option. Options are good. If you don't want something, don't use it. But don't assume that because you don't want it it's not beneficial to anyone!

They are not kicking around in Kotlin. Instead, it's Kotlin who kicks you around with its null-safety features,

You've said this many times, and I'm sure you're not making it up. However, I haven't seen any concrete examples of the problems so it's hard to really have an opinion on. If you have !!(?) all over your code where you shouldn't, search for it and post some examples; maybe they're solvable problems? (though I think posting to dart-misc would be better than keeping this issue growing!)

[zoechi]

@tantumizer I don't think it's the right place for this kind of discussion and your arguments are IMHO just more of what you already explained as your opinion and I could only reply again with my opinion which would probably lead to a stackoverflow (Bob locks this issue)

[lrhn]

Dart used to have covariant generic classes. It still does, but it used to too.

That means that the []= operator of List isn't type-safe, nor is add. That's nothing new. In Dart 2 there will have to be an implicit check of the parameter, so you can think of it as

void operator[]=(int index, Object value) {
  value as E;
  // assign value at index
}

That means that if you do:

List<Fruit> x = new List<Apple>();
x.add(new Orange());

you get a runtime error when you try to add an Orange to a List<Apple>.

So, your program is "correct" in the sense that the type system won't catch the error. We expect it to fail at runtime. That's nothing new, Dart 1 has the same problem in checked mode, and Dart 2/strong mode is fairly close to forced strong mode (with some extra bells and whistles, and a few stronger requirements around functions).

And yes, you can assign a List<int> to a variable of type List<int?> due to covariant class generics, but it will throw at runtime if you try to add null to it.

The real problem is that generics classes are covariant in the type parameter, but the type parameter may occur contravariantly in some methods. Those methods are wrong. They are also incredibly convenient. As a compromise, all such methods (add, []=, etc) need a runtime check, and they should get one automatically in Dart 2.

If you look at Java, they picked a much more precise approach where you can assign a List<Apple> to List<? extends Fruit>, but not to a List<Fruit>. You can't call add on the former, so that problem is prevented, at the cost of some complexity in the type system.

[eernstg]

@tatumizer

A List<int?> is-not-a List<int> (it's the other way around due to Dart's covariant generics).

This "the other way around" leads to the same paradox. We have List<int>, and we pass it to a function that accepts List<int?> as parameter. Now, the function is free to assign list[0]=null.

This is again covariant generics and it has nothing to do with nullability per se: You pass a subtype List<int> into a context where a supertype List<int?> is expected, which is fine by any type system. However, covariant generics makes contravariantly positioned occurrences of the type argument unsafe. For instance, list[0] = null provides null as the argument where the statically known argument type is int? but the runtime argument type is int, and that will fail because it's (actually, though not visibly) a downcast.

Treating Null is not-an-object, as @eernstg suggests, doesn't solve this problem, because anyway, we must be able to convert List into List<int?> (*)

I don't expect the treatment of null as not-an-Object to make any difference here, because T will be a subtype of T | Null, no matter where in the type hierarchy Null is sitting. This also means that List<int> will be a subtype of List<int?> as long as List is covariant in its type argument.

(It's a separate discussion whether we want to allow a class to declare that a given type argument is invariant or contravariant, in addition to the covariance which is chosen by default. Covariant generics was chosen for Dart because the designers wanted to give programmers simpler types to work with --- in particular, simpler than Java wildcards --- in return for this particular kind of unsafety. We may or may not wish to strike that balance slightly differently in the future, such that developers can make this trade-off in more than one way; this could allow things like the list[0] = null example above to be a compile-time error rather than a run-time error).

There are, of course, ways to resolve the paradox - e.g. by adding "mutable" keyword, treating list as a value, implementing copy-on-write, etc (see swift).

I believe Swift arrays and dictionaries are covariant and user-defined types are invariant, which is probably too weird for Dart. ;-) But, sure, there is no end to the available language designs in this domain, with associated trade-offs.

I have a conjecture that all this trickery in swift is a direct ramification of null-safety. Not sure - please opine.

I wouldn't expect null-safety to sit at the core of this, I would expect null-safety to be just one among many different cases where variance arises. With a little luck, this means that anything that makes variance work well will be good for handling null-safety as well.

[eernstg]

.. So it comes down to runtime check, right? When we write list[i]=new Orange(), compiler should generate code that checks .. runtime types (of list and of value) ..

True. Considering the example in the comment above, let's call the actual type argument of the list T. Then we only know that T <: Fruit and there is no guarantee that Orange <: T, so a runtime check must be performed at list[i]=new Orange().

However, there's nothing stopping a compiler from generating a customized version of the method which only gets called when the actual type of list is List<Fruit> (exactly, i.e., it doesn't get called when list is a List<Apple> or indeed List<S> for any S which is different from Fruit). In that version of the compiled method it is known to be safe to assign an Orange to any entry, so no dynamic check is needed in the loop. This effectively means that a check per entry is replaced by a check per invocation of applesToOranges.

Compiler faces a problem here: given two types T and T1 (known only in runtime), find out whether T1 is assignable to T. This is a hard problem - performance penalty might be way too high. Notice that the check has to be done on each value being assigned, like in the example

  copy10Frutis(List<Fruit> from, List<Fruit> to) {
     assert(from length>=10 && to.length>=10);
     for (int i=0; i<10; i++) to[i] = from[i];  // value-by-value runtime type check!
  }

This is more tricky so you probably shouldn't expect a compiler to be able to do much about it. However, if we assume that it tries really hard then you could do the following: The test that the value of from[i] is acceptable as an argument to to.[]= can be abstracted: If the type argument of from is a subtype Tfrom of the type argument of to, Tto then the type of from[i], Ti is known to satisfy Ti <: Tto, so since Tto <: Tfrom we also have Ti <: Tfrom, so there is no need to check it. So we can start by checking Tto <: Tfrom, and if that's true then we run the loop with no checks; otherwise we run the loop with the check.

But, of course, a compiler is not likely to take that many steps in order to find a very special optimization opportunity. It is a trade-off: Do we want to support an invariant modifier on formal type parameters? If we do, there will be fewer runtime checks and there will be fewer reuse opportunities.

[eernstg]

Thank you, too! Your input is helpful in keeping us aware of the costs of more strict checking. Dart is a language where completely strict static checking has never been the goal, so we do have to be careful.

[munificent]

And because they are uncertain, we don't have justification for the feature

Fortunately, we don't need to justify the feature, we just need to justify running an experiment for the feature, which is all we're currently doing. The results of that experiment will help justify whether or not we should land it. :)

[munificent]

rather about strong typing in general.

That is out of scope for Dart in general, and certainly out of scope for this issue, which is about adding ? syntax. Dart has already decided to have a static type system.

[lrhn]

Definitely aware of it. The current Dart treats all type annotations as "nullable", so null can go anywhere, and nothing prevents you from trying to use a null value by calling a method on it or using it as a test condition. That's basically the two ways you can use a value in Dart, and "using as a test condition" stands out because it doesn't obviously lead to a "no such method" error because there is no method involved, unlike all other uses of null. It's still an error in checked mode because boolean conversion is defined to explicitly check for not being a bool.

If Dart gets non-nullable types, then the valid type of a test condition will likely be just (non-nullable) bool. You'll probably also not be able to do foo.bar() directly if foo has a nullable type, you need to do if (foo != null) foo.bar(); or (hopefully) something shorter like foo!.bar().

[lrhn]

True, it only has to explicitly check for null because all other non-bools were already caught by the type parameter on the conversion closure:

(bool v){
  assert(v != null);
  return identical(v, true);
}(o)

Effectively boolean conversion does check against the value not being a bool, it just splits it into two parts: the bool type on v and the != null in the assert. It could have been:

(b) {
  assert(v is bool);
  return identical(v, true);
}(o)

instead, it would just give a different error.

[DanTup]

What percentage of NPEs is due to uninitialized variables? If it's 50% (very likely), we get a useful feature almost for free. I think it's more than 50%.

This percentage is nothing like 50% in any of our (C#) codebase. There are so many sources null; different expectations between empty collections and null, unobvious optional/mandatory parts of an object graph.

If you can't encode in your types that null is invalid, how should a caller of your function know? Runtime? What if you change it later to no longer be valid - a breaking change only found at runtime (and only when you pass a null)? Your rules make it unreasonably difficult to make a previously valid null no longer valid.

One of the selling points of Dart is to help you find mistakes at dev time, not let your user find them at runtime. Software is too complex to expect to hit every permutation of code paths and state during testing (not to mention more expensive than finding at "compile" time).

I'm not sure why you're so against non-null but you always have the option not to use it. There are lots of people that REALLY want it (blame our bad inherited codebase if you want, but improving and maintaining existing code is a real requirement for most software developers) even it will come with some edge cases.

I've had NPEs in release versions of Android apps from Google, Twitter, MS and other large software companies. This absolutely is not only an issue for sloppy/inexperienced devs that don't test.

[eernstg]

I think the treatment of null is especially tricky in the cases where it is actually an interface issue: "This data structure can contain null / should never contain null", "If this argument is absent, pass null / It is an error if this argument is null". In these cases the standard (implicit) treatment of nullability is error prone, and an explicit treatment is likely to improve correctness in the same way as other explicit interfaces. It is also difficult to address this kind of issue with anything else than an invariant which is stated explicitly, maintained at each relevant point, and hence globally valid; that is, for these cases it fits well to express nullability as a type.

For local variables the situation is somewhat different (because they may need to be "null for a while, but eventually non-null"). Definite assignment analysis was already mentioned several times as a helpful mechanism to handle that, but it is an interesting idea that locals might benefit from being treated less strictly.

[DanTup]

Sure, most bugs are due to complexity. Obvious cure is to reduce complexity.

All of your comments are about reducing complexity. Allowing you to "opt out" of nulls where they don't make sense is reducing (accidental) complexity.

I'm not sure why you're so against non-null but you always have the option not to use it

I wrote 10 posts about it. In short: the medicine costs money and may have severe side effects

If you don't like the side-effects, you don't have to take the medicine :)

I understand why you don't want these, but I don't understand why you think nobody should have them. Surely devs are able to make their own minds up and choose to use this feature where it makes sense? Many other languages have non-nulls and they're generally considered to be a great thing (you're the first person I've ever seen against the idea).

Anyway; I feel we're going in circles and I don't want to add more emails to people subscribed to this case. I'd love to have non-nulls (I really don't think forced nulls belong in any modern language), but I trust the Dart Team will do what makes the most sense. I don't think I can add anything more (and my opinion doesn't count for much, I've probably written the least Dart code of anyone here!).

[eernstg]

@eernstg but where these nulls passed as parameters come from? Aren't they uninitialized variables? Because they spread. You call constructor, pass (uninitialized) var, and the null gets stored in the object, so it looks like initialized field, but the origin of the problem is in another place, where it wasn't initialized. If you remove root cause, you have less problems like this. So the transitive closure of the problem of uninitialized vars can be more than 50% of cases.

The proposal where non-null types by default are used for interfaces (in particular, for function/method signatures and for field types) and local variables may be more flexible (maybe even nullable by default) will actually stop the propagation: The constructor invocation will reject the null as an argument and null won't be stored in the object.

Another matter is that we may prevent the propagation of null statically or dynamically. If we prevent it statically then local variables will be less convenient at usage points (because they will more often be nullable, and hence there will be a T | Null to T downcast for some T in more places), and if we prevent it dynamically (by accepting that downcast silently, but inserting a dynamic check) then the code may fail in more locations that are not obviously error prone.

In any case, the null won't propagate outside the block where that local variable is declared.

Another source of nulls is: functions sometimes returning null, like map[]. This can be cured by another method that throws "not found" instead (as discussed earlier). Not only it throws, but includes the name of key in the message. Quite convenient. Maybe there's a couple of other cases. By targeting those individually, you solve big part of problem naturally.

Sure, the interface of Map.operator[] could have a nullable or non-null return type (if we use a non-null bound on the type arguments, such that we prevent null keys and values, that is!), and then we could communicate the "not found" outcome using an exception or by returning null. Both are useful designs, and we should be able to express them.

[fsc8000]

and then we could communicate the "not found" outcome using an exception

This would be bad design from a performance point of view. I strongly advise against using exceptions for general control-flow.

[munificent]

Hey kids following along at home! @eernstg noted that my original grammar suggestion would allow invalid things like:

class Foo extends Bar? {}

He fixed that by splitting out the type production into a separate one that does not include the optional "?". I updated the grammar here to follow that.

[eernstg]

.. what if we had a special value for "uninitialized"?

You just gave me an idea there! ;-)

What if we treat null as the special value meaning "uninitialized" for variables which have a non-null type but no initializing expression?

This means that we can allow for a local variable C x; to be declared and initialized to null even though C is a non-null type. We can use standard type checking to ensure that x is not made null by assignment, so an actual null value will unambiguously mean that x is being used before initialization, and that should be caught by checking dynamically at every evaluation of x where it is not yet statically known to have been initialized. For cases where definite assignment analysis proves that there is no way to encounter the uninitialized value it makes no difference (there are no locations where the check must be performed), but for cases where static analysis does not prove that initialization has taken place at a location where x is evaluated, there would be a dynamic check.

As usual, we can make the choice to insert the uninitialized-check (i.e., null-check) implicitly, or we may emit a diagnostic message, but that is a separate issue.

The point is that we can allow for "eventually non-null" behavior backed up by a combination of static and dynamic checks which is sound and convenient, which is quite Darty. ;)

[munificent]

Oh, that is an interesting idea!

That might really increase the usability of "eventually non-null" use cases, without punishing the performance of common cases where we can eliminate the runtime checks through definite assignment analysis.

It's also not entirely without precedence in Dart. We have runtime checks for accesses to top level variables to check for cyclic references:

var a = b;
var b = a;

main() {
  print(1);
  print(a);
  print(2);
}

This prints "1" before throwing a runtime error.

I'd like to think about this some more, but once we start playing with implementing non-nullable types, this is definitely an idea to put on the table.

[chalin]

What if we treat null as the special value meaning "uninitialized" for variables which have a non-null type but no initializing expression?

I haven't been following the entire thread, but I believe that what you are referring to was part of the original proposal, see Section B.3.4.

[eernstg]

what you are referring to was part of the original proposal, see Section B.3.4.

Indeed, I hadn't discovered that -- sorry!

[munificent]

I haven't been following the entire thread, but I believe that what you are referring to was part of the original proposal, see Section B.3.4.

Oh, sorry, thank you for pointing that out! I've read your spec several times, but it's a little too big to hold it all in my increasingly small brain space.

[munificent]

did you consider making nullability a property orthogonal to type (as opposed to "part of the type")? E.g. reading "String? x" not as "x or type nullable String", but as "x or type String, nullable".

Yes, we have discussed making nullability a property of declarations and not types.

I'd have to dig up our old notes, but my recollection was that it ends up feeling very limited in abitrary ways, especially around generics.

class Sorter<T extends Ordered?<T>> {}
class Sorter<T extends Ordered<T?>> {}
class Sorter<T extends Ordered?<T?>> {}

Only the second is valid according to the proposed syntax.

There are other arguments, which we can discuss if you think the idea is worth it.

I'll gently remind you that this issue is only for tracking experimental non-nullability syntax. The semantics are, of course, vital, and we have thought about them a lot, but they are out of scope here. When we start working on them, we (probably me) will write up a more detailed proposal to send out before we start putting real effort into it.

This issue is deliberately vague on semantics because that comes in a later phase.