Allow variadic control of generics (i.e. List)

[joeconwaystk]

The following code shows two seemingly identical-in-behavior code blocks, but one is a type error and the other is not. If I am misunderstanding something, please feel free to close with minimal comment.

The method f takes a List<dynamic> and adds a int to it. Block 1 succeeds, and block 2 fails with a type error. Both blocks pass the same list literal argument. The difference is that the failing block first assigns the list literal to a local, inferred variable before passing it to the function.

void main() {
  // Block 1
  final a = f(["string"]);
  print(a);

  // Block 2
  final b = ["string"];
  f(b);
  print(b);
}

List<dynamic> f(List<dynamic> list) {
  list.add(0);
  return list;
}

In block 1, the argument's type is inferred to be List<dynamic> because of the location of its instantiation. In block 2, the argument's type is inferred as List<String> first, before being successfully upcast to List<dynamic>. But within block 2, this makes adding an int illegal.

I think this is surprising: extract an argument into a local variable and the behavior of the program would change. I also think it invalidates some guarantees about the type of a variable in a given lexical scope.

[matanlurey]

This is working-as-intended. There are smarter folks to give the gritty details, but basically:

var a = ['string'];

... is the same as ...

List<String> a = <String>['string'];

While:

List<dynamic> a = ['string'];

... is the same as ...

List<dynamic> a = <dynamic>['string'];

This might not make sense in this particular example, but imagine the following:

List<num> evilNumbers = [1, 7, 13];
evilNumbers.add(99.9);

With your suggestion, this would be invalid code, because we'd create a List<int>, upcast it to a List<num>, and then fail, dynamically (at runtime) trying to add a double to a List<int>. With the current behavior, if the type parameter is unspecified, type inference fills it in based on the context. Here are some more neat examples:

T fetch<T>() => /* Implementation Detail */

void main() {
  String name = fetch(); // == fetch<String>();
  example(fetch()); // == example(fetch<List<int>>());
}

void example(List<int> numbers) {}

I think the frustration you're feeling is due to the use of <dynamic>, which unfortunately in Dart2 just doesn't mesh well with the rest of the language like it did in Dart1. For example, I have a pretty thorough set of examples where raw types (i.e. List foo) break a lot of assumptions in Dart2, especially to new users (because it is the same as List<dynamic> foo): https://github.com/dart-lang/sdk/issues/33119.

I think your particular question is why extracting an argument changes the behavior of a program, and that is (un)fortunately because most of the time (I'd say, 95 to 99%, or more), you want the collection or generic inferred/reified using the most data available. In the case of ["string"], the only information available is that only element is a String. If we didn't do that, you couldn't write:

final b = ['string'];

// Would be a runtime error, List<dynamic> is not a List<String>.
List<String> c = b;

Hope that helps!

[joeconwaystk]

Thanks for writeup Matan. I think we're talking about something similar, but not quite (also, don't worry, not frustrated - just raising a flag).

To clarify, I agree that the type inference behavior makes sense. I am claiming that the type system is incorrect here, and I don't have a suggestion on its resolution.

I'll use another example, without dynamic.

Let's say I have a function that takes a num. I have a guarantee that any instance methods of num are valid for the argument, I am not allowed to pass a non-num value to this function. I can pass subtypes of num as long as I treat them as num.

void f(num n) {
 // I've been guaranteed n is an instance of num (or null)
}

However, if the argument is List<num>, the same guarantees doesn't exist. There is different behavior if the the argument is a List<int> vs. List<double>, but I have written valid Dart code that I have no way of knowing can be invalid at runtime.

// This will fail if arg is List<int>
void f(List<num> list) {
  list.add(0.0);
}

Contrast this with down-casting a collection. I cannot pass a List<dynamic> to f because the types of each element cannot be guaranteed to be num. This makes sense; it allows me to guarantee that every element of list is a num in this method. But it doesn't guarantee me the validity of that type, as shown here. @leafpetersen

[matanlurey]

I'm not a language expert and definitely feel free to wait for @leafpetersen (we should try and help capture some of this conclusion on the language site, if possible), but what you are talking about is covariance and contra-variance.

In Dart1, dynamic was special, it was basically any. It's now, in the type system, the same as Object (in fact there are some issues opened by @lrhn to make dynamic and Object basically the same thing).

For generic types specifically, the types unfortunately are only very useful (statically) for reads. Contrast this with Dart1, where the types were almost always safe for writes, but you could not guarantee something annotated with List<num>, had numbers at all:

// Legal, and common, Dart 1.
void main() {
  var x = [];
  x.add('I am not a number silly');
  example(numbers);
}

void example(List<int> numbers) {
  for (var x in numbers) {
    print(x is num); // Might be False (in this case it is!)
  }
}

... produces no static or runtime errors, until a + b fails at runtime.

Dart2 guarantees any annotated type, including generics, when read, is that type (or "greater"):

void main() {
  var x = [0, 0.5, 1]; // Implicit List<num>
}

void example(List<int> numbers) {
  for (var x in numbers) {
    print(x is num); // Will always be True
  }
}

A side-effect of this choice is that writes have to do a little extra work. In your case:

void f(List<num> list) {
  if (list is List<int>) {
    return; // Or throw, expected List<num> or List<double> etc.
  }
  list.add(0.0);
}

If Dart ever gets ways to control variance ( 🙏 🙏 🙏 ), you could write something like:

void f(List<in num> list) {
  // Guaranteed that 'list' is EXACTLY a List<num>, not a List<int> or List<double>.
}

FWIW, there is a fairly famous S/O question on almost this: Why are Arrays invariant, but Lists covariant? - in this case for Java, but the answers apply here as well (except we don't have invariant Arrays or invariant types yet).

[joeconwaystk]

I think we are saying the same thing. If I take the example from the accepted S/O answer and convert it to Dart, I get a runtime exception and the analyzer can't tell me about it beforehand.

void main() {
  List<int> arr = [1, 2, 3];
  List<num> arr2 = arr;
  arr2.add(2.54); // throws at runtime
}

If the solution is that the language needs variance specifiers, and that collection types have to use these specifiers, awesome. But as it stands now, I can write code that is always invalid at runtime and I don't get a warning.

[matanlurey]

@joeconwaystk:

I think we are saying the same thing.

For sure! Just trying to help this issue turn into something actionable :)

If I take the example from the accepted S/O answer and convert it to Dart, I get a runtime exception and the analyzer can't tell me about it beforehand.

FWIW, in Java using List you'd have the exact same behavior. The only real difference is that Array is invariant, and Dart does not have arrays (per-say) or invariance (yet, at least).

But as it stands now, I can write code that is always invalid at runtime and I don't get a warning.

That's the case across the spectrum of languages similar to Dart too, including Java. The goal of Dart2, unfortuantely, is not lack of errors, but rather, heap soudnness - so that optimizing compilers, if presented with a List<String>, can be guaranteed that every element is a String (or null, for now).

There are many other places in Dart2 runtime errors are possible:

void main() {
  List<String> x = [];
  List<Object> y = x;
  List<int> z = y; // Error: List<String> is not a List<int>
}

void main() {
  String x;

  // NPE
  print(x.substring(0, 1));
}

We are exploring how to tighten type system and static checks without waiting for a "Dart 3.0" (https://github.com/dart-lang/sdk/issues/33749), it would definitely be useful to add your experiences and requirements. Some other popular threads include:

• https://github.com/dart-lang/sdk/issues/33119
• https://github.com/dart-lang/sdk/issues/31410

[rich-j]

@matanlurey - the S/O article Why are Arrays invariant, but Lists covariant? is for Scala not Java. As you know Scala has a "strong static type system" and "support for functional programming". As the answer points out, the Scala List is an immutable data structure and covariant in assignment.

Your example above converted to Scala (with explicit typing) does not compile (which is good):

def test()= {
  val x: List[String] = List.empty
  val y: List[Object] = x    // Valid
  val z: List[Int] = y    // Compile error - type mismatch; found: List[Object] required: List[Int]
  val z2: List[Int] = y.asInstanceOf[List[Int]]   // Runtime error - really a programmer error for forcing an unsafe typecast
}

As a developer, I appreciate the compiler catching the type mismatch but also letting me explicitly override if I need to.

The dartz contributed package https://github.com/spebbe/dartz provides immutable collections (IVector, IList, IMap, ISet) and containers (Option, Either, Tuple). We are using this package in a project that we're converting from Typescript. Our backend is written in Scala so the hybrid object/functional style is familiar to us.

All of our data structures are immutable and strongly typed. The lack of support for covariance and contravariance is one of our biggest ongoing headaches. To add to the headache, the variance issues show up as a runtime casting error that has a useless stack trace.

Better support for dartz would be appreciated. I know the developer is having trouble with variance issues and also higher-order functions.

[lrhn]

The issue you are hitting here is that Dart generics are covariant and not safe. We allow you to pass a List<int> as a List<num> even though you can't actually do all List<num> things on it. That has advantages and disadvantages. The advantages include being able to use a List<int> where a List<num> is expected without having to copy or wrap it, and that works as long as all you do is read from the list. If you try to write the coviariance isn't safe any more, and you get a run-time error.

You are saying that you don't think the advantages outweigh the disadvantages. You are not alone, other people want the type system to be completely safe too. There are also people who do use the unsafe-but-convenient features and don't want that use-case to get more complicated. Immutable collections are actually a case where invariance works because you never write to them.

I don't think we have any current plans of changing the default, but at some point we might want to investigate whether we can allow other variants as opt-in.

[rich-j]

@lrhn I didn't say anything about relative advantages and disadvantages. I did point out that I appreciate the (Scala) compiler finding unsafe type usage but also allowing me to override when (I think) I know better. I continue rambling about immutable data structures and the challenges we've encountered due to incomplete variance support in Dart. I was trying to address @matanlurey previous comment to "add your experiences and requirements".

Your comment that "Immutable collections are actually a case where invariance works" is a bit misleading. Invariance (i.e. not covariant and not contravariant) always works for both mutable and immutable collections. As you point out, mutable collections can be used with different variances as long as developers don't break things (e.g. use invariantly or immutably). Immutable collections are always safe to use covariantly (okay one link to a variance description).

So my rambling is really just another request for Dart to support variance specifiers. I'm by no means a language expert but I have appreciated languages (e.g. Scala) that just seem to do the right thing and keep me from being stupid. This can also be said for immutable data structures.

[eernstg]

@rich-j wrote:

okay one link to a variance description

OK, I'll comment a bit on that. ;-)

During the work on designing wildcards and writing Adding Wildcards to the Java Programming Language, we were certainly aware of the history, including unsafe variance and safe covariance (the latter was published in 2006, but it is based on a language, gbeta, which had been using that approach for about 10 years).

Later, when Dart was designed, all classes were made covariant in all type arguments, and "covariant parameters" are hence subject to dynamic checks—not by accident, but in order to strike a different balance between the complexity of type annotations in source code and the amount of dynamic checking.

I've been pushing for support for use-site invariance declarations (let's just assume that we would use a modifier spelled 'exactly'):

List<exactly num> xs = ...;
xs..add(42).add(4.2); // Safe

so even though this issue was closed with 'working as intended' (which is true for Dart of today), we might be able to express at least invariance at some point in the future.

Aside: I don't think contravariance is equally important: If some entity should work as a sink, use a function! (The parameter types of a function type are contravariant).

Nevertheless, I actually think that it makes sense for Dart to use the covariant types by default, even though it introduces some potential type errors at run time.

The point is that this works quite nicely in situations where an instance, say, a List<T>, is created and populated "near the location where it is created" (e.g., in the body of a constructor, or at least in several methods of the same class). This means that it will often be trivial to ensure that the static type is List<T>, not List<S> for some supertype S of T: so it's effectively accessed invariantly.

Later in the life of said object it may be accessed via covariant supertype (say, as an Iterable<S> where T <: S), and if it is being used but not modified, the constraints enforced in Java for a List<? extends T> will almost automatically be satisfied: We are restricting ourselves to the "read-only" interface, because we are using that instance in a near-functional manner.

In short, a style where objects subject to covariance are handled in a near-functional manner (create and populate it in a narrow "owner" context, then use it in "client land" as a read-only entity), those dynamic errors will be rare in practice. You could also say that there's a very subtle nudging effect in the direction of using that style. ;-)

Still, if we have an exactly modifier on type arguments then we can enforce invariance in the cases where some objects may travel further during that "populate" or "read-write" part of their life, and we can of course also make the "owner" context statically safe, rather than relying on knowing the actual type argument precisely because we've just created the object a few lines earlier.