Shadowing a non-local variable

[eernstg]

Promotion of fields is probably the most prominent request we have in response to the release of null safety. We could support the coding style where a field is copied into a local variable in the following way:

class C {
  num n;
  C(this.n);

  void foo() {
    shadow n; // Desugars as `num n = this.n`.
    if (n is int) {
      // `n` is promoted here.
      n = 42; // Desugars to `this.n = n = 42;`.
    }
  }
}

This would help keeping the local variable and the field in sync (so it's less error prone than the "manual" version where we simply declare var n = this.n; and remember to write it back as needed). It is also rather explicit: the declaration of the local variable would be an error if there is no field with the same name, and the declaration immediately sends the signal to a reader of the code that this local variable is intended to be synchronized with the instance field with the same name.

The idea easily generalizes to class variables (static num n2 = 0; and shadow n2;) and top-level variables.

Assignments to the local variable will be a compile-time error in the case where the associated non-local variable does not have an implicitly induced setter.

Surely it will be complex to cover all cases, such that any updates to the local variable is guaranteed to be accompanied by an update to the field, but I'm sure we can sort out the details.

If it turns out to be too much of a breaking change to make shadow a built-in identifier (this means that no type can have the name shadow, but that would be really rare anyway because user-written type names in Dart are almost universally Capitalized), then we could consider a syntax like var this.n;.

[esDotDev]

Exactly. This code is as "unsafe" as anything we're talking about here:

class A extends StatelessWidget {
  double getHeight() => 20;
  @override
  Widget build(BuildContext context) {
    return SizedBox(height: getHeight());
  }
}

class B extends A {
  @override
  double getHeight() => -1; // I just broke the sub-class
}

A programmer, wanting protection from any future sub-class, should write:

 double getHeight() => 20;
 ...
final height = max(0, getHeight()); //This looks redundant to untrained eye
return SizedBox(height: height);

But I don't think anyone really does this, at least not for that reason, because taken to the extreme, this logic will put you into quicksand and just add a tons of relatively pointless verbosity to your code.

Its much more efficient to just trust your internal logic is what you wrote, and put the onus on future developer to read your code, and not break anything.

[Levi-Lesches]

Within OOP paradigm, any subclass may override any method and destroy your logic.

My point was that your problem is with OOP and inheritance in general, not Dart. It definitely has points that can be confusing, but it also allows you to write much more expressive code. If you're looking for a short-hand to a common problem, if-vars are the way to go. Getting rid of OOP or type/null soundness is a bit out-of-scope for this issue.

Specifically, you're looking for the Dart equivalent of final methods. The @nonVirtual annotation will show a warning if someone overrides a method/getter/etc with this annotation. Alternatively, you can create a final class with the @sealed annotation, so the class cannot be extended.

[Levi-Lesches]

If the price for stopping the compiler from constantly complaining about nulls in your already null-checked code is declaring your class @sealed...

No, the price for stopping the compiler complaining about null is !. If-vars can also help. The price for not letting your getters/methods change values every second is @nonVirtual, @sealed, or stable (#1518), but those aren't needed here.

[lrhn]

1. Maybe.
2. Maybe.

Can the compiler ever figure that out for you? Probably not. Is it type-sound? Yep. Compiler is happy.

[eernstg]

That was a lot of comments! :smile:

@Hixie, I'd like to comment on this:

FWIW, I feel the language should help the programmer, not the other way around.

Certainly! - but it is not obvious what that would mean. Here is a perspective on this proposal that emphasizes how it would help developers (and that explains my original motivation for the proposal a bit more than the initial posting did):

I've seen many situations when code was migrated to null safety where existing code seemed to work smoothly, but the migration created a dilemma:

// Legacy code.

void use(int i) {/* ... perform some action ... */}

class C {
  int i;
  void task() {
    if (i != null) {
      use(i);
      i++;
    }
  }
}

For some reason we can't migrate i to be non-nullable, so we'll have int? i;.

// Migrated to null safety, alternative 1.

void use(int i) {/* ... perform some action ... */}

class C {
  int? i;
  void task() {
    if (i != null) {
      use(i!);
      i = i! + 1;
    }
  }
}

This alternative is inconvenient, because it could be within reach to show that the invocation of use won't change the value of i. However, the compiler doesn't perform closed-world analyses so it wouldn't be sound to allow us to omit the !.

There is another alternative, which relies on the correctness of caching the value of i. This is a property of the software that a compiler would never be able to decide, but developers make lots of decisions about how to achieve a particular outcome when they write code, and they need to consider "business rules" like that all the time.

So we may well prefer the following:

// Migrated to null safety, alternative 2.

void use(int i) {/* ... perform some action ... */}

class C {
  int? i;
  void task() {
    var j = i; // Create a local variable `j` that "shadows" `i`.
    if (j != null) { // `j` gets promoted.
      use(j); // So we don't need the null check.
      j++; i = j; // But we must remember to write back the updated value.
    }
  }
}

A shadowing variable does just these two things: It copies the value of the shadowed variable into the local variable at the declaration, and it copies the value back into the shadowed variable whenever the shadowing variable is mutated. So the following code does exactly the same as alternative 2:

// Migrated to null safety, alternative using `shadow`.

void use(int i) {/* ... perform some action ... */}

class C {
  int? i;
  void task() {
    shadow i; // Create a local variable `i` that "shadows" `this.i` and gets that initial value.
    if (i != null) { // The local variable `i` gets promoted.
      use(i); // No null check.
      i++; // Write-back done by compiler.
    }
  }
}

So the language helps the developer in the following ways:

• The word shadow serves as a reminder that the variable i is shadowed by a local variable. The fact that they have the same name makes it easier to see which variable any given shadowing variable is shadowing, and it makes the declaration more concise.
• The initialization is implicit, but as soon as we see shadow we know it's = this.i.
• If we ever modify the local variable then the compiler will insert a write-back operation to the shadowed variable, which is easy to forget if we use alternative 2. If the write-back operation is an error (because there is no setter for the shadowed variable) then the mutation of the shadowing variable is also a compile-time error.

The shadow mechanism is indeed less principled than stable getters (#1518), and it is of course wrong to use it if it's a bug to cache the shadowed variable (for the given amount of time), but I think that the shadow mechanism can help developers get this code right, whereas the manually written alternative 2 is rather error prone.

[Levi-Lesches]

@eernstg Clarifying question: is there a meaningful difference between this issue and if-vars, or are they just two different proposals for syntax? The only thing I can think of would be setting the shadow to a new value, which would automatically update the corresponding field, but that seems like it should be par for if-vars as well. If not, which would you prefer?

[eernstg]

The if-vars feature (#1201) would allow us to declare a new local variable in a new location (#1210 and #1420 allow that, too, but in expressions in general). But, as you mention, none of those mechanisms are intended to write back the new value when the local variable is mutated.

I general, it wouldn't be safe to introduce any such write-back operations, unless we'd add a lot of extra constraints on the expression used to initialize the new local variable:

class Pair<X> {
  X left, right;
  Pair(this.left, this.right);
}

class C {
  List<Pair<num>> points = [Pair(1, 2)];
  test() {
    var index = 0;
    if (var points[index].x is int) { // Declares a new local, `var x = points[index] as int;`.
      index++;
      x = 24; // What should this assignment do?
    }
  }
}

If we wish to make the assignment x = 24 a write-back operation then it would assign 24 to the new local variable x, and also perform something like points[index].x = x. However, that assignment would throw, because index is now 1, and points contains only one element.

So should we cache the value of index or points[index] in order to be able to re-evaluate the expression that provided access to the getter x? We could require that e is a stable expression (#1518, if we introduce that) in order to allow constructs like if (var e.x is int) ..., but that is a very strong restriction on the constructs allowed by the current proposal.

In summary, I don't think if-vars can support write-back semantics, and a similar mismatch exists for #1210 and #1420.

However, the ability to introduce a new local variable implicitly would work very well together with shadow variables. Using the approach in #1210 we could do this:

var int? global;

void main() {
  ...
  if (shadow: global is int) {
    global++; // We get the promotion and soundness, and ensure write-back.
  }
}

[Levi-Lesches]

Subscribed to #1210 :)

Also, I'm not sure I understand -- how would shadow handle the case you provided with points [index]?

class Pair<X> {
  X left, right;
  Pair(this.left, this.right);
}

void main() {
  List<Pair<num>> points = [Pair(1, 2)];
  int index = 0;
  if (shadow: points [index] is int) {
    // what's the shadow's name? 
    // points and index is already taken, and points [index] should be a regular expression
    // say, for the next example, it's called shadow. 
    index++;
    shadow = Pair("new", "pair");  // where should this new object go? In index 0 or index 1?
  }
}

[Levi-Lesches]

When we assign n = 42, which means this.n = n = 42, a rogue setter for this.n fields may interfere with our guarantee.

Good point. What about desugaring into this.n = 42; n = this.n? The values may still not be in sync, but the shadowed n is playing by the rules defined by the getter and setter, so it's really just sugar and nothing else.

class Person {
  int? _age;
  int? get age => _age;
  set age(int value) => _age = value + 10;  // doesn't time fly? :)

  String grow() {
    shadow age;
    if (age == null) return "I'm not born yet.";
    // age can be promoted now
    age += 5;   // is really old value + 15
    // This translates to: 
    // this.age += 5;
    // age = this.age;
    return "Now I'm $age years old!";
  }
}

void main() {
  final Temp temp = Temp();
  print(person.grow());  // "I'm not born yet."
  person.age = 5;        // _age == 15
  print(person.grow());  // "Now I'm 30 years old!"
}

That example may be confusing, but to the dev who wrote the setter, the shadow is working exactly as intended.

[eernstg]

@Levi-Lesches wrote:

how would shadow handle the case you provided with points [index]?

I haven't looked into this kind of generalization in detail. The basic proposal given here is intended to support safe shadowing (copy-in, write-back) of an existing getter/setter pair (that might be written explicitly, or it might be implicitly induced by any non-local variable), relying on a developer-provided promise that it isn't a bug to cache the value returned from the getter.

These declarations are expected to be found in an enclosing lexical scope, or in the interface of the current class/mixin (so we'd implicitly prepend this), or in an extension. This means that the choice of getter/setter is stable (as defined in #1518): It is either a global/static property, or it is a property of this (and this is stable over the complete execution of any instance member body).

In other words, we're not going to get a different value from time to time because we're calling a different getter, we can only get a different value because the same getter returns a different value from time to time.

And the developer-provided promise mentioned above says that as long as we're calling the same getter, it's OK to cache the value.

I don't think it's feasible to extend shadow variables to handle any getters that aren't accessed in a stable manner. In particular, we don't want to shadow expressions like points[index], because we have no notion of stable methods. (We could have that, but I don't want to take a dive into that tarpit ;-).

So we'd need to commit to working on a specific receiver by accessing it with a stable expression, and then we could shadow a getter:

class Pair<X> {
  X left, right;
  Pair(this.left, this.right);
}

void main() {
  List<Pair<num>> points = [Pair<int>(1, 2)];
  int index = 0;
  final point = points[index]; // `point` is a stable expression.
  if (shadow: point.left is int) { // Introduces local `num left = point.left;`.
    // `left` promoted to `int`.
    index++; // No problem, `point` is still the same object.
    left = "A string"; // Compile-time error.
    left = 3.14; // Accepted at compile time, but `point.left` setter throws.
  }
}

So we could generalize shadows to handle getter/setter pairs accessed on a stable expression, and it might be worthwhile. But it might not carry its own weight, considering that we need a strong developer-provided commitment that caching isn't a bug, and that gets more an more delicate if we generalize the ways in which we can specify that getter/setter pair.

[eernstg]

@tatumizer wrote:

If the goal of write-back is to guarantee that "the local variable and the field are in sync",

The mechanism as such doesn't provide any guarantees about being in sync, and there is no end to the ways in which you could violate that sync-ness.

The use of a shadow variable relies on a developer-provided assurance that it is not a bug to cache the value obtained from the underlying getter, and it is not a bug to write back the value whenever the shadowing variable is updated. I think that's true in a large number of cases in practice, and when that's true the automatic write-back would probably help avoiding some bugs.

[lrhn]

What Erik says. Or, in other words: You are opting in to functionality which represents and expects a certain underlying behavior, but which does not guarantee that behavior. If the expected behavior holds, then your code does what it looks like. If not, you've broken the abstraction, but that is on you, not the compiler. The code is still type sound no matter what. That's one thing the compiler is not willing to compromise.

You're opting in to something which can fail. That's the same as reading a late variable. It's a known failure-point, it's statically detectable, and it's visible in the source (in some way).

That's where implicitly and unsafely promoting an instance variable differs: There is nothing local in the promotion or the use which suggests that it can fail, it looks exactly like the infallible local variable promotion. You have to go one step further back, to the declaration, to see that it's not a local variable (and then remember that it makes a difference).

[Levi-Lesches]

@eernstg okay so I see from your example that we should only allow shadow to be used on variables (peferably stable ones) and not any expression. That makes sense, and fixes the problem of shadow list [index].

Out of all the "declaration expression"/if-vars/etc. proposals I've read (like 6 of them), I support this one since it most closely resembles how this would be done without today, which makes it essentially just sugar, and is more readable IMO than declaring variables inline. To further readability, I would support forcing shadow to be a statement on its own line -- like regular variable declarations -- rather than an expression, to avoid ambiguity/really long lines.

[mit-mit]

https://github.com/dart-lang/language/issues/1980 was closed as a duplicate of this. 1980 had suggested using the keyword fix rather than shadow.

[aloisdeniel]

Really looking forward to a solution for this.

Since I don't like using the ! operator (potential runtime exceptions), my code has this pattern almost everywhere :

final amount = this.amount;
if(amount != null) {
  return execute(amount);
}

I really like the Swift if-let optional chaining approach which is really simple and doesn't introduce complex new concepts (like shadow in this thread).

For example, this would give :

if final amount = this.amount {
  return execute(amount);
}

Here, the scope is clear for the local shadowing variable, in my opinion.

[esDotDev]

Where's the null check, is it implicit somehow?

[ds84182]

I have an alternative proposal for shadowing. Instead of using shadow as a keyword, reuse with which can act as a way to "import" within the current scope:

with amount;
if (amount != null) {
  return execute(amount);
}

In cases where you don't want to pollute your scope with random "with" imports, they can be nested within expressions:

if (with amount: amount != null) {
  return execute(amount);
}

Or potentially it can be used as a prefix inside an expression to lift it into a variable binding? (trying to think of a way to avoid duplicating "amount"):

if (with amount != null) {
  return execute(amount);
}

In general, with bindings attempt to use the last identifier in the expression as the name of the introduced variable. If this isn't possible or desired, as can be used to set or change the name of the binding:

with amount.value as amount;

Also, when used as an expression prefix or a statement, multiple bindings can be introduced with ,:

with widget.child, widget.leading, widget.trailing;
return Row(children: [if (leading != null) leading, /* etc. */]);

Perhaps cascade syntax can be overridden within the with binding to introduce bindings:

with widget..child..leading..trailing, foo, bar;

For mutability, it makes sense to allow with bindings to be mutable, where they refer to an associated setter. For example:

with json..["foo"]..["bar"];
foo = "foo value";
bar = "bar value";

Of course, I think it would be wise to make the syntax equivalent to a lazy & locally cached variable if used like this. Otherwise you risk unnecessary calls to getters.

---

Anyways to be honest, I'm just throwing things at a wall to see what sticks here. Would like to see improvements here around variable shadowing so there's less work needed when promoting a field.

[dancamdev]

Where's the null check, is it implicit somehow?

I think @aloisdeniel refers to using the conditional to check the value for not being null.

currently in Dart you need to conditionally check a variable or use the ! notation.

i like the solution that @aloisdeniel proposed and swift already implements a lot

[ykmnkmi]

Zig like if expression test for null:

if (/* this. */ amount != null) |amount| {
  return execute(amount);
}

[munificent]

I really like the Swift if-let optional chaining approach which is really simple and doesn't introduce complex new concepts (like shadow in this thread).

For example, this would give :

if final amount = this.amount {
  return execute(amount);
}

The in-progress pattern matching proposal has two features that when combined should help a lot of cases like this:

• An if-case statement to match a single pattern in the condition of an if statement.
• A null-check pattern to match when a value is non-null.

Combining those two features would let you write:

if (this.amount case var amount?) {
  return execute(amount);
}

(Technically, you could even omit the this., though this is a little strange looking:

if (amount case var amount?) {
  return execute(amount);
}

You still have to say the name twice, once to refer to the field and once to refer to the name of the new variable being bound, but it's otherwise pretty terse.

[dancamdev]

I really like the Swift if-let optional chaining approach which is really simple and doesn't introduce complex new concepts (like shadow in this thread).

For example, this would give :

if final amount = this.amount {

  return execute(amount);

}

The in-progress pattern matching proposal has two features that when combined should help a lot of cases like this:

• An if-case statement to match a single pattern in the condition of an if statement.

• A null-check pattern to match when a value is non-null.

Combining those two features would let you write:

if (this.amount case amount?) {

  return execute(amount);

}

(Technically, you could even omit the this., though this is a little strange looking:

if (amount case amount?) {

  return execute(amount);

}

You still have to say the name twice, once to refer to the field and once to refer to the name of the new variable being bound, but it's otherwise pretty terse.

This looks quite confusing to me, gotta be honest, especially the approach omitting this.. It's not so clear which one is the unwrapped and null safe variable.

[eernstg]

The proposal to support patterns is definitely a very powerful bundle of features, so it's not a surprise that many things can be expressed using patterns. However, patterns wouldn't serve the same purpose as the mechanism proposed here (that is, shadow declarations):

A pattern is capable of introducing new variables, and it is capable of initializing those variables (e.g., by calling getters). So we would get the same effect using a pattern as we would get from the initialization of a shadow variable. However, that's only half of the semantics of shadow variables, and we won't get the other half.

A shadow variable is designed to shadow something, and this notion is supported by implicitly generating code to write back the new value whenever there is an assignment to the shadow variable, and to report a compile-time error at every assignment to the shadow variable if the shadowed entity doesn't support assignments (in other words, a shadow of a final variable is a final local variable).

It is not reasonable to use the same implicit write-back semantics for a pattern, because it cannot be assumed in the general case that it is correct to perform this write-back operation.

For instance, we could certainly use a pattern to declare a local variable index and give it an initial value v by calling a getter someObject.startingIndex, but it is perfectly reasonable to do this in order to mutate that variable without having a write-back at each assignment. For instance, we might have a list of objects someObject.xs, and we might want to iterate over the elements of someObject.xs starting at index v, but it could be a serious (and perhaps subtle!) bug to write back the new value of index on every assignment (as in someObject.startingIndex = index = newValue).

It's simply wrong to assume that every local variable is intended to shadow any given variable which is used to initialize it, and that's the reason why I think we would need an explicit keyword like shadow in order to enable the write-back operations.

This is good for correctness because we will never get the write-backs unless we have explicitly asked for them by writing shadow. In return, when this commitment has been made, we will never forget the write-backs. And then we can use any other mechanism (that doesn't contain the word shadow) in the case where write-backs should not occur.

[munificent]

(Technically, you could even omit the this., though this is a little strange looking:

if (amount case amount?) {

  return execute(amount);

}

You still have to say the name twice, once to refer to the field and once to refer to the name of the new variable being bound, but it's otherwise pretty terse.

This looks quite confusing to me, gotta be honest, especially the approach omitting this.. It's not so clear which one is the unwrapped and null safe variable.

Oops, I accidentally left off a var. The actual syntax would be:

if (this.amount case var amount?) {
  return execute(amount);
}

Is that clearer?

[nate-thegrate]

Now that patterns have been fully implemented for a while, I think it'd be good to revisit this issue and decide whether it's worth pursuing.

ints can use bitwise operators while nums can't, so I'll use that as an example:

class C {
  num n;
  C(this.n);
  bool get condition1 => n < 0xFF;
  bool get condition2 => n < 0x0F;

  void foo() {
    shadow n;
    if (n is! int) return;

    if (condition1) n &= 0x0F;
    if (condition2) n &= 0xF0;
  }
}

In this use case, having a shadow keyword is a clear advantage: even with Dart 3 pattern features, there's no elegant way to pull off something like the foo() method above.

However: The current Dart 3 features work beautifully under slightly different circumstances.
What if, instead of reassigning a var, we're updating a final?

class C {
  final Iterable n;
  C(this.n);
  bool get condition1 => n.isNotEmpty;
  bool get condition2 => n.length >= 2;

  void foo() {
    if (n case List<int> intList) {
      if (condition1) intList.add(0xFF);
      if (condition2) intList[1] &= 0x0F;
    }
  }
}

If a goal can be accomplished via mutation rather than reassignment, then shadow isn't needed.

My opinion is that we now have an abundance of ways to parse objects and handle different types. There are scenarios where the ability to shadow a non-local variable would be very helpful, but many (perhaps all) of these cases can be handled with pattern matching, or just by adjusting the class structure.

[esDotDev]

@nate-thegrate AFAIK this feature proposal exists primarily to ease the issues around type-promotion and null-checks since the release of NNBD.

Can you use pattern matching to solve the canonical use case here?

  int? i;
  void f() {
    if (i == null) return;
    print(i.isEven);       // Compiler Error
  }

[nate-thegrate]

@esDotDev this is a good point: I didn't mention nullable types at all, but they're another viable use case for shadowing.

Having a way to promote a non-local variable within the scope of a function would be awesome:

n &= 0x0F;             // with type promotion
n = (n as int) & 0x0F; // without type promotion, gross

But when compared to the clunkiness of type casting, adding a single null check character isn't that bad in my opinion.

print(i.isEven);  // with type promotion
print(i!.isEven); // without type promotion

If the function needs to reassign a nonlocal variable (i.e. i = something), then without shadowing as an option, we're stuck with using a bunch of null checks throughout the function.

But if you just need to read a value, or if updating the value can be done with a method rather than reassignment, then you can accomplish it with pattern matching:

int? i;
void f() {
  if (this.i case int i) {
    print(i.isEven);
    // more non-nullable code can go here
  }
}

Depending on what f() needs to accomplish, it could be implemented with an if-case statement as shown above, or with a switch statement or switch expression.

My opinion is that we aren't missing out on much without the shadow keyword, and adding it will obviously come with costs:

• time & effort for implementation
• probably can't use shadow as a variable name anymore
• it's one more language feature for new developers to learn

That being said, if the Dart language implemented shadowing, I would happily use it in several places.

[munificent]

Can you use pattern matching to solve the canonical use case here?

Yup:

int? i;
void f() {
  if (i case var i?) {
    print(i.isEven);
  }
}

Now that patterns are out, I find myself using this idiom quite often.

[Reprevise]

If we get something like "guard-let" (#2537) then I wouldn't have any need for shadowing. My main use case for shadowing is if I need to do multiple operations on the variable and my dislike for over-nesting.

current:

int? i;
void f() {
  if (i case var i?) {
    print(i.isEven);
    print(i / 2);
    print(i / 4);
  }
}

proposed:

int? i;
void f() {
  shadow i;
  if (i == null) return;

  print(i.isEven);
  print(i / 2);
  print(i / 4);
}

guard (my preference, probably):

int? i;
void f() {
  guard let i = i? else {
    return;
  }

  print(i.isEven);
  print(i / 2);
  print(i / 4);
}

[mateusfccp]

@Reprevise The best code from the three you showed was the first.

For the second one, we can already do it without shadow:

int? i;
void f() {
  final i = this.i;
  if (i == null) return;

  print(i.isEven);
  print(i / 2);
  print(i / 4);
}

There's no advantage to shadow if you are not going to reassign i, as in your example.

[Reprevise]

@mateusfccp I am well aware that it's possible without shadow as it would just desugar to a non-final form anyway. My preference for 3 just comes from my dislike of over-nesting, especially when there are multiple if case statements.

[esDotDev]

Yup:

int? i;
void f() {
  if (i case var i?) {
    print(i.isEven);
  }
}

Now that patterns are out, I find myself using this idiom quite often.

I'm confused how this is checking for non-null.

This makes sense case int i but case var i? doesn't read to me as a non-null pattern match.

Is there anything in the docs that discusses this sort of usage?

[Edit] nm, I found it https://dart.dev/language/pattern-types#null-check

[esDotDev]

It's cool that you can utilize pattern matching and a dedicated null-check pattern to do this, but it really feels like a step backwards in readability. It's not semantically intuitive and additionally will be quite hard to developers to lookup implementation details on ? as it is becoming so overloaded.

When you compare it to the dedicated keyword there is no comparison in terms of readability:

if(shadow i is int){
  return i.isEven;
}

It's semantically intuitive and also very easy for a developer to lookup what shadow does.

[lrhn]

But if we had local variable declarations, that could also be:

if ((var i = this.i) != null) {
  // Use local i
}

That can be used for many more things, where shadow is fairly limited to solving a single problem.

It won't do write-through like shadow, but that really feels like a too surprising and too subtle behavior IMO.