Sealing or watching List objects, or copy-on-write List objects

[Hixie]

In Flutter, we pass lists around but it is a convention that once you have passed a list to an object, you no longer "own" the list and are not allowed to modify it.

Unfortunately, nothing in the language, API, or tooling actually support this, and so it's very easy for our customers to think they are keeping ownership of the list and for them to then run into trouble when either the changes have no effect, or the changes unexpectedly have an effect, depending on what they were trying to do.

---

Possible ways we could address this:

• having a way to "seal" a List object and thus prevent people who have references to it from ever modifying the list again.
• having a way to be notified when a List object created from a literal is changed, so that we could then throw with a message that says "you can't modify the list"
• have a way to very cheaply "copy" a List object created from a literal, using techniques such as "copy on write", so that we can say that later modifying the list is fine but doesn't affect the widgets that used the original copy. (Ideally such a solution would be compatible with passing const lists to const constructors but having those same constructors usable with new and getting this copy-on-write behaviour.)

[zoechi]

Can't this be solved with https://api.dartlang.org/stable/1.20.1/dart-collection/UnmodifiableListView-class.html ?

[lrhn]

I'm not sure I entirely understand the problem setting.

If you have a list that you can modify, and you pass it to someone else, and then they can modify it, but you can't, then it can't be the same object. You need some kind of protocol to ensure that, and it can't just be using plain modifiable lists that you store and pass around. That is, you can't talk about an "owner" of an object as anything but who has a reference to it. If you need to switch owners, you need more than one object, and you need to create the new object at the correct time.

We don't have a way to "seal" any List, especially the standard list, and I don't think we want it. Likewise, we don't want any notification on normal lists (or objects in general). That kind of overhead you have to opt in to.

For copy-on-write, you still need a new instance for each domain, so you need to wrap at the pass-over point, and you probably want to wrap it immediately the list is created, otherwise you can still modify the original without anybody knowing. You need complete life-time control of the objects in question.

You can easily make a SealableList that wraps a list and adds a seal() method. Then you can modify until you are ready and then seal it. Then nobody can change the list again.

class SealableList<E> extends DelegatingList<E> {
  bool _sealed = false;
  SealableList() : super(<E>[]);
  void checkSealed { if (_sealed) throw new StateError("List has been sealed"); }
  void seal() { _sealed = true; }
  E oprator[](int index) => super[index];
  void operator[]=(int index, E value) {
    _checkSealed();
    super[index] = value;
  }
  // etc.
}

That gives you the option of creating a list that can be sealed, but it's opt-in. It's not something you can do to somebody else's list - if they have a reference to a normal mutable list, nothing can take that away from them again.

Ditto for a notify/intercept-on-change list, and a copy-on-write list, e.g.:

class _CopyOnWriteListShare {
  // Number of other CopyOnWriteList instances sharing the same base list.
  int count = 0;
}

class CopyOnWriteList<E> extends ListBase<E> {
  List<E> _base;
  _CopyOnWriteListShare _share = null;
  CopyOnWriteList() : _base = <E>[];
  CopyOnWriteList.from(List<E> elements) : _base = elements.toList();
  CopyOnWriteList._(this._base, this._share);
  List<E> toList({growable: true}) {
    if (!growable) return _base.toList(growable: false);
    if (_share == null) _share = new _CopyOnWriteListShare();
    _share.count++;
    return new CopyOnWriteList<E>._(_base, _share);
  }
  void _copyOnWrite() {
    if (_share != null) _unshare();
  }

  void _unshare() {
    if (_share.count > 0) {
      var copy = _base.toList();
      _share.count--;
      _base = copy;
    }
    _share = null;
  }
  E operator[](int index) => _base[index];
  void operator[]=(int index, E value) {
    _copyOnWrite();
    _base[index] = value;
  }
  // etc.
}

[Hixie]

Flutter's framework wants you to write code like this:

        ...
        new Row(
          children: <Widget>[
            new Flexible(child: new Text('Everything is awesome')),
            new Checkbox(
              value: config.configuration.stockMode == StockMode.optimistic,
              onChanged: (bool value) => _confirmOptimismChange(),
            ),
          ],
        ),
        ...

The Row object uses the list passed into its children argument directly. We can't afford to copy it.

This means people can instead do things like this:

         List<Widget> list = <Widget>[
            new Flexible(child: new Text('Everything is awesome')),
            new Checkbox(
              value: config.configuration.stockMode == StockMode.optimistic,
              onChanged: (bool value) => _confirmOptimismChange(),
            ),
          ];
        ...
        new Row(
          children: list,
        ),
        ...
        list.add(new Text('Hello?'));

...which will cause all kinds of problems.

We'd like to either make it so it doesn't cause these problems (e.g. copy-on-write lists), or make it so we can catch these cases either at analyzer time, compile time, or runtime (e.g. sealing lists), to help authors avoid them. We have gotten confused questions from customers who end up in situations like this and don't understand why it's not doing what they want.

[Hixie]

(Our requirements are that it not make using the framework any more awkward syntactically or any slower in release builds.)

[lrhn]

I see the problem. It's not a problem that can be solved easily. Adding features that allow you to seal an existing object that someone else has created isn't really feasible - list or otherwise (and a feature that only works for List seems like a kludge).

In general, if someone passes you an object, you don't get to decide anything about that object. Either you make your own copy of it, or you live with what the creator of the object has chosen for you.

If you have a policy that says that you shouldn't change the list, your clients should follow that, just as they should follow all the other policies (like not implementing the Flexible interface in a way that's incompatible with general Widget behavior).

One thing you could do is to remember the original length of the list, and check occasionally whether it has changed. That would at least catch the "add extra element to list" bug, but won't catch if someone replaces an element.

I can see that copying everything is an overhead, especially if there are many small lists. Copying a list shouldn't be that slow. If you copy to a non-growable list, it should be one new-space allocation, two writes (header and length) and a mem-copy of the contents of the source list, but all copying costs, and all allocation increases memory churn, even one as simple as this.

All in all, I don't see the requests here as viable. Especially if you don't want anything to be slower.

Now, if we had positional rest-parameters, your clients could pass parameters without creating a list, and you would receive them as a list that nobody else has a reference to. That might solve the problem without giving anyone a way to mess with other people's Lists.

Or a way to transfer elements from one list to another, leaving the original list empty. But that only works for growable lists - you can't empty a fixed-length list, so you would need to handle a fixed-length list differently.

[Hixie]

This doesn't have to be solved with something that affects release-mode runtime semantics. For example, we could have an API that is only enabled in checked mode, like "debugSeal()", which sets a flag which add/remove/etc assert is false. (One could imagine even that you would pass a string to debugSeal and the assertion would use that string if it throws, to make debugging this viable.)

If you have a policy that says that you shouldn't change the list, your clients should follow that, just as they should follow all the other policies

Well, yes, but the problem is how to help them when they don't realise that that is the policy.

[Hixie]

To give an example of how this can just be a debug-mode or analyzer-level feature:

like not implementing the Flexible interface in a way that's incompatible with general Widget behavior

We plan on addressing that particular issue with an analyzer lint.

[Hixie]

Here's an example of a customer being tripped up by this: https://github.com/flutter/flutter/issues/8952

[zoechi]

@Hixie did you consider https://github.com/google/built_value.dart and https://github.com/google/built_collection.dart (I assume you did - just to be sure ;-) )

[Hixie]

@zoechi I don't see how those help. The class we have to use is whatever [] literals give us, and we want to avoid any copies.

[lrhn]

I can see only one solution that is at all plausible, and that is to "lazy-clone" a list with copy-on-write semantics. That would allow the original owner of the list to keep a reference to it and modify it, without it costing you anything if they don't do it. It does mean that each List must be at least two objects. It already is for growable lists, but fixed-length and const lists are just single objects. Const lists are "easy" because you can't write anyway, so you can give the same reference to all users, and "lazyClone" would just return itself. Fixed length lists has to fail cloning, which is annoying.

Also, this is just VM semantics, dart2js should also support the operation, and there is no good way to share list structures when you use native JS lists. So, not really viable as a JS strategy.

So, maybe, just maybe, it could be implemented as an optimization strategy for the VM's implementation of List.toList(). It only triggers when you use a growable list as base and result, and it would just replace the class pointer for the list objects with a CopyOnWrite class.

It is not something we can handle at the library level, and probably not even at the language level, so I'm reassigning this as a VM feature.

[rakudrama]

This problem could be detected by static analysis tools, specifically typestate analysis, although this is more advanced than the tools we currently have which are very lint-like.

A typestate analysis could track the modifiablility of the list, assigning the state 'unmodifiable' from the point of being passed to the constructor, validating that the list does not flow to operations that would modify it. In effect, this is simulating the 'read-only' bit in the analysis. The analysis also has to track backwards to see if there are other access paths to the list, since all pointers to the list become unmodifiable. Imprecise aliasing (both forwards and backwards) is a source of false positives (spurious reports) in such an analysis, but often, like https://github.com/flutter/flutter/issues/8952, the aliasing relationship is precise. If you are happy detecting most, or just many, design pattern violations, the noise from false positives can be tuned out.

[Hixie]

That would be wonderful.

[leafpetersen]

Do you actually ever need the list to be modifiable in the first place (and only subsequently sealed)? In the code snippet you gave, it seems like a literal form for an unmodifiable (from creation) list would suffice?

[Hixie]

We commonly build the list incrementally, e.g.: https://github.com/flutter/flutter/blob/master/packages/flutter/lib/src/material/scaffold.dart#L235 https://github.com/flutter/flutter/blob/master/packages/flutter/lib/src/material/app_bar.dart#L349

[eseidelGoogle]

We have another customer get confused by the gotcha in Flutter for which this is a possible workaround just now (modifying a list, repeatedly passing the same list to a new Widget, and being confused that his UI didn't change).

[Hixie]

cc @floitschG this is the bug we were talking about the other day

[rakudrama]

We are spending a lot of energy to avoid copying one small list. So, why not copy the list inside the framework? This seems to be what the users expect, and no doing so is causing them pain.

The copy can be unmodifiable, which will be approx half the size (one object instead of two, no internal fragmentation). If you have lots of such lists, it will be beneficial to have the more compact copy and have larger growable lists GC-ed.

Is copying really too slow? Most of the ideas are adding various kinds of code to the system which is not free, and might end up being more expensive than copying the list.

If copying the list is too slow, find out why. The VM could have special knowledge of new List<Widget>.unmodifiable(list) in order to make the copying as fast as possible, possibly using hand-written code for the common case of copying from standard corelib lists.

[Hixie]

It's not one small list, it's hundreds of lists, every 16ms. We have a budget of about 8ms for the entire frame. We already frequently go over budget. We really can't afford to be doing any extra work that we don't need to be doing, especially memory copies. It would be much cheaper to not do any work in release mode at all, and do a couple of extra branches of work in debug mode.

[rakudrama]

It is not clear that avoiding a copy is necessarily the fastest way. You might avoid extra work by copying.

Consider an app that uses

children: <Widget>[new ...]

and

var list = new List<Widget>(5);
...
children: list

and

children: const <Widget>[]   // perhaps via a default value

These three lists are different types. The code that iterates the children will slower because each access now is an if-then-else on the type of the list. If the code walking the children list is one piece of code that is used by all widgets in the app, each additional list type it could slow down the whole app. By copying you could force a single type and avoid this if-then-else tax (except in the copying :-). If you create the lists once and walk them many times you might come out ahead by copying.

You can hunt for this kind of problem (multiple kinds of list) in observatory.

You alluded to doing more in debug mode. You could in debug mode copy the list to a shadow _children list and periodically check that the contents are identical to the copy.

[leafpetersen]

I'm also pretty skeptical about this being an overall win. Unless it's purely a debug option, you're trading off making everything everywhere a little bit slower against a possible win at one point. And if it's purely debug, then doing the defensive checks in the framework seems at least worth exploring.

I don't know what parameters the VM uses for allocating growable lists, but it is pretty much inevitably either over-allocating them for your use case, or else already copying when it grows them. Either way, I have to wonder whether you wouldn't be better served both performance and correctness wise to have the APIs take an immutable list. The copy that happens when you cross the API and you did in fact start with a mutable list has a cost, but you will be shrinking the object in the process (getting rid of over-allocated space and moving to a more compact representation) which means that the GC will do less copying. And if it is a reasonably common pattern to pass in a list literal, then the intermediate literal could/should be optimized away, and the immutable form generated in place.

Some concrete data here might be useful.

[Hixie]

We'd love the API to take only immutable lists. Unfortunately the common case is passing in a non-const list literal, and those are mutable.

[Hixie]

@floitschG @munificent @leafpetersen Friendly ping. We're finding a lot of new users end up tripping over this. Just being able to seal lists in debug mode and assert would be super helpful (especially if we could provide a custom message for the assert!). I'd be happy to try to do this entirely on the framework side but I don't really know how to do that.

[matanlurey]

final bool _assertionsEnabled = (){
  var enabled = false;
  assert(enabled = true);
  return enabled;
})();

List<T> debugSeal<T>(List<T> list) {
  if (_assertionsEnabled) {
    // Could replace with 'FlutterUnmodifiableList' if you want a custom message.
    return new List<T>.unmodifiable(list);
  }
  return list;
}

void main() {
  // List in prod, UnmodifiableList in debug.
  var numbers = debugSeal([1, 2, 3]);
}

[Hixie]

I don't understand. Where would that code go?

[matanlurey]

Ostensibly where you want to create a frozen list?

If you are asking how to watch a modifiable list to make sure it is not modified, then that is a hairy story. Angular basically does this, but using zones, and is a source of a lot of frustration (and perf issues). You can see a small example here.

[Hixie]

Suppose you expose a class Foo that takes a list in its constructor.

class Foo {
  Foo(this.list);
  final List list;
}

The following is the user code (which we can't change -- if we could change it, we could just remove the bug!):

void main() {
  List list = [1, 2, 3];
  Foo foo = new Foo(list);
  list.add(4); // this should throw with a custom message if possible
}

How would you change Foo to make the list immutable or to otherwise catch the case of the user changing the list?

[matanlurey]

So Angular has a debug mode check where it checks the entire tree one more time, and asserts that the identity of the object has not changed, with an exception for List or Map or Iterable, where we assert the inside hasn't changed. You could do something similar.

(This isn't language specific, so if you're interested in learning more ping me)

[Hixie]

The problem is that there's no way to associate the list that got mutated with the place that mutated the list, as far as I can tell, so any message we could output when we detect that a list that didn't change identity nonetheless changed contents would be very hard to do anything with, which might frustrate our users more than the current behaviour.

(I'm also concerned that such a check would be prohibitively expensive even in debug mode. We have some O(N^2) asserts in there and they are proving very expensive already.)

[lrhn]

On Thu, Jun 15, 2017 at 11:42 PM, Ian Hickson notifications@github.com wrote:

Suppose you expose a class Foo that takes a list in its constructor.

class Foo { Foo(this.list); final List list; void run() { } }

The following is the user code (which we can't change -- if we could change it, we could just remove the bug!):

void main() { List list = [1, 2, 3]; Foo foo = new Foo(list); foo.run(); list.add(4); // this should throw with a custom message if possible }

How would you change Foo to make the list immutable or to otherwise catch the case of the user changing the list?

You cannot make a list immutable. You can potentially make a new list from the mutable list and make the new list immutable. We have a function that does this without copying, but it's not exposed because it cannot be implemented in a consistent way across Dart2JS and the VM (it's destructive and acts differently on the original list, and it only works for the built-in growable list an not for, say, a fixed-length list, which is still mutable).

What you can do is to wrap the incoming list in debug mode. You can't prevent modification, but you can detect it:

class DontModifyList extends DelegatingList { final List _original; DontModifyList(List original) : _original = original.toList(), super(original); void _validate() { if (super.length != _original.length) throw StateError("Original List Modified!"); for (int i = 0; i < super.length; i++) if (!identical(_original[i], super[i])) throw new StateError(...); } // on every method, or every method that you care about: T operator[](int i) { _validate(); return super[i]; } T get length { _validate; return super.length; } ... }

This is obviously costly, both in space and computation, but the only way to detect whether something you don't control has changed is to keep something to compare it to.

If you worry about the size, you can keep a hashCode instead.

class DontModifyList extends DelegatingList { int _hash; DontModifyList(List original) : super(original) { _hash = _computeHash(); } void _validate() { if (_hash != _computeHash()) throw StateError("Original List Modified!"); } int _computeHash() { int hash = Hash.initialValue; /// Some hash implementation that is sufficiently efficient. hash = Hash.combine(super.length); for (int i = 0; i < this.length; i++) { hash = Hash.combine(super[i].hashCode); } return Hash.finalize(hash); } // on every method, or every method that you care about: T operator[](int i) { _validate(); return super[i]; } T get length { _validate; return super.length; } ... }

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dart-lang/sdk/issues/27755#issuecomment-308873956, or mute the thread https://github.com/notifications/unsubscribe-auth/AEo9B5jllBo-DXcU5eicOI0QQ-V3niemks5sEaVXgaJpZM4KpCoj .

-- Lasse R.H. Nielsen - lrn@google.com 'Faith without judgement merely degrades the spirit divine' Google Denmark ApS - Frederiksborggade 20B, 1 sal - 1360 København K

• Denmark - CVR nr. 28 86 69 84

[Hixie]

A new List-like class doesn't solve the problem either since most call sites will be using a list literal -- see the example above.

This is why this bug exists -- because it's not clear that there is a solution short of either adding a debug-mode assert-based API that seals lists, or something similar in the VM or core API.

[lrhn]

On Fri, Jun 16, 2017 at 8:06 AM, Ian Hickson notifications@github.com wrote:

A new List-like class doesn't solve the problem either since most call sites will be using a list literal -- see the example above.

I know. You can't solve the problem of someone creating a list literal and changing it later. The most you can do is to detect it or create a copy that the caller can't change. We do not plan to make it possible to freeze the general growable list that is created by literals. It's a very specific operation that doesn't fit on the List interface in general, and it is not a feature that we can expose safely and efficiently in all cases.

We could make it possible to create unmodifiable list literals, by adding a syntax for that, but that would not change the fact that users might not use that syntax (it will be longer than the plain list literal syntax, unless we add new delimiters or get very creative with parsing parentheses - heck, I would actually like (1, 2, 3) as an iterable literal if it wasn't so ambiguous).

So, my suggestion for an actual solution is (still) for the VM to optimize the List.unmodifiable constructor so that if it gets the system _GrowableList class, it creates a new unmodifiable list that shares the list content, and changes the original list to do a copy on write. That is, an implementation change, not a library or language change, but one you can depend on when using the VM.

I wouldn't expect the VM team to read this bug, so I suggest filing a separate issue to request such a feature.

This is why this bug exists -- because it's not clear that there is a solution short of either adding a debug-mode assert-based API that seals lists, or something similar in the VM or core API.

True. That's why I was suggesting the debug-mode wrapper that detects modifications. It won't seal the list, because you can't seal the original list anyway, but it cause the user's program to fail in debug mode if they do change the list, which will at least alert them to the problem.

That's not a solution, just a tripwire for the problem.

/L -- Lasse R.H. Nielsen - lrn@google.com 'Faith without judgement merely degrades the spirit divine' Google Denmark ApS - Frederiksborggade 20B, 1 sal - 1360 København K

• Denmark - CVR nr. 28 86 69 84

[Hixie]

You can't solve the problem of someone creating a list literal and changing it later.

Well, I mean, you can. You can do copy-on-write semantics for lists, or you can do list sealing, either as an API feature or as a debug-only feature. These aren't novel ideas, other languages have done them. :-)

Copy-on-write semantics aren't ideal because while they prevent the error to some extent, they don't guide you to the solution (they just make the list mutations have no effect at all, rather than giving you a message).

I'm not a big fan of "magic" solutions unless they are guaranteed. Relying on a particular constructor happening to have the right semantics most of the time seems like a recipe for disaster. What if the developer generated their list via Iterable.toList? What if it's not growable? What if it's a const list, or a new List list, or any number of other possible cases?

I don't fully understand what you're proposing though. If you think that's what we should do, could you file the VM bug for me describing what you think the VM should do?

That's not a solution, just a tripwire for the problem.

A tripwire would be completely satisfactory for us. We just need a way to guide our developers to the right model. There's lots of places in our API where you can shoot yourself in the foot but we detect the gun and flag the issue, either in a lint, or a debug-only assert, or some such, and we've had much success with this approach.

[lrhn]

On Fri, Jun 16, 2017 at 9:29 AM, Ian Hickson notifications@github.com wrote:

You can't solve the problem of someone creating a list literal and changing it later.

Well, I mean, you can. You can do copy-on-write semantics for lists, or you can do list sealing, either as an API feature or as a debug-only feature. These aren't novel ideas, other languages have done them. :-)

I don't want sealing a list to be a debug-only feature. Would that mean i could only be used inside an assert? The typing would be ... interesting.

Having copy-on-write semantics for lists in general is not an option. It wouldn't work well when compiled to Javascript. The VM could do it, it just had to decide which operations introduce the copy-on-write effect - you have to create a new instance from the old one in some way, which is why I suggested List.unmodifiable - and it could even avoid creating a new list entirely if the input is already unmodifiable. It could work with List.from as well, if you want both versions to be modifiable. Still, it's a performance optimization only, so the VM can do that any time they want to, and dart2js can't really (they use JS Array objects directly for lists, with no wrapper, so there is no good way to share a backing store).

Copy-on-write semantics aren't ideal because while they prevent the error to some extent, they don't guide you to the solution (they just make the list mutations have no effect at all, rather than giving you a message).

I'm not a big fan of "magic" solutions unless they are guaranteed. Relying on a particular constructor happening to have the right semantics most of the time seems like a recipe for disaster. What if the developer generated their list via Iterable.toList? What if it's not growable? What if it's a const list, or a new List list, or any number of other possible cases?

Or what if it's an instance of MyCleverList that you have never heard of (say an ArrayList implementation like the on in Java, which has different performance characteristics to the Dart growable list)?

Then there is nothing you can do. Nobody can, no code will be able to affect that class in a way that it doesn't support itself. If we add an toUnmodifiableList() or an unmodifiable flag to the toList method, then each class can decide for itself how to make itself unmodifiable.

It seems that growable literal lists is the common issue here, but if you need to fix all ways to send a modifiable list to efficiently prevent modification, there won't be a way to do that.

I don't fully understand what you're proposing though. If you think that's what we should do, could you file the VM bug for me describing what you think the VM should do?

That's not a solution, just a tripwire for the problem.

A tripwire would be completely satisfactory for us. We just need a way to guide our developers to the right model. There's lots of places in our API where you can shoot yourself in the foot but we detect the gun and flag the issue, either in a lint, or a debug-only assert, or some such, and we've had much success with this approach.

In that case, I think that is what you should do.

In debug mode, wrap the incoming list in a way that occasionally checks whether the list has changed. If it changed, throw and let the user know that their program is violating assumptions. In non-debug mode you just use the (hopefully) unchanged list.

/L -- Lasse R.H. Nielsen - lrn@google.com 'Faith without judgement merely degrades the spirit divine' Google Denmark ApS - Frederiksborggade 20B, 1 sal - 1360 København K

• Denmark - CVR nr. 28 86 69 84

[davidmorgan]

https://github.com/google/built_collection.dart provides immutable collection types.

They are a perfect fit for collections in "business objects" -- which 1) benefit the most from immutability, and 2) can afford the slight performance penalties. (Or may even get performance gains because of not having to copy defensively).

The library also has copy-on-write wrappers for collections, but doesn't expose them. They are used to provide "toList" "toSet" and "toMap" methods that don't copy unless they have to.

[Hixie]

I don't want sealing a list to be a debug-only feature. Would that mean it could only be used inside an assert?

We have lots of debug-only features in the Flutter framework. It means they do something when asserts are enabled and do nothing otherwise.

Or what if it's an instance of MyCleverList that you have never heard of (say an ArrayList implementation like the on in Java, which has different performance characteristics to the Dart growable list)?

Well either it would also implement the debug-only feature, or it wouldn't. In the former case, great. In the latter case, it probably doesn't matter, because someone clever enough to be using custom lists in their build methods has probably first experimented with regular lists and so has already learnt about this particular footgun and doesn't need our guardrails.

In debug mode, wrap the incoming list in a way that occasionally checks whether the list has changed. If it changed, throw and let the user know that their program is violating assumptions.

Can you show me what this would look like? Here's the code you can change:

class Foo {
  Foo(this.list);
  final List list;
}

void remember(Foo foo) {
  // This function will asynchronously use foo.list multiple times in the future
  // and depends on its contents not changing.
}

This is the code that can't change:

void main() {
  List list = [1, 2, 3];
  Foo foo = new Foo(list);
  remember(foo);
  list.add(4); // this should throw with a custom message if possible
}

What changes can we make to the first block that would result in the problem being flagged?

[matanlurey]

@hixie: This is going to be potentially expensive, but you could do the following:

final _firstCheck = new Expando<List>();

// TODO: Implement better. Could use ListEquality from pkg/collection.
bool _didChange(List a, List b) => a.length != b.length;

void assertNoChanges<T>(List<T> list) {
  assert(() {
    var firstCheck = _firstCheck[list];
    if (firstCheck == null) {
      _firstCheck[list] = list.toList();
      return true;
    }
    if (_didChange(firstCheck, list)) {
      throw 'Flutter contract error: List was mutated after passed...';
    }
    return true;
  });
}

And for usage:

void remember(Foo foo) {
  assertNoChanges(foo.list);
  // Every-time you use foo.list, call assertNoChanges(foo.list) first.
  // Alternative you could make a method that asserts and returns, i.e.
  // List<T> safeList(List<T> list) {
  //   assertNoChanges(list);
  //   return list;
  // }
}

[Hixie]

We could do that. See https://github.com/dart-lang/sdk/issues/27755#issuecomment-308875664 for why we don't. :-)

[munificent]

void main() {
  List list = [1, 2, 3];
  Foo foo = new Foo(list);
  remember(foo);
  list.add(4); // this should throw with a custom message if possible
}

It's not clear to me that training users that code like this explodes at runtime is a good thing. Dart does not have an ownership model and one of the benefits it provides to users is the simplicity of not having to think about ownership. But in this code, you are adding a new axis of complexity to working with objects. With some APIs if you pass an object — here a list — then all of a sudden, certain methods on the list are now verboten. Worse, the failure is only detected at runtime.

That doesn't seem like an easy to use system to me.

For example:

writeList(List items) {
  print(items.join(" ");
}

void main() {
  var list = [1, 2, 3];
  writeList(list); // "1 2 3"
  list.add(4);
  writeList(list); // "1 2 3 4"
}

This code is fine, and does what the user (presumably) wants. With freezing, we have to train users that some code like this is OK and other code like it is not. For objects as fundamental as lists, that seems like a lot of cognitive overhead.

This is probably too big of a change to the API, but if you want more control over the kinds of collections your widgets store, maybe it would be better to control that explicitly instead of letting users create them on their own. Instead of:

        ...
        new Row(
          children: <Widget>[
            new Flexible(child: new Text('Everything is awesome')),
            new Checkbox(
              value: config.configuration.stockMode == StockMode.optimistic,
              onChanged: (bool value) => _confirmOptimismChange(),
            ),
          ],
        ),
        ...

More like:

        ...
        new Row().children
          ..add(new Flexible(child: new Text('Everything is awesome')))
          ..add(new Checkbox(
            value: config.configuration.stockMode == StockMode.optimistic,
            onChanged: (bool value) => _confirmOptimismChange(),
          ))
        ...

(I realize the cascade syntax is kind of gross, but you get the idea.)

[Hixie]

Widgets are meant to be immutable, so that doesn't really work.

Flutter does have an ownership model, in that once you pass a child list to a widget, that object now owns the list. The problem is that we have no way to catch contract violations.

[munificent]

Widgets are meant to be immutable, so that doesn't really work.

You could freeze the widget once it's passed to Flutter, right? It's not clear to me what's different between:

var list = [];
list.add(new Flexible());
list.add(new Checkbox());
return new Row(children: list);

And:

var row = new Row();
row.add(new Flexible());
row.add(new Checkbox());
return row;

In both cases, you have a set of objects that are mutable for a while and then locked down, right?

[matanlurey]

I'll admit I haven't used every language, but is there an example of a mainstream one where it is trivial to implement the features you want @Hixie? I'm asking because as much as I agree this looks like a pain point I can't even imagine how it is feasible.

[Hixie]

Several languages have copy-on-write arrays, e.g. PHP or ObjectPascal. These would solve the problem, albeit not with the most help to the developer.

Marking a list as immutable is something Dart already supports (const arrays throw when you try to mutate them). Having a flag you can set in debug mode which each mutating accessor of the class used for literals and for new List asserts is not set seems trivial to add.

In general having a class that can be flipped into "read-only" mode is not particularly new. I don't really understand why this bug is controversial or difficult to implement.

[Hixie]

@munificent Yes, we could do this by not using list literals — we wouldn't even need to make a change as drastic as you propose, we could just require that everyone use FlutterLists instead of Lists, and make FlutterList do what I'm describing here (it would be pretty trivial to implement). The problem is that this doesn't satisfy the requirements described above, namely, that this works with list literals. It would also be significantly more expensive, and would preclude using const objects.

[munificent]

Marking a list as immutable is something Dart already supports (const arrays throw when you try to mutate them).

That's a little different. Const lists are always immutable. What you propose is a list that is mutable for a while — and you specifically want to allow the user to mutate it — and then it later loses that mutability in a way that's only detectable at runtime.

[Hixie]

There's a TODO in the list code that points to a way this could be implemented: https://github.com/dart-lang/sdk/blob/master/runtime/lib/array.dart#L112

[lrhn]

I have no doubt that the feature can be implemented. We already do something like it internally in closely controlled sitiuations. It is still not something that we want the platform to expose as a user-facing feature, for a number of reasons. We don't want the List interface to have a freeze method. That's just not an operation that makes sense in general, and not something we want everybody to implement for their list implementations. It's also too fragile a design if anyone can freeze a list that someone else is using. That fragility hurst the most when it affects the most used type of list, so even freezing the default growable list is almost as bad as being able to freeze all lists.

We have no plans to make lists from list literals freezable. Being freezable must be opt-in for the code that creates the list.

That leaves some alternatives. We could add syntax for an unmodifiable list literal that is not const. Users would likely have to write slightly more to get it, but it would work in general. Or you could create an unmodifiable list from the existing list (List.unmodifiable), and there we can specialize it to an efficient implementation (no-copy + copy-on-write) for the default growable list in the VM, at least if the VM engineers thinks its a good idea that won't hurt performance of all lists.

Those are the options I see if you want to ensure that "your" list is not modified by whoever created it as a modifiable list and kept a reference to it.

[davidmorgan]

+1 for umodifiable list literal -- but only if the receiver can determine whether the list is unmodifiable. Otherwise, you still need a convention, or to defensively copy.

[yjbanov]

(update: added unique_ptr and std:move for completeness)

I've seen the opposite of this problem too, when a widget takes a list and then mutates it internally (inside the constructor or the build method) without realizing it's mutating a potentially shared object.

@matanlurey

is there an example of a mainstream [language] where it is trivial to implement the features you want [...]?

Rust C++ const reference parameters, unique_ptr, std:move

Both implementations are type system based and therefore have zero cost. I like the Rust approach more, because you pay the complexity of understanding ownership once, but it is useful in many domains, including efficient memory management, sharing data across objects, sharing data across threads. The part that I'm not a big fan of in Rust is that ownership is in-your-face even for hello-world size programs. I'm suspecting this is because Rust doesn't have a GC and that's one way to prevent use-after-free kinds of errors. I wonder if Dart, being a managed language, can have something an order of magnitude simpler but as powerful.

@munificent

Dart does not have an ownership model and one of the benefits it provides to users is the simplicity of not having to think about ownership.

I consider ownership a framework/library concern more than a language concern. For example, the fact that List throws ConcurrentModificationError while being iterated means that its iterator "owns" the list (at least the list's mutators), except that this ownership is enforced as a runtime check rather than using the type system. I'm not sure that having to deal with runtime errors (potentially in production) is any simpler than dealing with a static compiler warning. Runtime checks are also less efficient and every framework/library has to invent its own. So, whether the language has ownership built-in or not, the library/framework will add this complexity regardless and users will pay for it (already do in Flutter's case).

I think the important question is whether the language should assist frameworks and libraries in expressing ownership and users understand it. My opinion is yes, it should. I have a feeling this will come up again when we start talking about concurrency. Perhaps Dart can have something much simpler than what Rust does?

[lrhn]

I've seen the opposite of this problem too, when a widget takes a list and then mutates it internally (inside the constructor or the build method) without realizing it's mutating a potentially shared object.

@matanlurey https://github.com/matanlurey

is there an example of a mainstream [language] where it is trivial to implement the features you want [...]?

Rust https://doc.rust-lang.org/book/second-edition/ch04-01-what-is-ownership.html C++ const reference parameters https://stackoverflow.com/questions/2627166/difference-between-const-reference-and-normal-parameter

I think C++ const does the opposite of what you want – it prevents the receiver from modifying the object, not the sender. You can pass a mutable array as a const reference parameter, it just means that the receiver can't change it, not that the original owner can't.

That is, it does prevent the opposite problem that you mentioned above, but not the original problem of this thread.