Collection<T> and ObservableCollection<T> do not support ranges

[robertmclaws]

Update 10/04/2018

@ianhays and I discussed this and we agree to add this 6 APIs for now:

    // Adds a range to the end of the collection.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void AddRange(IEnumerable<T> collection) => InsertItemsRange(0, collection);

    // Inserts a range
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void InsertRange(int index, IEnumerable<T> collection) => InsertItemsRange(index, collection);

    // Removes a range.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Remove)
    public void RemoveRange(int index, int count) => RemoveItemsRange(index, count);

    // Will allow to replace a range with fewer, equal, or more items.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Replace)
    public void ReplaceRange(int index, int count, IEnumerable<T> collection)
    {
         RemoveItemsRange(index, count);
         InsertItemsRange(index, collection);
    }

    #region virtual methods
    protected virtual void InsertItemsRange(int index, IEnumerable<T> collection);
    protected virtual void RemoveItemsRange(int index, int count);
    #endregion

As those are the most commonly used across collection types and the Predicate ones can be achieved through Linq and seem like edge cases.

To answer @terrajobst questions:

Should the methods be virtual? If no, why not? If yes, how does eventing work and how do derived types work?

Yes, we would like to introduce 2 protected virtual methods to stick with the current pattern that we follow with other Insert/Remove apis to give people hability to add their custom removals (like filtering items on a certain condition).

Should some of these methods be pushed down to Collection?

Yes, and then ObservableCollection could just call the base implementation and then trigger the necessary events.

Let's keep the final speclet at the top for easier search

Speclet (Updated 9/23/2016)

Scope

Modernize Collection<T> and ObservableCollection<T> by allowing them to handle operations against multiple items simultaneously.

Rationale

The ObservableCollection is a critical collection when it comes to XAML-based development, though it can also be useful when building API client libraries as well. Because it implements INotifyPropertyChanged and INotifyCollectionChanged, nearly every XAML app in existence uses some form of this collection to bind a set of objects against UI.

However, this class has some shortcomings. Namely, it cannot currently handle adding or removing multiple objects in a single call. Because of that, it also cannot manipulate the collection in such a way that the PropertyChanged events are raised at the very end of the operation.

Consider the following situation:

• You have a XAML app that accesses an API.
• That API call returns 25 objects that need to be bound to the UI.
• In order to get the data displayed into the UI, you likely have to cycle through the results, and add them one at a time to the ObservableCollection.
• This has the side-effect of firing the CollectionChanged event 25 times. If you are also using that event to do other processing on incoming items, then those events are firing 25 times too. This can get very expensive, very quickly.
• Additionally, that event will have ChangedItems Lists that will only ever have 0 or 1 objects in them. That is... not ideal.

This behavior is unnecessary, especially considering that NotifyCollectionChangedEventArgs already has the components necessary to handle firing the event once for multiple items, but that capability is presently not being used at all.

Implementing this properly would allow for better performance in these types of apps, and would negate the need for the plethora of replacements out there (here, here, and here, for example).

Usage

Given the above scenario as an example, usage would look like this pseudocode:

    var observable = new ObservableCollection<SomeObject>();
    var client = new HttpClient();
    var result = client.GetStringAsync("http://someapi.com/someobject");
    var results = JsonConvert.DeserializeObject<SomeObject>(result);
    observable.AddRange(results);

Implementation

This is not the complete implementation, because other *Range functionality would need to be implemented as well. You can see the start of this work in PR dotnet/corefx#10751

    // Adds a range to the end of the collection.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void AddRange(IEnumerable<T> collection)

    // Inserts a range
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void InsertRange(int index, IEnumerable<T> collection);

    // Removes a range.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Remove)
    public void RemoveRange(int index, int count);

    // Will allow to replace a range with fewer, equal, or more items.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Replace)
    public void ReplaceRange(int index, int count, IEnumerable<T> collection);

    // Removes any item that matches the search criteria.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Remove)
    // RWM: Excluded for now, will see if possible to add back in after implementation and testing.
    // public int RemoveAll(Predicate<T> match);

Obstacles

Doing this properly, and having the methods intuitively named, could potentially have the side effect of breaking existing classes that inherit from ObservableCollection to solve this problem. A good way to test this would be to make the change, compile something like Template10 against this new assembly, and see if it breaks.

---

So the ObservableCollection is one of the cornerstones of software development, not just in Windows, but on the web. One issue that comes up constantly is that, while the OnCollectionChanged event has a structure and constructors that support signaling the change for multiple items being added, the ObservableCollection does not have a method to support this.

If you look at the web as an example, Knockout has a way to be able to add multiple items to the collection, but not signal the change until the very end. The ObservableCollection needs the same functionality, but does not have it.

If you look at other extension methods to solve this problem, like the one in Template10, they let you add multiple items, but do not solve the signaling problem. That's because the ObservableCollection.InsertItem() method overrides Collection.InsertItem(), and all of the other methods are private. So the only way to fix this properly is in the ObservableCollection itself.

I'm proposing an "AddRange" function that accepts an existing collection as input, optionally clears the collection before adding, and then throws the OnCollectionChanging event AFTER all the objects have been added. I have already implemented this in a PR dotnet/corefx#10751 so you can see what the implementation would look like.

I look forward to your feedback. Thanks!

[jnm2]

@weitzhandler Thomas is probably right. Also, I generally prefer to send a single event no matter what. Here's a mockup I did of a way to leave the event-combining decision in the hands of the user of the collection: https://gist.github.com/jnm2/d950053fd3825818371b87730f208839#file-eventbufferingcollection-cs Not sure how much of that is interesting to most people, though.

[weitzhandler]

@jnm2 commented on Jan 22, 2018 Thomas is probably right.

I understand it may be out of scope, but in the other hand, since the new functionality makes use of clusters (here and here), the need for this pattern might be important here. Let's let @karelz decide.

Also, I generally prefer to send a single event no matter what.

You're missing the point. The idea is to raise as less possible events as we can. So in case of a mixed action (for example replacing some and adding more), we want to split up the events to few groups and raise them separately, while notifying the property change (Count, Item[]) only once.

Here's a mockup I did of a way to leave the event-combining decision in the hands of the user.

1. I'm not sure we would want to give the user this decision.
2. I personally went for simplicity and minimalism, but let's let the fx team decide on that. Bear in mind that it's just been committed and it's still awaiting some API Review.

[jnm2]

You're missing the point. The idea is to raise as less possible events as we can. So in case of a mixed action (for example replacing some and adding more), we want to split up the events to few groups

I think I'm just taking that further by sending either zero or one collection change event; that's even fewer. The intentional tradeoff being, reset events are more likely. Sending a property change events for Count is not something I've ever found useful.

[weitzhandler]

I think I'm just taking that further by sending either zero or one collection change event

Imagine you have a collection containing alpha, bravo, charlie. You want to add a new collection bravo, delta, echo, foxtrot starting from 2nd position. Instead of raising 3 different type of events, it will ignore the equal item bravo, raise a Replace event for charlie, and raise one grouped Add event for echo and foxtrot. And so on (see tests). That's why we do need to separate out the events. Let us not forget that ObservableCollection was made for UI data-binding purposes, and this pattern of grouping is merely to preserve unnecessary UI thread calls and updates, I've been using it in my projects and it vastly improved the UI performance over ObservableRangeCollection offered by the Xamarin team, especially when using very frequent modifications to the collection (such as a filtered view updated upon user key stroke).

Sending a property change events for Count is not something I've ever found useful.

Count may useful in many instances. One of them is you want to style your view based on the count in the collection, or want to generate a new page in the index of the Count property and some more. And there is also the indexer (Item[]), which can come handy too.

[jnm2]

I get it, but coming from a LOB domain and UI perspective, I would almost always prefer a reset event in the scenario you give rather than multiple grouped events. (Or more accurately, a single replace event which replaces the entire minimally-scoped starting range with the entire minimally-scoped ending range.)

[weitzhandler]

I get it, but coming from a LOB domain and UI perspective

That's my top field, I mostly do LoB.

I would almost always prefer a reset event

You wouldn't prefer a Reset. Not if you measured its performance cost. Resets are worst. They immediately clear the UI and build it all anew - not what we want, we want to reuse as many items as we can so the UI doesn't refresh things we don't need to. Besides, partial Resets with index indication isn't supported, and also according to the documentation, Reset indicates the collection was cleared.

Or more accurately, a single replace event which replaces the entire minimally-scoped starting range with the entire minimally-scoped ending range.

That's what it does. What's out of the existing indices will be Added, not Replaced.

[jnm2]

Same.

UI perf measurements have shown that ten nonconsecutive adds are worse for performance than a single reset. Of course, replace rather than reset when it's possible to represent that with the API.

[weitzhandler]

UI perf measurements have shown that ten nonconsecutive adds are worse for performance than a single reset.

Not the results I've seen. Adds update the items one by one and the view doesn't flicker as it does with Resets. That what actually motivated me to divide the events up to clusters. Of course trying to group the events instead of raising them individually whenever possible if the items are consecutive.

[jnm2]

The UI controls I was working with did not flicker either way, but the lag due to a reset was similar to the lag caused by a couple of Add events. Full invalidation isn't necessary when you see that the property descriptors are the same and almost the row instances are the same.

[weitzhandler]

I was recently working on a Xamarin project and tried various scenarios. I used the ReplaceRange to update a drop down, and I experienced flickering upon resets, unlike grouped events. WPF is most likely much faster anyway. Besides, in my implementation, there shouldn't be several consecutive Add events they will all be combined to a single batched Add (for example).

[jnm2]

That kind of platform sensitivity is what makes me think the batching of events should be at least somewhat configurable. Or then again, maybe I should just continue to keep away from ObservableCollection. 😜

[weitzhandler]

The individual events in my implementation will only occur in non-consecutive replacements - not additions, which is good because the UI will only refresh the required indices.

[weitzhandler]

@karelz Can we push this issue into 2.1, this has been a long issue that is rather compelling in almost any UI and MVVM scenario, and improves performance drastically.

The API I proposed is no much other from the API that was already approved. It just added more flexibility and a few more essential options.

[karelz]

I don't think it is a good idea to rush this API. Moreover, technically speaking we are past API freeze. Given that the change and the new API shape didn't get enough scrutiny from community / experts tells me it is not as simple as it looks. If I remember correctly, even the original approval was a bit open-ended with "we need more implementation validation, before we are fully ok with the API". Once we ship the API, we will live with it forever. Therefore, we should be fairly sure it is right. I don't have the confidence from the discussion above that we are there yet.

[weitzhandler]

@karelz I understand. Thanks for your response tho. I hope we're gonna get there some time soon. But I'd like to request from you to link the 2nd round API from the original post.

[karelz]

The issue is marked api-ready-for-review. It is on our list. We have just been prioritizing review of APIs critical for 2.1. I hope we will be able to get to the Future backlog in 2-3 weeks.

[hutterm]

@karelz what's the status on this issue? @weitzhandler just a heads up, I was trying to use your implementation with a grouped CollectionViewSource and it seems that one doesn't pick up on range changes ( with a new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Remove, list, index) it only removes one element at index)

[weitzhandler]

Well I was developing against the existing tests. I shall try adding tests that verify that. I don't know if we will be able to add tests that make use of CollectionViewSource but I'll try tracing down the error.

[terrajobst]

The proposal sounds good, but we need to have a design that answers these questions:

• Should the methods be virtual? If no, why not? If yes, how does eventing work and how do derived types work?
• Should some of these methods be pushed down to Collection<T>?

We are concerned that derived classes that already override the existing virtual methods will end up getting not being notified for removals and/or they are getting the even raised for single items and the bulk operation.

[weitzhandler]

@terrajobst That's a valid concern indeed, I have been messing with it in the past, making the range-methods non-virtual, see this test, for example. But if we are to make them virtual, then it should be degraded into Collection<T>, it doesn't really make sense otherwise IMHO. @redoced It seems to me that CollectionViewSource isn't part of the .NET Core, can you please link the CollectionViewSource you're talking about and let us know how you plugged in the new ObservableCollection<T> with the range functionality?

@karelz all of these concerns were also valid when the suggestion was approved back ago without my small additions. Can we advance this issue? Very compelling for any XAML dev.

[karelz]

@weitzhandler I am not sure what you're asking for. If there are concerns about API shape or behavior, then they need to be addressed prior to finalizing & approving the API. If we overlooked some concerns earlier, it does not mean there is a free pass to ignore them now. It could (and did) happen that approved API gets rejected in PR, because new API concerns arise which were missed during previous API approval review. We are not slaves of process. The process is here to help us come up with solid APIs which developers will use for decades. We want to deliver high-quality APIs and ideally learn from our past experience to make APIs even better now. And yes, API design is HARD and sometimes TRICKY.

[karelz]

cc @danmosemsft @safern @ianhays to see if it would fit our 3.0 plans (with UI stacks coming to .NET Core) - the API had a share of back-and-forth and it could use some love from our team ...

[weitzhandler]

Hey @karelz and thanks for your kind and detailed response. I was just hoping for this issue to get some more attention, I find it really compelling.

[safern]

I've assigned the issue to myself in order to take a deep look into it and decide whether it fits in the framework or not for our 3.0 plans.

[safern]

@ianhays and I discussed this and we agree to add this 4 APIs for now:

// Adds a range to the end of the collection.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void AddRange(IEnumerable<T> collection)

    // Inserts a range
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void InsertRange(int index, IEnumerable<T> collection);

    // Removes a range.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Remove)
    public void RemoveRange(int index, int count);

    // Will allow to replace a range with fewer, equal, or more items.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Replace)
    public void ReplaceRange(int index, int count, IEnumerable<T> collection);

As those are the most commonly used across collection types and the Predicate ones can be achieved through Linq and seem like edge cases.

To answer @terrajobst questions:

Should the methods be virtual? If no, why not? If yes, how does eventing work and how do derived types work?

Yes, we would like this methods to be virtual to follow what we're currently doing in ObservableCollection.

Should some of these methods be pushed down to Collection?

Yes, and then ObservableCollection could just call the base implementation and then trigger the necessary events.

Marking as ready for review.

[weitzhandler]

@safern

TBH I find it disappointing that the predicate isn't in the API, because there won't be a way to refine the added collection, hence reducing the number of events occurring when modifying the collection instead of combining the entire action to one. If the approved methods will be virtual, and will be open ended to allow adding the predicate-enabled methods later, that will be bearable. Just bear in mind that predicate methods exist in List<T>.

Have you read my comment and my PR?

[safern]

From reading the threads extensively I figured that the best thing to do regarding making the methods virtual or not, is that we shouldn't make this virtual methods but follow the Collection pattern where we have underlying protected virtual methods as that allows people to add filtering and custom behavior when inserting/removing items. So then the API would look something like this:

    // Adds a range to the end of the collection.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void AddRange(IEnumerable<T> collection) => InsertItemsRange(0, collection);

    // Inserts a range
    // Raises CollectionChanged (NotifyCollectionChangedAction.Add)
    public void InsertRange(int index, IEnumerable<T> collection) => InsertItemsRange(index, collection);

    // Removes a range.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Remove)
    public void RemoveRange(int index, int count) => RemoveItemsRange(index, count);

    // Will allow to replace a range with fewer, equal, or more items.
    // Raises CollectionChanged (NotifyCollectionChangedAction.Replace)
    public void ReplaceRange(int index, int count, IEnumerable<T> collection)
    {
         RemoveItemsRange(index, count);
         InsertItemsRange(index, collection);
    }

    #region virtual methods
    protected virtual void InsertItemsRange(int index, IEnumerable<T> collection);
    protected virtual void RemoveItemsRange(int index, int count);
    #endregion

Just bear in mind that predicate methods exist in List. Have you read my comment and my PR?

I understand your disappointment, but as reviewers have stated previously in this thread, this are complicated APIs and it is always complicated to satisfy all customers out there with exactly what they need, we're trying to handle and expose quality and maintainable APIs, because once we add those APIs, we have to stick with them for the rest of time. So we have to be careful on what we add and how we add them. I would like to keep this proposal as simple as it can be and then for the predicate APIs we can open another issue, focused on those APIs and have the right discussion on what are the problems and how to address them. That minimizes risk and PR would be smaller, easier to review and we would get things right.

I'm not holding back on them, I know they exist in List<T>, but I would like to keep this proposal and addition as simple and useful as possible at the same time. I think that with this first APIs that we're going to bring to the review and most likely would be approved, the most common use cases will be covered.

So from looking at the thread, the complications where brought by the Predicate APIs, so if we want to minimize discussion in this super long/old issue and get APIs approved and going, let's keep this simple and start another discussion focused on the predicate APIs once this issue is approved and APIs are added, how does that sound?

[weitzhandler]

@safern

From reading the threads extensively... I understand your disappointment

Only wanted to make sure you went through it. Thank you very much for your time. I do understand.

// Will allow to replace a range with fewer, equal, or more items.
// Raises CollectionChanged (NotifyCollectionChangedAction.Replace)
public void ReplaceRange(int index, int count, IEnumerable<T> collection)
{
     RemoveItemsRange(index, count);
     InsertItemsRange(index, collection);
}

Kinda missing the point. The goal is raise as less events as possible and avoid unnecessary events. Please take a look at this. This will avoid all unnecessary events when replacing the collection. Disregard the predicate.

[terrajobst]

Looks good, but we should add a ReplaceItemsRange:

public partial class Collection<T>
{
    public void AddRange(IEnumerable<T> collection);

    public void InsertRange(int index, IEnumerable<T> collection);

    public void RemoveRange(int index, int count);

    public void ReplaceRange(int index, int count, IEnumerable<T> collection);

    // These are override by ObservableCollection<T> to raise the event:

    protected virtual void InsertItemsRange(int index, IEnumerable<T> collection);

    protected virtual void RemoveItemsRange(int index, int count);

    protected virtual void ReplaceItemsRange(int index, int count, IEnumerable<T> collection);
}

[weitzhandler]

I'd be happy to be assigned too.

[safern]

All yours 😄

[weitzhandler]

Tx. I'd need a bit of help tho.

Under what solution is Collection.cs? Looks like it's part of Lib, I'm not sure where to get started.

Please have a look here.

[safern]

You need to make the change first in coreclr for the implementation: https://github.com/dotnet/coreclr/blob/master/src/System.Private.CoreLib/shared/System/Collections/ObjectModel/Collection.cs

The reference assembly that exposes Collection is System.Runtime: https://github.com/dotnet/corefx/blob/master/src/System.Runtime/ref/System.Runtime.cs#L3944-L3977

[safern]

Should I first open a separate PR in the coreclr repo for the changes in Collection?

Yes, you can open a simultaneous PR in corefx to expose the APIs in the reference assemblies, add tests and also add the implementation for ObservableCollection. However the CI builds will fail until the coreCLR PR is merged and corefx consumes a new version of the coreclr package.

After adding the APIs in Collection implementation in order to test your changes in CoreFX, once you added the APIs to the reference assembly for Collection you will want to follow this steps to build CoreFX wit a custom version of coreclr: https://github.com/dotnet/corefx/blob/d9193a1bd70eee4320f8dcdd4e237ee9acf689e9/Documentation/building/advanced-inner-loop-testing.md#compile-corefx-with-self-compiled-coreclr-binaries

Please let me know here or in gitter if you need help from my side 😄

[weitzhandler]

Where are the tests for Collection<T> located?

[safern]

They are located in here: https://github.com/dotnet/corefx/tree/master/src/System.Runtime/tests/System/Collections/ObjectModel

[alexsorokoletov]

Finally some justice for the observable collections :) Wonder though how it would work in real-life existing derived classes with AddRange

[adrientetar]

FWIW, two other methods I'm missing in IList/ICollection are GetRange and Reverse. The bare-bones interfaces provided mean I have to either give up on returning an interface in my properties or give up the nice extra methods...

[weitzhandler]

@safern Is there a proper guide on adding tests for System.Private.CoreLib (they're added in corefx), so that it runs against the clr on my machine? I'm reading this but I'm not certain this is the right one, can you please confirm or elaborate?

[safern]

I think following this will be enough: https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/developer-guide.md#testing-with-private-coreclr-bits

[ChaseFlorell]

Unless I'm thick and missing something super obvious, this appears to be landing in NetCore as opposed to NetStandard. Does it need to be implemented in both, will this cover NetStandard?

[danmoseley]

@chaseflorell this repo is for.NET Core as you mention. The dotnet/standard repo is the place where the standard is defined and requests should go in In issues there

[weitzhandler]

Because the Collection is in Private Lib in CLR, I find it really hard and time consuming to set up the tests for my changes in the CLR. Gave it a few shots but didn't have much of success, need to find a free afternoon to get it done 💔

[safern]

Gave it a few shots but didn't have much of success, need to find a free afternoon to get it done 💔

Please let me know or ping me on gitter if you get stuck at some point.

[SkyeHoefling]

Before I realized that the mono corefx implementation is forked from here I got AddRange and InsertRange working locally. Today I spent some time porting it to the main repository to see if I could get it to work with the xUnit tests.

@weitzhandler I would be happy to combine our efforts so we can get this wrapped up as I am able to get the CoreCLR build working with my xUnit tests.

My Current Implementation: Collection (only):

• AddRange & xUnit Tests
• InsertRange & xUnit Tests

Update

I ended up taking this all the way to the point where I can submit a PR to complete this. Looking forward to everyone's feedback

[Grauenwolf]

Do we have an open ticket for adding AddRange to the concurrent collections?

[terrajobst]

@safern is this on track on being done soonish? If so, we could still add it to .NET Standard.

[safern]

Awesome. Yeah PRs are our in CoreCLR and CoreFX. Will try to get them in ASAP

[SkyeHoefling]

@safern @terrajobst What is the process to getting this into the Standard from here? I went ahead before I started dev on this and created an issue to track this in the mono project as well. mono/mono#13265.

I am happy to continue working on anything additional that needs to be done on the other projects to help get this in. This is really going to be a great addition to .NET Standard

[sharpe5]

WPF developer from a large exchange-listed company here.

It's incredible how much of a difference a proper .AddRange() function will make to the speed of ObservableCollection. Adding items one-by-one with a CollectionChanged event after every add is incredibly slow. Adding lots of items with one CollectionChanged at the end is orders of magnitude faster.

tl;dr AddRange() makes the difference between a huge grid with 10,000 items being fast, fluent and responsive vs. slow, clunky and unresponsive (to the point of being unusable).

So this PR gets my +1 vote. Of all the improvements to grid speed, this is by far the most important.