[DataGrid] Indicate internal loading state

[githorse]

Duplicates

• [X] I have searched the existing issues

Latest version

• [X] I have tested the latest version

Summary 💡

The loading prop is useful when I'm doing some computation outside of the grid and need to indicate that the grid data is stale. However, with large data and certain settings, the grid itself may take noticeable time to reload the data, during which time the loading state is off and the grid doesn't indicate that anything is happening.

Is there some way to indicate that loading is still going on (this time, inside the grid)? A couple of ways to handle it:

Combine the internal loading state with the external one, so that the loading overlay is displayed when loading={true} or the grid is loading internally. Presumably that should be controlled with a flag so the current behavior can be maintained if desired.

Provide some sort of onContentReady callback when the component finishes loading/repainting. This is more cumbersome for the developer but probably allows more fine-grained control over loading states. (It might also be useful for other purposes.)

Ideally we'd have both of these options.

Examples 🌈

No response

Motivation 🔦

My real-world example: I have a large dataset, say 500k rows, and I'm doing my own filtering outside of the grid with custom filter widgets. My filtering code turns on a loading flag while it's working, so I can pass that to the grid to indicate that something is updating. But after my code returns a new dataset of 500k (filtered) rows and passes that to the grid, there's a delay of several seconds before the grid itself updates.

Here's the sequence of events:

1. User clicks some filter widget.
2. Filter code sets loading={true}. Grid displays loading overlay.
3. Filter code churns for a second or two.
4. Filter code spits out new filtered record set.
5. Filter code sets loading={false}. Grid stops displaying loading overlay, but the data (and total) is wrong.
6. \<several seconds elapse>
7. Grid updates with new set of rows and displays total at the bottom.

I need a way to indicate during step (6) that the grid is still loading and the displayed data is out of date.

Order ID 💳 (optional)

46840

[DanailH]

Hi, @githorse thanks for raising this!

If I understand your case correctly - you first load 500k rows, then filter them outside of the grid, without using what we provide as filtering options, and then supply the filtered rows (a subset of those 500k rows) back to the grid via updating the rows prop? If this is correct then the loading time is kind of expected because the grid is a completely new data set so it needs to update all the internals. A better approach is to move your filtering logic to the grid by using the custom filtering option or enable lazy loading and only load a range of rows.

[githorse]

Hi @DanailH - I have an existing filtering ecosystem that does all the work for me (with some nifty custom filtering widgets), so it's much easier to just let that calculate the new record set and pass it into the grid. That said, I'm definitely looking into integrating my filters with the grid filtering model to see if it will improve performance, though there are certain complications.

But the problem I'm describing here is more general. What if I just download a new set of 500k records from the server? I will still see a second or two of loading time with no visual indication that it is happening. Same sequence of events, basically:

1. User clicks refresh button (or, say, picks a new date range).
2. API code sets loading={true}. Grid displays loading overlay.
3. Download happens for a second or two.
4. API spits out new record set.
5. API code sets loading={false}. Grid stops displaying loading overlay, but the data (and total) is wrong.
6. \<several seconds elapse>
7. Grid updates with new set of rows and displays total at the bottom.

Or maybe the record set isn't terribly large, but the grid needs to do some complicated aggregations, groupings, pivoting, etc. There might be any number of reasons why the grid is going to spend a non-negligible chunk of time calculating something, and it seems like during that time I need a way to indicate to the user that something is going on, no?

[githorse]

Update: even when I use the grid's built-in filtering, I still get a delay of a second or two when applying a filter over a large dataset, during which time the screen is basically frozen and there's no indication that anything is happening. So in this case too I need some kind of onLoadingStart/onLoadingEnd hook, or a built-in loading overlay, or both. (I have not tried lazy loading yet, so that may improve the performance in this particular case.)

[DanailH]

I was going to suggest that if your set is so big (500k) you wouldn't want to load all of it initially. It would be a much better user experience if you either lazy load it or paginate it and then use server-side filtering and sorting to just fetch the needed subset of rows. That being said there is a known issue related to the grids' performance with larger sets (>100k) which is related to how the state is being managed, especially when you use lazy loading. The problem is explained in depth here https://github.com/mui/mui-x/issues/8085

[githorse]

Thanks @DanailH -- I will definitely look into lazy loading and/or pagination for this use case. I agree, I don't think I'm leveraging the best tools for the job currently.

That being said, I still think any specific performance problem is beside the way here. Unless we are claiming that the grid will never take a perceptible amount of time to perform any operation on any dataset (clearly not the case in my experience), it seems to me like we need a way for the grid to communicate that internal loading state.

[wasib-sureify]

I am facing a similar issue where i load the data grid with 100k rows, and when i try to perform a search (quick filter) on the data grid, it takes a couple of seconds to filter the results. While it is searching/filtering the UI is stuck and there is no way to inform the user that it is processing things internally. It would be nice if there was a way to show a loader or a spinner while the search was happening.

@githorse @DanailH Any Idea how I can do this with the data grid?

[romgrk]

@wasib-sureify I'm not sure it would be enough for that use-case. When we do filtering, the whole UI thread is blocked because we're computing stuff non-stop. The only way we could keep the UI responsive is to implement async filtering on a worker thread for large datasets. Filtering should get better after #9120 is completed. I've also listed in #9167 issues specific to quick filters, as well as some workarounds you can use to improve the performance.

[romgrk]

@wasib-sureify #9206 might also be of interest to you.

[wasib-sureify]

@romgrk This is insane!! I was able to notice the search performance in quick filters improves from 4678ms to 46ms for the 100k rows example in your branch. As mentioned by you, if the changes from #9120 get merged, the upfront processing cost for the buffer should be well within acceptable range too!

I will be keeping an eye out on those PR's and the suggestions you shared for the time being. Cant wait to test these changes if and when they get merged and available in the mainline.

[oliviertassinari]

keep the UI responsive is to implement async filtering on a worker thread for large datasets

@romgrk I think that we could also keep the UI responsive by running the filtering during idle time (with requestIdleCallback()), in chunks. We started to do this in #1170 for the generation of the fake data in the docs demos. I imagine it has the advantage to avoid the need to transfer a large dataset between the main thread and the worker thread. I don't know, maybe SharedArrayBuffer can solve this problem too.