[data grid] Measurement runtime performance

DanailH commented 3 years ago

[x] I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

Some time ago I made some effort to create a snapshot of the current performance state of the DataGrid and XGrid based on a couple of parameters.

Motivation 🔦

This ticket is aimed to provide visibility into those metrics and also for a place to track future progress and update the measurements. In the future, it will be probably beneficial for us to take those measurements as part of the pipeline and catch performance regressions before we ship the code.

To give an overview of what I used as base configuration for the grid:

function DataGrid100() {
  const { data } = useDemoData({
    dataSet: 'Commodity',
    rowLength: 100,
    maxColumns: 20,
  });
  const pagination = {
    pagination: false,
    autoPageSize: false,
  };

  return (
    <div style={{width: '100%', height: 500}}>
      <DataGrid
        {...data}
        {...pagination}
        checkboxSelection
        // localeText={bgBG.components.MuiDataGrid.localeText}
        components={{
          Toolbar: GridToolbar,
        }}
      />
    </div>
  );
}

Here for each DataGrid and XGrid they are variants with 100, 1000, 10000, and 100000 rows and with and without pagination enabled (keep in mind that DataGrid having more than 100 rows the pagination will automatically kick in).

The versions used are as follows:

"@material-ui/core": "^4.11.4",
"@material-ui/data-grid": "^4.0.0-alpha.30",
"@material-ui/x-grid": "^4.0.0-alpha.30",
"@material-ui/x-grid-data-generator": "^4.0.0-alpha.30",

Below you can find information about what has been done. Let's begin.

Examples 🌈

`DataGrid100`

Performance measurements with 100 rows (without pagination)

Bundle size: 323.7 KiB
First contentful paint: 600 ms
Time to interact: 900 ms
Sorting time: 40 ms
Filtering: 63 ms
Scrolling experience: 30 ms for virtual page change (~50 fps)
Keyboard navigation: ~12 ms moving to the next cell / ~20 ms for virtual page change (~43 fps)

Performance measurements with 100 rows (with pagination)

Bundle size: 323.7 KiB
First contentful paint: 600 ms
Time to interact: 1 s
Sorting time: 154.54 ms
Filtering: 254.31 ms
Scrolling experience: -
Keyboard navigation: ~14 ms moving to the next cell (~46 fps)
Pagination page change time: 38.77 ms

`DataGrid1000`

Performance measurements with 1000 rows

Bundle size: 323.7 KiB
First contentful paint: 600 ms
Time to interact: 900 ms
Sorting time: 42.79 ms
Filtering: 66.78 ms
Scrolling experience: 35 ms for virtual page change (~49 fps)
Keyboard navigation: ~13 ms moving to the next cell / ~22 ms for virtual page change (~45 fps)
Pagination page change time: 51.24 ms

`DataGrid10000`

Performance measurements with 10000 rows

Bundle size: 323.7 KiB
First contentful paint: 600 ms
Time to interact: 900 ms
Sorting time: 82.76 ms
Filtering: 115.09 ms
Scrolling experience: 33.58 ms for virtual page change (~50 fps)
Keyboard navigation: ~13 ms moving to the next cell / ~25 ms for virtual page change (~42 fps)
Pagination page change time: 51.56 ms

`DataGrid100000`

Performance measurements with 100000 rows

Bundle size: 323.7 KiB
First contentful paint: 600 ms
Time to interact: 1.3 s
Sorting time: 563.05 ms
Filtering: 392.34 ms
Scrolling experience: 30.15 ms for virtual page change (~50 fps)
Keyboard navigation: 21.48 ms moving to the next cell / 24.36 ms for virtual page change (~45 fps)
Pagination page change time: 55.75 ms

`XGrid100`

Performance measurements with 100 rows (without pagination)

Bundle size: 325 KiB
First contentful paint: 600 ms
Time to interact: 600 ms
Sorting time: 39 ms
Filtering: 57 ms
Scrolling experience: 20 ms for virtual page change (~54 fps)
Keyboard navigation: ~12 ms moving to the next cell / ~25 ms for virtual page change (~43 fps)

Performance measurements with 100 rows (with pagination)

Bundle size: 325 KiB
First contentful paint: 600 ms
Time to interact: 600 ms
Sorting time: 89 ms
Filtering: 161 ms
Scrolling experience: 29 ms for virtual page change (~52 fps)
Keyboard navigation: ~15 ms moving to the next cell / ~30 ms for virtual page change (~46 fps)
Pagination page change time: 40 ms

`XGrid1000`

Performance measurements with 1000 rows (without pagination)

Bundle size: 325 KiB
First contentful paint: 600 ms
Time to interact: 600 ms
Sorting time: 47 ms
Filtering: 69 ms
Scrolling experience: 42 ms for virtual page change (~49 fps)
Keyboard navigation: ~19 ms moving to the next cell / ~32 ms for virtual page change (~43 fps)

Performance measurements with 1000 rows (with pagination)

Bundle size: 325 KiB
First contentful paint: 600 ms
Time to interact: 700 ms
Sorting time: 52 ms
Filtering: 70 ms
Scrolling experience: 20 ms for virtual page change (~54 fps)
Keyboard navigation: ~12 ms moving to the next cell / ~25 ms for virtual page change (~43 fps)
Pagination page change time: 42 ms

`XGrid10000`

Performance measurements with 10000 rows (without pagination)

Bundle size: 325 KiB
First contentful paint: 600 ms
Time to interact: 600 ms
Sorting time: 89 ms
Filtering: 102 ms
Scrolling experience: 81 ms for virtual page change (~45 fps)
Keyboard navigation: ~20 ms moving to the next cell / ~40 ms for virtual page change (~45 fps)

Performance measurements with 10000 rows (with pagination)

Bundle size: 325 KiB
First contentful paint: 600 ms
Time to interact: 700 ms
Sorting time: 91 ms
Filtering: 81 ms
Scrolling experience: 20 ms for virtual page change (~54 fps)
Keyboard navigation: ~12 ms moving to the next cell / ~25 ms for virtual page change (~43 fps)
Pagination page change time: 40 ms

`XGrid100000`

Performance measurements with 100000 rows (without pagination)

Bundle size: 325 KiB
First contentful paint: 600 ms
Time to interact: 800 ms
Sorting time: 437 ms
Filtering: 413 ms
Scrolling experience: 88 ms for virtual page change (~48 fps)
Keyboard navigation: ~19 ms moving to the next cell / ~41 ms for virtual page change (~45 fps)

Performance measurements with 100000 rows (with pagination)

Bundle size: 325 KiB
First contentful paint: 600 ms
Time to interact: 800 ms
Sorting time: 433 ms
Filtering: 350 ms
Scrolling experience: 20 ms for virtual page change (~54 fps)
Keyboard navigation: ~12 ms moving to the next cell / ~25 ms for virtual page change (~43 fps)
Pagination page change time: 45 ms

Final thoughts 🙏

First of all, if you have made it here - thank you 😅. It's quite a long description. What is left to be done is to benchmark how competitors such as AG Grid are scoring in similar configurations so we know where we sit in terms of performance.

All the code is in my own repo but we cal always move it. You can check it here -> https://github.com/DanailH/mui-datagrid-measurements

oliviertassinari commented 3 years ago

@DanailH Thanks for opening the issue. A couple of thoughts:

a. "Bundle size". I think that we should ignore the notion in this issue. We have #37, to first measure it continually on each PR. Then #18 with the actionable steps to improve. So far, we got little upvotes on the issue, so I'm not sure that people care. We could learn more in a future survey. I suspect that we could do a lot better. b. "First contentful paint". The value is, all the same, we also don't support SSR. I don't see why that this dimension would matter going forward. I propose we ignore. c. "Time to interact". I think that we could improve the way we measure it. We currently include in the measure the time it takes to build the test data set. This adds variance in the measures. d. https://www.crankuptheamps.com/blog/posts/2020/01/23/grid-comparison-2/ a comparison blog post to evaluate the "best" data grid. The blog post is significantly biased. Since when "best" is 80% about the performance? But maybe it's a pain that many end-users experiences and has grown as a differentiator 🤷‍♂️

Regarding the next step. I think that we should create a PR with a new benchmark folder as we have in the core that is automated (not in the CI). I would propose we measure based on d. :

Time to first render, excluding the time to generate the data set
Measure FPS with slow scrolling
Measure FPS with fast scrolling
Memory usage for loading the data set
FPS for streaming data

We could also add a few bonuses:

sorting time, I would expect demanding configuration to handle this server-side. I assume for the client-side, we only have to make sure, it's not terribly bad.
filtering time, I would expect demanding configuration to handle this server-side. I assume for the client-side, we only have to make sure, it's not terribly bad.

As for how we measure, we can copy the infra we have in the main repo. We can also look into the automated benchmark created by @Janpot in https://github.com/mui-org/material-ui-x/issues/1933#issuecomment-874074947. There are also probably more examples to copy from online.

@DanailH How does it sound? Do you want to continue the effort?

DanailH commented 3 years ago

@oliviertassinari sure I can pick this up on the side and do the automated measurements as a follow-up.

AndrijBartenev commented 3 years ago

You are doing a great job, but it looks like it's better to split the scrolling into two parts: horizontal and vertical. I've just noticed that fps is much better for vertical scrolling, and not that good for the horizontal one.

m4theushw commented 3 years ago

@AndrijBartenev In the latest beta release, we introduced a new virtualization logic. Now both vertical and horizontal scroll should run at the same FPS.

mui / mui-x