[DataGrid] Lazy loading, Infinite Loading

oliviertassinari commented 4 years ago

[x] I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

Provide a way to only load a fraction of the data, to prioritize data loading for the visible parts of the screen without pagination.

Examples 🌈

During the very first iteration on the data grid https://github.com/mui-org/material-ui/pull/18872, we come up with the following API:

export interface DataProviderGetListParams {
  sorting: SortingType;
  pagination: {
    start: number;
    end: number;
  };
}

export interface DataProviderType {
  getList: (params: DataProviderGetListParams) => Promise<any[]>;
}

export interface DataGridProps extends StandardProps<React.HTMLAttributes<HTMLDivElement>, DataGridClassKey> {
  /**
   * Manage the communication with the data store.
   */
  dataProvider?: DataProviderType;

Where the dataProvider default to a value dedicated to client-side operations with the rows prop. It can probably be improved.

Motivation 🔦

It's a sub-problem of #240.

Benchmark

dtassone commented 4 years ago

We already handle server-side pagination: https://material-ui.com/components/data-grid/pagination/#server-side-pagination. You can override the number of rows in rowCount.

We don't have the infinite scrolling behavior, which will consist of hooking the virtualization to the user server, so we will tell the server which index range we need to render. We also need to add the infinite rowCount behavior or some kind of event when the user reaches the bottom of the viewport, so we load more rows and the scrollbar can resize.

austinlangdon commented 3 years ago

We just started using XGrid and it's a terrific component! But no infinite scroll API is disappointing to see - we happen to utilize infinite scroll over pagination in most of our app.

We've been using this library for infinite scrolling needs. Perhaps something to benchmark?

https://github.com/danbovey/react-infinite-scroller

mbrookes commented 3 years ago

no infinite scroll API is disappointing to see

@austinlangdon Glad to have you as a user, and thanks for the feedback. We do try to make it clear that the product is still in development, both on the product page in the store, and in the documentation. Is there something we should change to make that clearer?

jlil commented 3 years ago

I was going through the docs and this was not clear to me either. Reading rows virtualization and Server-side data makes one believe infinite scroll/loading is supported. The lack of this feature is also a blocker for our project.

vd-yasir commented 3 years ago

@austinlangdon can you share how you used xgrid with https://github.com/danbovey/react-infinite-scroller

austinlangdon commented 3 years ago

@austinlangdon can you share how you used xgrid with https://github.com/danbovey/react-infinite-scroller

@vd-yasir Our solution was to update our API request to get a large amount of data from our API as recommended (~1000 records). Most likely the user will never scroll past the first couple hundred rows

DanailH commented 3 years ago

As I started working on this problem it would be interesting to hear what are your thoughts on the infinite/lazy loading with a combination of the autoHeight prop. Should this case be possible and if yes how would the interaction look?

oliviertassinari commented 3 years ago

Should this case be possible

I don't think so, it doesn't make any sense to me. IMHO autoHeight if meant to reduce the gap with a regular/simple <Table>, nothing more.

DanailH commented 3 years ago

After discussing it further this is what it will be delivered in the first iteration:

A callback that will be invoked when the scroll reaches or gets near the last currently loaded row. This will provide a way for the rows to be updated and essentially load more rows
A way to alter the default threshold of when that callback is invoked (at the very bottom, or X pixes before the bottom of the grid)
If the autoHeight prop is set that callback will not be called
An example of how to use the provided functionality to achieve an infinite scroll functionality

oliviertassinari commented 3 years ago

An example of how to use the provided functionality to achieve an infinite scroll functionality

I would propose we add it into https://material-ui.com/components/data-grid/rows/#updating-rows as it also where we document the streaming. Then, in the future, once the page is too long, we could maybe move both into a new https://material-ui.com/components/data-grid/data/ page for data loading?

Considering that pagination over 100 rows is only available in the Pro plan. We can either have this feature inside DataGrid or XGrid. I don't have any preferences. DataGrid maybe for the sake of covering developers that don't/can't load 100 rows as we have seen many times in the Autocomplete: https://github.com/mui-org/material-ui/issues/18450?

oliviertassinari commented 3 years ago

A subset of the issue was fixed, the cursor pagination with infinite loading.

We have #1247 as a follow-up. It's not very clear to me, but #1247 might be what most developers are looking for. I'm not soo sure. With PostgreSQL when you have 10,000 rows, it's quite easy to get a total row count. However, when you have 1,000,000 rows, it becomes a real challenge (COUNT(*) becomes too slow). In the former case, #1247 would yield a better UX. In the latter case, you might not have the choice but to leverage the solution in #1199.

RockyMM commented 2 years ago

It's 10 months later, but just to leave a comment - naively counting rows will most likely lead to performance problems. However, there could be always a way to optimize the row count query, and if all fails, PostgreSQL, Oracle, but also some other RDBMSs have an option to deliver estimated row count which should fire up in milliseconds.

I think if someone wants to use it, then let them. The performance penalty goes on them.

mui / mui-x