[data grid] Can't scroll to end of dataset fast with paginationModel='server'

[teddis]

I'm using DataGridPro and was loading the entire dataset up to 30-40K rows. Once loaded, one can scroll fast and effortlessly from the start to the end of the dataset leveraging the browser cache.

However, this is extremely slow in a production environment and was forced to switch to infinite scroll with server-side pagination, using RTK query API to fetch pages. The problem is now that while one can infinitely scroll, the scrollbar is constrained to the the currently loaded pages and a bit to scroll to the next page. I specify the rowCount to the actual total length of the data, but the scrollbar doesn't reduce to allow one to scroll quickly to the end or in-between. Must I also now include a pager to navigate through the entire set? Is it possible to achieve the desired behavior I've described?

Search keywords:

Search keywords:

[michelengelen]

Hey @teddis this should definitely be achievable. @MBilalShafi could you take a look here please?

[teddis]

Hey @teddis this should definitely be achievable. @MBilalShafi could you take a look here please?

Awesome. How?

[MBilalShafi]

Hey @teddis,

I think your request aligns more with the row lazy-loading rather than infinite loading. The basic difference between lazy and infinite loading is the rowCount known and unknown. In the case of lazy-loading, since the row count is known, we can calculate the scroll height and render the full page due to which a user can scroll to any given subset of data that will be loaded lazily.

Have you been able to give a try to the lazy loading?

[teddis]

Yes thank you, lazy loading was the solution!

[github-actions[bot]]

:warning: This issue has been closed. If you have a similar problem but not exactly the same, please open a new issue. Now, if you have additional information related to this issue or things that could help future readers, feel free to leave a comment.

@teddis: How did we do? Your experience with our support team matters to us. If you have a moment, please share your thoughts in this short Support Satisfaction survey.

[teddis]

Hi @MBilalShafi Reopening...

I'm running into some problems getting lazy loading to work flawlessly.

1. Sort works flawlessly, however, filterModel is problematic.

(a) My server receives expected arguments and returns expected results requested by fetchRows with sortModel and filterModel. (b) When I use QuickFilter search, I usually see correct results flash twice. But I don't see fetching occur more than once. (c) When filtering a boolean column, I usually see correct results briefly flash on screen, followed quickly by an entire page of skeleton media loading ux that never loads. When I scroll down, part of my results appear. (d) Sometimes with that boolean filter, I'll see old results shown, but my fetchRows function has returned different results.

2. Are there any better examples/tutorials on how to use lazy loading works internally?

3. Is Lazy Loading stable or still under development? I notice the use of apiRef.current.unstable_replaceRows...

Thanks for your help.

Ted

EditOfferTargets.tsx:

import * as React from 'react'
import {ChangeEvent, MouseEvent, useCallback, useEffect, useMemo, useRef, useState} from 'react'

// ** MUI DataGrid
import {
  DataGridPro,
  GridColDef,
  GridColumnHeaderParams,
  GridColumnMenu,
  GridColumnMenuProps,
  GridEventListener,
  GridFetchRowsParams,
  GridFilterInputBoolean,
  GridFilterInputSingleSelect,
  GridFilterInputValue,
  GridFilterItem,
  GridFilterModel,
  GridFilterOperator,
  GridFooterContainer,
  GridLogicOperator,
  gridPageCountSelector,
  GridPagination,
  GridRenderCellParams,
  GridRowModel,
  GridRowSelectionModel,
  GridToolbarColumnsButton,
  GridToolbarContainer,
  GridToolbarDensitySelector,
  GridToolbarFilterButton,
  GridToolbarQuickFilter,
  useGridApiContext,
  useGridApiEventHandler,
  useGridApiRef,
  useGridSelector,
} from '@mui/x-data-grid-pro'
import {GridSortModel} from "@mui/x-data-grid"
import {useLazyGetOfferTargetsQuery, useUpdateOfferTargetsMutation} from "../../store/apps/api/warmlink"
import _, {debounce} from "lodash"

interface Props {
  offerId: string
}

interface PageState {
  firstRowIndex: number,
  lastRowIndex: number
}

type OfferTargets = OfferTarget[]
type FetchRowsResults = { slice: OfferTargets, total: number }

export default function EditOfferTargets({offerId = '0'}: Props) {
  const apiRef = useGridApiRef()

  const initialQueryParams: GetOfferTargetsParams = {
    offerId: offerId,
    offset: 0,
    take: InitialPageSize,
    sortModel: [],
    filterModel: {items: []}
  }

  const [queryOptions, setQueryOptions] = useState<GetOfferTargetsParams>(initialQueryParams)
  const [rowCount, setRowCount] = useState(0)
  const [rows, setRows] = React.useState<OfferTargets>([])
  const [selectionModel, setSelectionModel] = useState<GridRowSelectionModel>()
  const [triggerGetOfferTargets, {isLoading}] = useLazyGetOfferTargetsQuery()
  const [updateOfferTargets] = useUpdateOfferTargetsMutation()
  const [pageState, setPageState] = useState<PageState>({
    firstRowIndex: 0,
    lastRowIndex: 0
  })

  const fetchRows = async (params: GridFetchRowsParams): Promise<FetchRowsResults> => {
    const body = {
      ...queryOptions,
      sortModel: params.sortModel,
      filterModel: params.filterModel,
      offset: params.firstRowToRender,
      take: params.lastRowToRender,
    }
    try {
      console.log(`fetchRows: `, JSON.stringify(params))
      const result = await triggerGetOfferTargets(body).unwrap()
      return {slice: result.rows, total: result.total} as FetchRowsResults
    } catch (error) {
      console.error(error)
    }
    return {slice: [], total: 0} as FetchRowsResults
  }

  // The initial fetch request of the viewport.
  React.useEffect(() => {
      (async () => {
        console.log(`initial fetch`)
        const {slice, total} = await fetchRows({
          firstRowToRender: 0,
          lastRowToRender: InitialPageSize,
          sortModel: [{field: 'name', sort: 'asc'}],
          filterModel: {
            items: [],
          },
        })
        setRows(slice)
        setRowCount(total)
      })()
    },
    // eslint-disable-next-line react-hooks/exhaustive-deps
    [])

  // Fetch rows as they become visible in the viewport
  const handleFetchRows = React.useCallback(
    async (params: GridFetchRowsParams) => {
      console.log(`handleFetchRows: `, JSON.stringify(params))
      const {slice, total} = await fetchRows(params)
      apiRef.current.unstable_replaceRows(params.firstRowToRender, slice)
      setRowCount(total)
    },
    // eslint-disable-next-line react-hooks/exhaustive-deps
    [],
  )

  const debouncedHandleFetchRows = React.useMemo(
    () => {
      console.log(`debouncedHandleFetchRows`)
      return debounce(handleFetchRows, 1000)
    },
    // eslint-disable-next-line react-hooks/exhaustive-deps
    [fetchRows],
  )

  const onSortChange = (sortModel: GridSortModel) => {
    // Ensure name sort is only ASC or DESC, never undefined
    console.log(`onSortChange: `, JSON.stringify(sortModel))
    const nameSort = _.find(sortModel, s => s.field == 'name')
    if (nameSort != undefined) {
      console.log(`nameSort: `, JSON.stringify(nameSort))
      if (!nameSort.sort)
        nameSort.sort = 'asc'
      else
        sortModel.push({field: 'name', sort: 'asc'})
    } else {
      console.log(`nameSort: []`)
    }
  }

  const onFilterChange = useCallback((filterModel: GridFilterModel) => {
      console.log(`onFilterChange: `, JSON.stringify(filterModel))
      setQueryOptions({...queryOptions, filterModel: filterModel})
      setRows([])
    },
    // eslint-disable-next-line react-hooks/exhaustive-deps
    [])

  return (
    <div style={{height: '92%', width: '100%'}}>
      <DataGridPro
        apiRef={apiRef}
        autoPageSize
        checkboxSelection
        columns={columns as GridColDef[]}
        disableRowSelectionOnClick
        filterMode="server"
        getRowId={(row) => row.id}
        hideFooter={false}
        hideFooterPagination={true}
        ignoreDiacritics
        initialState={{
          sorting: {
            sortModel: [{field: 'name', sort: 'asc'}],
          },
          filter: {
            filterModel: {
              items: [],
              quickFilterLogicOperator: GridLogicOperator.Or,
            }
          }
        }}
        loading={isLoading}
        onFilterModelChange={onFilterChange}
        onFetchRows={debouncedHandleFetchRows}
        onRowSelectionModelChange={handleSelectionChange}
        onSortModelChange={onSortChange}
        pagination={false}
        rowBufferPx={150}
        rows={rows}
        rowCount={rowCount}
        rowsLoadingMode="server"
        rowSelectionModel={selectionModel}
        slots={{
          toolbar: CustomToolbar,
          footer: CustomFooter,
          columnMenu: CustomColumnMenu,
          pagination: CustomPagination,
        }}
        slotProps={{
          columnsManagement: {
            getTogglableColumns
          }
        }}
        sortingMode="server"
        sx={{backgroundColor: theme.palette.background.paper}}
      />
    </div>
  )
}

[github-actions[bot]]

:warning: This issue has been closed. If you have a similar problem but not exactly the same, please open a new issue. Now, if you have additional information related to this issue or things that could help future readers, feel free to leave a comment.

@teddis: How did we do? Your experience with our support team matters to us. If you have a moment, please share your thoughts in this short Support Satisfaction survey.

[MBilalShafi]

Hey @teddis,

Sorry about the late response. Is the issue resolved at your end, or there's still need to figure something out?

[teddis]

No worries. I couldn't get the smooth reliable UX I was hoping for with lazy loading, so I opted to find a way just to load the entire 10MB dataset locally into the grid to avoid lazy loading etc entirely.