After directly scrolling down to the last rows of a (big) lazy-loading grid, these last rows of the grid are not displayed

[Mitch64]

Description

When you have a big lazy-loading grid and you scroll down with the vertical scrollbar directly from the first rows to the last rows, these are not displayed. When you instead programmatically scroll to the end using grid.scrollToIndex(NUM_ROWS-1) all works without problems. The problem also does not occur (according to my tests) as long as the grid has not a horizontal scrollbar. That means, if the grid has less columns than the number of columns where a horizontal scrollbar is displayed, there is no problem. In the first place it is clear why the rows are not displayed. There is even a warning message in the logs: "Attempted to fetch more items from server than allowed in one go: number of items requested '500050', maximum items allowed '500'." I did some research in the JavaScript code using console.log() statements and it looks like the wrong number of rows to request from the DataProvider is calculated there, but as I am no expert I was not able to find the cause.

Expected outcome

The rows are displayed no matter how or where the horizontal scrollbar is scrolled to.

Minimal reproducible example

package com.test.example;

import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;

import com.vaadin.flow.component.button.Button;
import com.vaadin.flow.component.grid.Grid;
import com.vaadin.flow.component.orderedlayout.VerticalLayout;
import com.vaadin.flow.data.provider.AbstractBackEndDataProvider;
import com.vaadin.flow.data.provider.Query;
import com.vaadin.flow.data.renderer.LitRenderer;
import com.vaadin.flow.function.ValueProvider;
import com.vaadin.flow.router.Route;

@Route
public class MainView extends VerticalLayout {

  private static final int NUM_ROWS = 500000;
  private static final int NUM_COLS = 200;

  private static final class SimpleDataProvider extends AbstractBackEndDataProvider<String[], Void> {
    @Override
    protected Stream<String[]> fetchFromBackEnd(Query<String[], Void> query) {
      int offset = query.getOffset();
      int limit = query.getLimit();

      List<String[]> rows = new ArrayList<String[]>(limit);
      for (int i = 0; i < limit; i ++) {
        String[] r = new String[NUM_COLS];
        r[0] = "" + (offset + i);
        for (int j = 1; j < NUM_COLS; j ++) {
          r[j] = "val " + (offset + i) + "-" + j;
        }

        rows.add(r);
      }

      return rows.stream();
    }

    @Override
    protected int sizeInBackEnd(Query<String[], Void> query) {
      return NUM_ROWS;
    }  
  }

  private static final class SimpleValueProvider implements ValueProvider<String[], String> {

    private final int _colIdx;

    private SimpleValueProvider(final int colIdx_) {
      _colIdx = colIdx_;
    }

    @Override
    public String apply(final String[] source_) {
      return source_[_colIdx];
    }
  }  

  public MainView() {    
    Grid<String[]> grid = new Grid<String[]>();
    grid.setAllRowsVisible(false);
    grid.setDataProvider(new SimpleDataProvider());

    grid.addColumn(LitRenderer.<String[]>of("${item.col}").withProperty("col", new SimpleValueProvider(0))).setHeader("ROW");

    for (int j = 1; j < NUM_COLS; j ++) {
      grid.addColumn(LitRenderer.<String[]>of("${item.col}").withProperty("col", new SimpleValueProvider(j))).setHeader("COL " + j);        
    }

    add(grid);

    Button button = new Button("Scroll to end", e -> {
      grid.scrollToIndex(NUM_ROWS-1);
    });

    add(button);
  }
}

Steps to reproduce

1. Set up a Vaadin Flow project using spring-boot starter
2. Use the MainViewclass from above. Delete GreetingService.java
3. Test within your IDE running Application.java.: Directly after start, scroll (directly!) down to the last rows or to rows near the end. When NUM_COLS is 5, no problem exists. If NUM_COLS is, let's say 50, the problem exists. In my case the problems start, when NUM_COLS is 14 or above, because then the grid shows a horizontal scrollbar. However, when, after starting the program, you click the button below the grid to scroll to the last rows, no problem occurs. Only when the vertical scrollbar is used, the problem occurs.

Environment

Vaadin version(s): 24.0.4 OS: macOS 12.6.5 also tested on Windows 10 21H2 I checked with several browsers, Safari 16.4.1, Chrome 112.0.5615.137, Firefox 112.0.2 so the issue does not seem to be browser related

Browsers

Issue is not browser related

[yuriy-fix]

Let's check if it is a regression from Vaadin 23.

[sissbruecker]

Can reproduce this. As mentioned in the description, the horizontal scroll bar needs to be visible. Also this seems to be only reproducable if scroll bars are permanently visible. So for example on Windows, or also an macOS if scroll bars are configured to be shown permanently in the system settings. It's not reproducable on macOS if scroll bars are configured to only show when scrolling. It logs the same error message as reported above, which indicates that the grid tries to fetch all pages in a single request:

Attempted to fetch more items from server than allowed in one go: number of items requested '500050', maximum items allowed '500'.

It's also reproducable with Vaadin 23.3, so probably not a regression.

macOS, scroll bars permanently visible, not working

https://user-images.githubusercontent.com/357820/234600220-d59a21eb-e875-42dd-9fb4-6da641cc1f9f.mov

macOS, scroll bars shown when scrolling, working

https://user-images.githubusercontent.com/357820/234600314-f62ea57e-0b97-4aa6-9ea5-6106b180b9db.mov

[vursen]

This can be a bug in the virtualizer. It appears to render rows non-sequentially under the described circumstances.