Table: VirtualScroll with Lazy Loading and Filter with unknown number of total rows

[danieleconversa]

Describe the bug

I have a large dataset of which I don't know the total number of records. I want to initialize the table with a small number of items (20) and load items in bunches of 20 when the table is scrolled.

I have different behaviours with Primeng 13.4.1 and with Primeng 14 or 15.

Primeng 13.4.1 and Angular 13.0.3 https://stackblitz.com/edit/primeng-tablevirtualscroll-demo-frffd5?file=src%2Fapp%2Fapp.component.ts

I initialize the table array with 20 elements Array.from({ length: 20 }) and the table rows number also to 20. The table loads the data as expected, loading the first items and loading new items when scrolling. The issue I have in this scenario is that, if I load for example 200 items and then I use filters that reduce the number of items to 10, the first rows of the table are updated correctly but I still see the items loaded before.

Only after scrolling up and down several times, the table is refreshed correctly and I see the correct number of filtered items.

Primeng 14.2.2 and Angular 14.1.3 https://stackblitz.com/edit/primeng-tablevirtualscroll-demo-4th7fh?file=src%2Fapp%2Fapp.component.ts Primeng 15.2.0 and Angular 15.2.0 https://stackblitz.com/edit/primeng-tablevirtualscroll-demo-mchojj?file=src%2Fapp%2Fapp.component.ts

If I do not initialize the table array with a number of elements equal or superior to the total number of rows of my dataset (which is unknown but above 100k records), the table only loads the number of rows I specify, without triggering other load events for the following pages. The result is that, if I initialize the array to 20 elements, only the first 20 records are loaded. If I initialize the table array with 10000 or more elements, the scrolling becomes infinitely slow and long with empty rows, which is misleading and creates a bad user experience. Besides,if I initialize the table array with 10000 elements and filter down to only 10 records, for example, the records are updated correctly immediately but the scroll remains of a table of 10000 rows, which is huge, with empty rows.

Last thing to highlight, if I set the table rows to 10, the loading skips some rows leaving them empty (skeleton) or hungs.

Environment

Stackblitz with versions 13, 14 and 15

Reproducer

No response

Angular version

13.0.3 and 14.1.3 and 15.2.0

PrimeNG version

13.4.1 and 14.2.2 and 15.2.0

Build / Runtime

Angular CLI App

Language

TypeScript

Node version (for AoT issues node --version)

v16.14.0

Browser(s)

Chrome 114.0.5735.198

Steps to reproduce the behavior

Start scrolling the table and adding filters. Try to change the variable rows, which is used both for the table number of rows and for the initialization of the table array, to 10, 20 and 10000. Try to filter to see how table is updated.

Expected behavior

I expect the table to load data in chunks of 20 records as specified in table rows without having to initialize the table array to a 10000 elements or more. I expect that when a filter is applied, the table is refreshed accordingly without having to trigger many load events scrolling up and down.

[joelcorrales]

Had the same issue, when i reported this there was another problem with the loading method, they fixed that but not this. I get they made their own implementation of the scroller, but changed the functionality of the table too, in v13.4.1 the user can just scroll down and loads pages as needed, but not you need to put a lot of garbage empty data to achieve this, which kind of defeats the infinite scroll point of loading data as needed. I also notice that adding all the empty results makes the table super slow and even makes frozenColumns to slide in and out constantly while you scroll.

[ebrooks-delta]

@joelcorrales Hi. We will be stuck on v13 until it is fixed. It seems pretty critical to me. Aren't VirtualScroll + Lazy loading a very common implementation for large tables these days? Can I help fix this? Thanks

[drilko]

Was about to post the same issue but found this. Is there any way to avoid having to initialize a list of 100k+ undefined items? Another problem is that I don't know number of items upfront. My plan was to continue fetching until I get 0 items in response and then stop sending requests. Please provide some feedback on this.

[drilko]

I managed to hack through this somehow, code below with explanations:

<!-- Table definition, notice no handler for onLazyLoad output -->
 <p-table    
    [value]="data"
    [scrollable]="true"
    scrollHeight="flex"
    [rows]="rows"
    [customSort]="true"
    [virtualScrollOptions]="scrollerOptions"
    [virtualScroll]="true"
    [virtualScrollItemSize]="100"
  >

Component code:

@Output() lazyLoad = new EventEmitter<{
    event: TableLazyLoadEvent;
    load: (data: MyDto[]) => void;
  }>();

loading = false;
stopLoading = false;
lastLazyLoad: {
    first: number;
    rows: number;
  } = {
    first: 0,
    rows: 10,
  };
scrollerOptions: ScrollerOptions = {
    numToleratedItems: 10,
    delay: 300,
   // If appendOnly is false, scroll behaves weird after new items get added to the list (jumps up and down)
  // This could also be performance issue for some so test it out..
    appendOnly: true,
    onScroll: (event: { originalEvent: Event }) => {
      const el = event.originalEvent.target as HTMLElement;
     // You can adjust this threshold if you want to 
      const atBottom = el.scrollHeight - el.offsetHeight - el.scrollTop < 1;

      if (atBottom && !this.stopLoading) {
        this.triggerLoad();
      }
    },
  };
triggerLoad(firstLoad = false) {
    this.loading = true;
    const event = firstLoad
      ? this.lastLazyLoad
      : {
          first: this.lastLazyLoad.first + this.lastLazyLoad.rows,
          rows: this.rows,
        };
    // Smart components should bind to this output
    this.lazyLoad.emit({ event, load: data=> this.onDataLoad(data) });
    this.lastLazyLoad = event;
  }

onDataLoad(data: MyDto[]) {
    if (!data.length) {
      this.stopLoading = true;
      this.loading = false;
      return;
    }

    this.data.push(...data);
    // This was a hack to show new items
    // this.table.cd.detectChanges() wasn't working
    this.table.scroller?.init();
    this.loading = false;
  }

This is basically it. Code sure needs improvement but it is functional (for my use case). There is obviously a wasteful http call when loading the last page but most of people won't scroll through thousands/millions of items for that to happen. I haven't played with filters but I'm assuming it could be done with some adjustment.

[danieleconversa]

@drilko could you post a stackblitz working example? Thanks