Implement support for scrolling through the whole remote dataset

[roblotter]

We need to add support for scrolling through entire remote datasets - the server will tell InfiniteTable the size of the dataset, and InfiniteTable needs to render all those rows, even they are not entirely loaded yet.

[roblotter]

WIP Initial proposal

When there is grouping - feedback from team - too clunky, avoid

const data = {
   groupDataCount: 10_000, // the count of all data items in the DB
   groupRowsCount: 20, // 20 top-level rows in DB, although groupRows might 
   // currently hold less than that - but as the user scrolls down, groupRows will eventually have 20 items
   groupRows: [
      {
        groupKey: 'Management',
        groupRowsCount: 80, // have 80 child items in the DB for this group key 
       // needed in order to know how many group rows to render
       // although just the first 10 are present now - see the `groupRows` array

        groupDataCount: 2400, // this is the count of all actual data items (rows, so only leafs) in this group
        groupRows: [
           {
             groupKey: 'USA',
             groupData: [....], // have 100 data items initially, although in this group, there are 1400 items in database
             groupDataCount: 1400
           },
           {
             groupKey: 'UK',
             groupData: [....], // have 80 data items initially, although in this group, there are 2000 items in database
             groupDataCount: 2000
           },
           //.... another 8 child groups here
        ]
      },
      // ... other top-level group rows
   ]
}
<DataSource data={data} />

The idea is that as the user scrolls the viewport of the table (which has a scrollbar to fit all the actual rows in the DB), a callback is called whenever new data is needed and the data passed to the <DataSource /> component is enhanced with the missing data that should be visible in the viewport in the current scroll position. In this way, we're externalising data fetching to the developers using InfiniteTable and thus they have complete flexibility in what tools they use to do data-fetching. This is very useful for advanced use-cases.

However, for simpler use-cases, we can provide a loadRows: ({ parentKeys, parentId: string | null, startIndex, endIndex, groupBy, sortInfo, })=> Promise<{data,totalCount}> method (and also another variation, loadGroupRowOnce: ()=> Promise) to make it easier to use by devs starting with InfiniteTable. The difference between loadGroupRow and loadGroupRowOnce is that the loadGroupRowOnce is only called once for a group row, so on subsequent collapse/expands, the child rows are all there, while when loadGroupRow is used, the child rows/groups for a group row are discarded on collapse and then need to be re-fetched on expand

Grouping + aggregation

For now, to keep things simple, I haven't added aggregation info in the data structure above, but that's coming soon.

[roblotter]

The structure proposed above is deeply nested - another option would be a more de-normalized one, like below (still early exploration)

const data = {
  groupData: {
    keys: [
      ['Management'],
      ['Management','TopLevel'], // group keys here as arrays or could be strings separated by '/' or configurable separator
      ['IT'],
      // initially just those 3 groups should be rendered
    ],
    keysCount: 200, // 200 total group rows on server
    groupValues: {
       'Management': {
           // this is a top-level group, so wont have a data array
          // but it can have aggregation values
         aggregations: { salary: 5000 }
       },
      'Management/TopLevel': { // group keys separated by '/' or a configurable separator
        data: [ 
            // initially say we have 100 records, while the groupCount is 500 - items on server
        ],
        groupCount: 500
      },
    }
  } ,
}

[roblotter]

This was implemented in v 0.2.3, so closing this