DataProvider API should not require developer to report the size fo the data

[mstahv]

Version 8.0.0.beta2

The DataProvider API is designed in such a way that it is very hard to provide scalable implementation for large data set. Getting the number of objects in the "query" often requires database to scroll through the whole result set which causes really slow queries. If non filtered use cases developers could often optimize it a bit (Grid), but in e.g. ComboBox there is no efficient way for developer to do it.

The lazy loading should not required developer to provide the size. The UI should just keep getting the next page until result set is smaller than requested. In real life users never scroll so long that this would become slow, instead they'll add more detailed search term.

[Legioth]

The API has explicitly been designed to allow introducing this feature, but those plans have not been realized since there hasn't been enough time to make any component implementation work accordingly.

For Grid, there are some edge cases where the internal bookkeeping of the component falls apart if it needs to adjust the scales of the scrollbars while data is being fetched.

For ComboBox, this would at the very least require changes to how the paging info is displayed (e.g. "1-10/246" that is currently displayed). There might also be some bookkeeping issues similar to the ones already encountered with Grid.

[neerup]

This would make LazyLoading much more powerfull and save developers a lot of work...

[StefanPenndorf]

And this could also be solution/workaround for #6290 -- assuming that users never scroll 200 000+ rows "just for fun" scrollbar can re-adjust until browser ceases to work. Thats far better than the current behaviour that renders grid uselesse if there are theoretically more than 200k records available but user only looks at the top 100 records

[neerup]

Not having this feature is really becoming a problem for us.. We have tried to hack a solution where we return a fictive row count and refresh with a larger row count when the user scrolls near the end. But the result is not pretty because of the many refreshes. Please make this a priority.

[drewoko]

One more possibly related issue from different angle - #10104.

[neerup]

Any chance this will be part of Vaadin 10?

[elmot]

@neerup , better to ask at https://gitter.im/vaadin/flow

[tsuoanttila]

Out of curiosity on this specific ticket, I made a quick PoC changing parts of the DataCommunicator to not deal with the row count but just keep on fetching until you hit a wall, it seems to work more or less out of the box. There's at least some problems when it comes to the Grid component and scrolling which makes it a bit unresponsive at times. I did not test it yet with ComboBox.

[neerup]

Please make this part of the Framework so we don't have to hack the rowcount callback

[mstahv]

Awesome @tsuoanttila, this is truly valuable features. Hopefully you can share a prototype soon for early testing.

[ilgun]

@tsuoanttila Waiting for that too.

[tsuoanttila]

As I currently am a bit busy, I am not able to make it into a proper add-on in the near future. You can check out the beef of the add-on from this gist https://gist.github.com/tsuoanttila/370dfbadcb27346db42a03de15f85fb9

You can use it with for example Grid with

public class SizelessGrid<T> extends Grid<T> {
    public SizelessGrid() {
        super(new SizelessDataCommunicator<>());
    }
}

[mstahv]

Stole your innovation right away 😜 and did some tests:

https://github.com/viritin/viritin/commit/7ac0e05c733b6612cec7b9b330a9198ed326ecf9

Once I somehow got some empty rows to the end of the Grid, but it still didn't break and seems to work otherwise just fine. If you think the example and the SizelessGrid looks good I'm ready to cut a release.

[markathomas]

Hi @TatuLund thanks for the example. Unfortunately, this does not fully fix the problem. As @mstahv noted, the grid has empty rows at the bottom any time the size of the returned list of elements is < the fetch limit. Setting the min push rows to a high number will fetch the first page but scrolling resets the push rows range and fetches some of the same rows over again. I'm afraid this isn't a viable workaround.

[mcooper7290]

I have a similar issue as described: https://vaadin.com/forum/thread/17554172

I tried @tsuoanttila SizelessDataCommunicator but it did not solve the problem. That workaround loads the first page, then calls fetch() many times loading small chunks of data until it's read everything even though the Grid has not been scrolled. The scrollbar is also never displayed. It's slightly better in the sense that the page does eventually load and not hang, but without the scrollbar being displayed it's still very broken.

[mstahv]

I was trying to make the prototype by @tsuoanttila work, but couldn't make it anymore. Throws an exception on the client side. I guess it worked with some certain Vaadin version only 😥

Has somebody a working setup with it?

[mstahv]

Looks like it was a regression, now reverted. Anyways, if somebody has used it in production environment, I'd love to hear your experiences. Hoping to make this an official feature at some point.

[mstahv]

I spent a while to finetune the solution. The component is now called LazyGrid in Viritin 2.11 and it also matches backend queries to "pages" (configurable size), to make it much easier to connect certain backend, such as those based on Spring Data.

Resources:

• A live demo
• Example in Viritin tests
• Example in spring-data-vaadin-crud special branch

[mstahv]

Wrote some of my ideas of the topic to a blog article: https://vaadin.com/blog/request-for-comments-lazy-data-binding-improvements

[vilmosnagy]

Is there a way to support this with Vaadin14?

[mstahv]

Not yet, but the team is currently working on making this a standard feature. I hope we have something to test in pre-releases in couple of weeks.

[mstahv]

Hey @mshabarov & @rolfsmeds, Isn't this thing done 🤔 I guess not completely as TreeGrid is still there to tackle? Maybe create a more specific one for it?