Improvements to make lazyquerycontainer suitable for combo boxes in combination with large result sets

[michael-simons]

Hi Tommi.

I am using your lazyquerycontainer with a custom JPA filter implementation for Vaadins Comboboxes. My own criteria based filtering works very well but in some cases we have really big unfiltered results.

When lazyquerycontainer is used with Vaadins Combobox, a moderate page size and more than some 1000 results, it doesn't work quite well as the Combobox requests the index and the LazyIdList needs to linear search all results for an index leading to querying all objects (not lazy anymore for nothing) and many, many queries.

My request has to components:

1. Introduce an ItemIdListFactory that produces id lists. Passing this to LazyQueryView keeps its immutable structure but providing an entry point for customization. The original functionality is now in DefaultItemIdListFactory, same as before. All tests are green.
2. Introduce a SmartItemIdListFactory which produces SmartLazyIdList. With the requirement of the idPropertyIds being comparable to each other and the assumption that they are in ascending order a binary search for an index can be used which makes the container suitable for ComboBox with more than (tested) 100.000 entries.

What do you think? The fork is based on tag 2.1.0. I've added a custom version for the time being.

I really hope this makes into your upstream and that you'll release a new version in the near future.

Again, thank you for your work and letting me participate.

Michael

[EduFrazao]

What a nice contribuition! :)

[michael-simons]

Thanks Eduardo, i hope it gets pulled :)

[michael-simons]

Hi Tommi, any chance of revisiting this? Would be awesome.

[tlaukkan]

Hi

I appreciate the thought and work put to study this issue.

There are some fundamental things to consider:

1) Existing implementation:

If you set idProperty to null then you are automatically using NaturalNumberIdList:

/**
 * Gets list of item IDs present in this view.
 * @return list of item IDs present in this view.
 */
public List<?> getItemIdList() {
    if (itemIdList == null) {
        if (queryDefinition.getIdPropertyId() != null) {
            itemIdList = new LazyIdList<Object>(this, queryDefinition.getIdPropertyId());
        } else {
            itemIdList = new NaturalNumberIdsList(size());
        }
    }

    return itemIdList;
}

2) The idea of the lazy query container is to pull the data from database on demand including the IDs. It does not really make sense to pull the entire table out to get the IDs? NaturalNumberIdsList uses the index of the rows as ID instead which solves the problem nicely.

3) If you decide to pull the IDS of the entire list to memory then you can easily just create a HashMap of index -> ID and the other way around to find corresponding IDs for indexes.

We can certainly add factory for ID list implementations if there are other implementations than the default ones adding real value.

Before doing this I would like to understand better your use case:

1) Why you can not use NaturalNumberIdsList? Is there some problem with combo boxes?

2) Are you planning to load all the IDs in memory or am I just understanding something wrong?

3) If you plan to load all the IDs in memory then what do you actually save by lazy loading? Not time or processing at least. Maybe memory if you have large rows?

[michael-simons]

Hello Tommi,

great, that you come back to this.

I got that, that LQC only fetches entries on demand. It works really great with tables.

The big problem is that i'm using the LQC with comboboxes. Comboboxes are implemented in such a way that they request the absolut index of the currently selected item over all items.

That is the will load all items through the ItemIdList in indexOf at least up to the item selected by calling lazyQueryView.getItem. My client wishes to use comboboxes and they will grind an app to halt, even with LQC with datasets bigger than 100.000 items.

I can assume that those datasets are sorted. So i can use a binary search for the given index. That also results in database queries most of the time but by far not that much.

So the goal is not like you said in 2 to pull the IDs, quite contrary. The solutions reduces the amount of database calls when the LQC is used in comboboxes and the items are sorted.

Have a great day, Michael.

[tlaukkan]

Hi

Could you try out what happens if you set idProperty to null?

Best regards, Tommi

[tlaukkan]

One more note here. For large table combo box use case I would suggest creating custom field which would actually execute search based on user input and show the matching rows in a combo box. Instead of trying to dynamically handle millions of rows in normal combo box. The combo box is just not designed for that and hence the problems in my opinion.