Tabulator: When a filter is applied, the selection index does not match up with the 'real' index.

[DatDucati]

ALL software version info

(this library, plus any other relevant software, e.g. bokeh, python, notebook, OS, browser, etc)

• OS: MacOS 13.0, also observed on Ubuntu 20.04
• Python 3.10.6
• panel == 0.14.1
• pandas == 1.5.1

Description of expected behavior and the observed behavior

Tabulator does not update its internal dataframe (or at least the indexes) if the dataframe is filtered.

See the example, if you filter by 'Astrometry' and select the first element, you will get index [0] (rowid=1), instead of [113] (rowid=114).

Complete, minimal, self-contained example code that reproduces the issue

app.py:

#! /usr/bin/env python

import panel as pn

from mainLayout import PlanetTable

planetTable = PlanetTable()

mainLayout.py:

import pandas as pd
import panel as pn

class PlanetTable:
    def __init__(self):

        pn.extension()
        # Populate Table

        # source https://github.com/mwaskom/seaborn-data/blob/master/raw/planets.csv
        self.planets = pd.read_csv("planets.csv", comment="#")
        self.tabulator = pn.widgets.Tabulator(self.planets)
        self.tabulator.disabled = True
        self.filter_active = False

        # Set up selection, and filter callbacks

        self.tabulator.param.watch(self.cb_select, ["selection"], onlychanged=False)
        self.filter_select = pn.widgets.MultiChoice(name="Disc Method")
        self.filter_select.link(target=None, callbacks={"value": self.cb_onFilter})
        self.filtered_df = self.planets

        for x in self.planets["pl_discmethod"].unique():
            self.filter_select.options.append(f"{x}")
            self.filter_select.options.append(f"Not {x}")
        self.tabulator.add_filter(
            pn.bind(
                self.__filter_table, pattern=self.filter_select, column="pl_discmethod"
            )
        )

        # Final Page

        self.dashboard = pn.Column(self.filter_select, self.tabulator)
        self.dashboard.servable()

    def cb_select(self, *events):
        print(f"[Selection] old: {events[0].old}, new: {events[0].new}")
        if events[0].new:
            if not self.filter_active:
                print(f"[Selection]\n{self.tabulator.value.iloc[events[0].new[0]]}")
            else:
                print(
                    f"[Selection] filter is active. Expected {self.filtered_df.iloc[events[0].new[0]]['rowid']}."
                    + f" Got: {self.tabulator.value.iloc[events[0].new[0]]['rowid']}."
                )

    def cb_onFilter(self, *events):
        if events[1].new:
            print(f"[Filter] Filter changed: {events[1].new}")
            self.filter_active = True
        else:
            print(f"[Filter] All filters removed")
            self.filter_active = False

    def __filter_table(self, df, pattern, column):
        for p_item in pattern:
            if "Not " == p_item[0:4]:
                pattern_mod = p_item[4:]
                df = df[df[column].apply(lambda x: pattern_mod not in x)]
            else:
                df = df[df[column].apply(lambda x: p_item in x)]
        self.filtered_df = df
        return df

Screenshots or screencasts of the bug in action

https://user-images.githubusercontent.com/9201509/200301976-93782dca-dfa3-41dd-af90-f28ebdb3d626.mov

[BeZie]

Hi all, I encounter the very same bug with panel version 1.3.0.
Is there any work around at the moment?

[sytham]

You can solve / work around your problem as follows. event.obj.current_view will give you the current, filtered view of the data. You can index into that using event.new, which is the iloc of the selection.

However, there is still a bug in tabulator. The indexing on a selection event (so what's in event.new) is inconsistent. When the data is sorted (by clicking a header label), event.new will give the location of the selected row in the original, unsorted view. So in this case, we need to index into the df using tabulator.value (or event.obj.value). If that weren't the case we could always use event.obj.current_view. So it either needs to always provide the iloc of the current view, or always the index of the original view, but not mix them.

It's possible to work around this by writing custom code that inspects event.obj.filters and event.obj.sorters, but better to fix the behavior of course.

[philippjfr]

Sorry for the long, long delay here. This has now been fixed in https://github.com/holoviz/panel/pull/7058