Timestamp columns in pandas ag-grid breaks event-handlers

[kirbinator8]

in the "Grid Events" example 1, the intended behavior (populating the div with row data) doesn't occur when timestamp columns exist in the dataframe.

[elimintz]

Could you please create a small working example to illustrate the problem?

[kirbinator8]

`import justpy as jp import pandas as pd

df_object = {'row1': [], 'row2': [], 'row3': [], 'row4': [], 'row5': []} df_object['row1'] = [pd.Timestamp(year=2016, month=4, day=8)]5 df_object['row2'] = ['100']5 df_object['row3'] = [100.345]5 df_object['row4'] = [3]5 df_object['row5'] = [100]*5

df = pd.DataFrame(df_object)

def row_selected(self, msg): print(msg) if msg.selected: self.row_data_div.text = msg.data self.row_selected = msg.rowIndex elif self.row_selected == msg.rowIndex: self.row_data_div.text = ''

def grid_test(): wp = jp.WebPage() row_data_div = jp.Div(a=wp) grid = df.jp.ag_grid(a=wp) grid.row_data_div = row_data_div grid.on('rowSelected', row_selected) grid.options.columnDefs[0].checkboxSelection = True return wp

jp.justpy(grid_test)`

[elimintz]

Thank you. This is a very weird bug. Something goes wrong in the front end but no error message is created. I'll keep working on this.

[bapowell]

The same thing happens when a UUID object column exists in the DataFrame.

[WolfgangFahl]

Your example is now in examples/issues and I can see how it is not working:

it would be good to see the expected behavior and be able e.g. to switch the two modes with a button

[WolfgangFahl]

I wonder whether the datatype handling while loading is linked to this:

def load_pandas_frame(self, df):
        assert _has_pandas, f"Pandas not installed, cannot load frame"
        self.options.columnDefs = []
        for i in df.columns:
            if is_numeric_dtype(df[i]):
                col_filter = "agNumberColumnFilter"
            elif is_datetime64_any_dtype(df[i]):
                col_filter = "agDateColumnFilter"
            else:
                col_filter = True   # Use default filter
            self.options.columnDefs.append(Dict({'field': i, 'filter': col_filter}))
        # Change NaN and similar to None for JSON compatibility
        self.options.rowData = df.replace([np.inf, -np.inf], [sys.float_info.max, -sys.float_info.max]).where(pd.notnull(df), None).to_dict('records')

[tholzheim]

The issue is that Timestamp is not JSON serializable. During the update process the justpy components are converted to a dict and then to json to send it via websockets:

Component Conversion

https://github.com/elimintz/justpy/blob/df309d80823f8e1fbccac17994876648ccbbb53e/justpy/htmlcomponents.py#L195 https://github.com/elimintz/justpy/blob/df309d80823f8e1fbccac17994876648ccbbb53e/justpy/htmlcomponents.py#L228 https://github.com/elimintz/justpy/blob/df309d80823f8e1fbccac17994876648ccbbb53e/justpy/gridcomponents.py#L113

Sending the updates via websocket

https://github.com/elimintz/justpy/blob/df309d80823f8e1fbccac17994876648ccbbb53e/justpy/htmlcomponents.py#L205

async def send_json(self, data: typing.Any, mode: str = "text") -> None:
        if mode not in {"text", "binary"}:
            raise RuntimeError('The "mode" argument should be "text" or "binary".')
        text = json.dumps(data)

Here the Error occurs and is not caught properly see https://github.com/encode/starlette/blob/bd219edc4571806edf80fd6a48c8ac3fbbadcf22/starlette/websockets.py#L171

Solution

Adjust AgGrid convert_object_to_dict to:

    def convert_object_to_dict(self):

        d = {}
        d['vue_type'] = self.vue_type
        d['id'] = self.id
        d['show'] = self.show
        d['classes'] = self.classes + ' ' + self.theme
        d['style'] = self.style
        options = self.options.deepcopy()
        for row in options.get("rowData", []):
            for k, v in row.items():
                if isinstance(v, Timestamp):
                    row[k] = str(v)
        d['def'] = options
        d['auto_size'] = self.auto_size
        d['events'] = self.events
        d['html_columns'] = self.html_columns
        d['evaluate'] = self.evaluate
        return d

[elimintz]

@tholzheim thanks for figuring this out. Instead of converting Timestamp to str, what do you think about converting it to milliseconds since the epoch?

[bapowell]

For types that are known to be non-serializable, e.g. Timestamp and GUID, I think it's fine to default to a str() conversion, as @tholzheim shows.

But it would also be nice to specify an optional conversion function that would be called for each row item. That way the user could specify, in a custom way, how each item gets represented in the row. For example, Timestamp could be converted to epoch milliseconds.

[WolfgangFahl]

@bapowell would you please add a different issue / Pull Request for your suggestion. I am closing this issue for the specific timestamp problem has been solved.