Separate source and display in TableView and TreeView

Is your feature request related to a problem? Please describe. Currently, the Cocoa backend is the only implementation to support arbitrary widgets in a TableView or TreeView. This is done by just returning a widget instead of a string or (icon, string) tuple in place of the data to be displayed. This is possible because the native NSTableView and NSTreeView are "view-based", i.e., their cells can display any Cocoa view.

This implementation cannot be copied to other platforms: in Windows, there is no direct support of arbitrary widgets in a Tree / Table (see #841). Gtk is more flexible by allowing data provided by the source to be rendered by a custom Gtk.CellRenderer. It does require an entire column to have the same renderer and is therefore less flexible than Cocoa.

Finally, a significant issue with the current Cocoa implementation is that the widget to be shown is defined by the source. It therefore is both the data and its representation. This becomes problematic for instance when displaying the same data by multiple views, as in the table example. Since a widget can only be drawn on screen once, all but one table will have an empty cell instead.

Describe the solution you'd like It would be good to have a uniform solution for table cells across platforms which can display more than just a string and an icon. I like Gtk's concept of defining a "renderer" or view for each column to represent the data and providing several commonly used renderers as default. Those could be:

icon + text
checkbox + text
progress bar
...

The user then provides methods to translate the column data to input for the renderer. This can be as simple as "bool -> checked state" or more complex such as "str -> icon".

In addition, one could allow the user to define custom views, though this will be more difficult to achieve in a cross-platform fashion. I envisage subclass of toga.Widget which takes column values during init and will be used by toga to construct a cell view for every platform.

Describe alternatives you've considered I'm open to suggestions. The very different platform implementations complicate this a lot.

+1 on the idea.

Let me know how can I help in order to make this happen 😃.

Definitely +1 on this idea; and also recognising that the devil is in the API details :-)

My immediate thought is that this is an extension of the idea of data sources. The core idea behind a data source is that you can abstract "what is displayed" from "how it is displayed". In the read-only view of the world, a data source needs to be able to describe a "row" of data, and how to get attributes of that row; exposing widgets means you're describing how to set attributes of that row.

So - if a data row described itself as "three elements long", we can currently display a three column table. If we also added an annotation which allowed the row to describe "column 0 is readonly; column 1 is a boolean; column 2 is text", that then defines the UI that can be used to render each cell. It then becomes the job of the table/tree to answer the question "how can I display an editable a boolean/text field".

This aligns reasonably well with the idea of GTK's cell renderers; the annotation types effectively maps to "what widget allows this content to be edited in that way".

If it helps constrain complexity, I'd be entirely comfortable with massively restricting what is possible in a table/tree cell to a very limited subset of options. There's a limited number of widgets that are going to be useful in this context; while it might be nice to allow end-users to define arbitrarily complex in-table widgets, I think our focus should be on solving the 80% case - and checkboxes, icons, buttons, progress bars, editable text, and maybe a handful of others covers that.

Yes, this may in fact be best defined as annotation of the columns in the data source. Properties such as "column is boolean" or "column is editable" are indeed properties of the data source and not the display.

I can see one limitation of this approach but we (you) may decide that its ok to live with it: A user may want to present the same type of data with different widgets. For instance, a percentage may be represented as a label with text "90%" or a progress bar.

I am also +1 on solving the 80% of cases and providing the tools for editing the data through the table. An API for custom displays would be a very powerful tool but I am already getting a headache when thinking about possible platform implementations. This may be something to consider for a future release...

I have been thinking about this some more and have come up with a draft suggestion: Instead of cells being represented by a str or a (icon, str) tuple, we could have a Cell class with a fixed set of attributes that represent the data type. Then there are two types of user interfaces to populate a data source:

Cells are directly created by the user and added to a Row or Node which is then added to the ListSource or TreeSource, respectively. This provides maximum control to the user and requires exposing Cell, Row and Node classes in the public API.
Cells, Rows and Nodes are created automatically for the user from a list or dictionary, similar to the current API. Reasonable defaults for cell attributes can be chosen depending on the value type (e.g., str -> text, bool -> checked_state) or one could play it safe and always opt for displaying the value as text.

In this concept, a Cell can have multiple attributes such as an icon, a checkbox and text. One would typically use at most two of those and set the others to None but this is not required. The value attribute will always be displayed as text and optional functions value_to_str and str_to_value can be provided for the CellView to use when converting the value to a string or when converting the edited text back to a value. The latter will also act as a validator.

For example:

class Cell:
    def __init__(self,
        value="",
        value_to_str=lambda x: str(x),
        str_to_value=lambda x: x,
        is_editable=False,
        icon=None,
        check_state=None,
        style=None,
        progress=None,
        activity=None,
    ):
        self._row = None
        self.value = value
        self.value_to_str = value_to_str
        self.str_to_value = str_to_value
        self.is_editable = is_editable
        self.icon = icon
        self.check_state = check_state
        self.style = style
        self.progress = progress
        self.activity = activity

    @property
    def value(self):
        return self._value

    @value.setter
    def value(self, value):
        self._value = value
        self._notify_change()

        .
        .
        .

    def _notify_change(self):
        if self._row is not None and self._row._source is not None:
            self._row._source._notify("change", item=self._row)

    def __repr__(self):
        return f"<Cell(value={self.value})>"

Usage of the first API could look like this:

row = Row(
    filename=Cell(
        value='README.rst',
        icon=toga.Icon(icon_path),
    ),
    mtime=Cell(
        value=1873.23,
        value_to_str=lambda x: datetime.fromtimestamp(x).strftime("%d %b %Y at %H:%M"),
    ),
)
source = ListSource(data=[row], accessors=['filename', 'mtime'])
source.append(Row(filename=Cell('CHANGELOG.rst'), mtime=Cell(2873.23)))

Usage of the second API could look like this:

source = ListSource(data=[('readme.md', 1873.23)], accessors=['filename', 'mtime'])
source.append('CHANGELOG.rst', 2873.23)
print(source[0])  # returns <Row(<Cell(value='CHANGELOG.rst')>, <Cell(value=2873.23)>)>

What do the two of you think? I have tried to find a balance between avoiding complexity and flexibility.

The downside of this approach is the relatively complicated structure of the Cell and its limitation to a specific number of attributes.

An alternative, closer to Gtk's approach, is to decouple the data source columns from the actual columns in the view. Every data source column holds only a single value per row (e.g., one column for icons and another for names). The view then defines renderers for every "ColumnView" which can take values from multiple columns in the data source.

This provides improved versatility because the data source no longer cares where an icon is displayed, it only provides a column with icons. It is then up to the View to choose where to display the icons: in their own column or combined with some text in another column. The downside is increased complexity: one would need to provide toga.ColumnView and toga.CellRenderer classes which integrate toga.Tree and toga.Table.

Edit: This API also has the advantage that the user can set column widths and other properties.

My immediate reaction is that Cell isn't something that should have any association with a data source. Data is data; Cell is presentation layer. The best way to expose the problem is something like the table demo that has 2 tables viewing one data source, viewing a boolean value. One table may want to display the boolean as a tick/cross image in a readonly capacity; the other may want a read/write checkbox. The display choice shouldn't require a different row definition.

However - your Cell concept does map well onto the way accessors are defined. The current API allows you to define an accessors list that allows you to describe which attribute of the data source should be used to populate the table value. We then do an implied conversion using str() to render the value, and allow the special case of a tuple to include an icon.

I'd argue what you're calling Cell is really the union of that accessor, an explicit definition of the conversion mechanism (e.g., do I want my boolean to display as "True/False", "Yes/No", or as an image?), a description of the input mechanism (if any), and how the value from that input should be persisted back onto the data source (e.g., if a text input needs to be cast to a float).

I'd argue what you're calling Cell is really the union of that accessor, an explicit definition of the conversion mechanism [...], a description of the input mechanism (if any), and how the value from that input should be persisted back onto the data source [...].

I agree. The downside of the Cell approach is that it combines many roles which should better be separated.

Accessors: I have ambivalent feelings towards them. I understand the convenience of iterating through the source to retrieve the rows / nodes and having the class attributes of a row / node return the data values. However, having dynamically named attributes instead of just column indices in a "matrix" or "tree" requires a lot of getattr() calls and may result in more complicated code. It's a nice API for the user though, unless they want to subclass or write their own sources.

In any case, the accessors should continue to be defined by the data source.
Conversion mechanism: This is probably better defined at the "column view" or Table / Tree level. As you say, different views may present data in different ways and will therefore need different conversion mechanisms for "data -> view".
Input mechanism: Again, this is probably better defined at the "column view" or Table / Tree level. Different views may provide different editing interfaces and will therefore need different conversion mechanisms for "view -> data".

I have put together an example in #1085 of how the second option with the definition of table columns and renderers could work . The Tree / Table basically now deals with Columns in the interface layer. Those have properties such as min_width, max_width and a renderer. This renderer entirely defines how a table column should be populated from the data source, taking values from one or more accessors in the data source and using them to display text, icons, checkboxes, etc.

beeware / toga

Separate source and display in TableView and TreeView #1066