Navigable table, aka tabular list

[kostmo]

Would like to have a list widget combined with a table's ability to display data in columns.

Here's one other request I've found for this functionality.

[jtdaugherty]

HI @kostmo, thanks for opening this.

I'm not exactly sure what you have in mind, but I'd be happy to talk through it here to learn more. For context, my intention is for the library to provide enough primitives to make things like this possible, and if it turns out that new primitives are needed, I'm happy to consider those and to add them. This sounds pretty high-level, though, so while I suspect I'd be inclined to leave it out of the core library, I'd still be happy to help figure out how to make it possible.

With that said, there is nothing stopping the list widget from displaying data in a columnar format since the rendering function for list items can be written to do that easily. The reddit post you mentioned predates Brick's new table module, so perhaps that would be useful (in a viewport). If not, then I'd want to know more about the use case before I can help.

[jtdaugherty]

@kostmo I just wanted to check in since it has been a while since this was opened. Is this something you'd still like to discuss?

[amano-kenji]

I'd like to discuss this. I want to see a fixed header that can be scrolled horizontally, but can't be hidden by vertical scrolling.

Without a fixed header, it's difficult to know what each column means in a tabular list. The tabular list should be horizontally scrollable because I may want to have a lot of colums that can't fit in the width of a box.

How can each list item show columns that have the same widths? Can such a list be horizontally scrolled?

Can I put a one-row table above a list with columns whose widths match those of the one-row table in a horizontally scrollable box? Then, I can simulate a horizontally scrollable tabular list with a fixed header.

If I scroll down or up a dynamically generated tabular list, column widths may change, and the horizontal scroll may be beyond the width of the line.

[amano-kenji]

I also want a list of input boxes in a vertically scrollable box. I may have a lot of input boxes that can't fit in a limited vertical space. I don't know how to dynamically divide input boxes into pages calculated by the available vertical space.

Otherwise, I would have to show one input box at a time.

[jtdaugherty]

@amano-kenji thanks for sharing your thoughts. It sounds like what you want is effectively a spreadsheet UI. There's a lot that could go into doing such a thing, and it would be complex enough that it goes well beyond the bounds of what I'd like to provide as built-in functionality in the library. However, I'm happy to provide my thoughts if you are trying to implement various aspects of that and would like some help.

Although the library is not going to provide something of this level of sophistication out of the box, I could definitely see it being something that could be built and distributed as a separate package. Are either you intending to work on such a thing in your own application? Or are you hoping that by making the request here, I'd be the one doing the implementation? While I am tempted by the challenge, it would be helpful to me to get clarity on your objectives before we get into discussion of the implementation issues.

[jtdaugherty]

I also want to point out that @kostmo's original request does not sound quite as involved as @amano-kenji's description, so we also need to be sure we are not discussing two different features on the same ticket.

[amano-kenji]

Since I'm the one who wants it more than you do, I would be the one who will do it if the idea takes off.

Note that I haven't learned brick, yet. But, I'm going to learn it soon for my application.

I don't necessarily have to select each column because when I press enter on a spreadsheet line, my application can show a plain vertical list of column items which can be edited by pressing enter and entering a new value in a pop-up dialog.

But, I want to have a fixed header and be able to horizontally scroll because that's necessary for a long list of columns that won't fit in a narrow box. A fixed header needs to be shown regardless of any amount of vertical scroll.

I want the column widths to be automatically calculated by the longest column items of all available rows. I don't want to truncate long column items. My application is going to fetch rows dynamically from SQLite.

Obviously, spreadsheet UI is better for me, but a horizontally scrollable tabular list with automatically resized columns can do.

[jtdaugherty]

Note that I haven't learned brick, yet. But, I'm going to learn it soon for my application.

Okay. In that case, it might be best to hold off on further technical discussion until you've been able to spend some time learning the library. That will help put my technical suggestions into context.

[kostmo]

Thanks for checking in on this, @jtdaugherty.

there is nothing stopping the list widget from displaying data in a columnar format since the rendering function for list items can be written to do that easily

The first question I had was whether it makes sense to try to augment a Table widget with list navigation capability, or to augment a List widget with column formatting. I suspected the latter would make more sense, and your statement seems to confirm it.

Then the next question is whether there exist any exposed (or exposable) APIs from the Table widget that can be used, or if we would have to re-implement it ourselves. The crux is incorporating "global" knowledge (i.e. about all of the items in the list) for the "local" (single) list-item display to determine column widths. Bonus points if it can consider just the subset of rows that are visible in the current viewport.

[jtdaugherty]

@kostmo in the use case you have in mind, how many rows of cells will this table-as-list have in it, on average?

[kostmo]

Probably on the order of 100.

[jtdaugherty]

Will the cells' contents all obey the List's restriction that they all have identical height?

[jtdaugherty]

And will you want any row/column/table borders to be drawn at all?

[kostmo]

Same height, no borders. A sticky "header" row as suggested above would be useful.

[jtdaugherty]

The first question I had was whether it makes sense to try to augment a Table widget with list navigation capability, or to augment a List widget with column formatting. I suspected the latter would make more sense, and your statement seems to confirm it.

The latter option might work. Either option runs the risk of straining the original abstractions past what they're intended for, but if I had to pick the option that I think has the best composability story, it's to make it possible to break tables up in a way that could then be subjected to list navigation. (There's still the question of how to deal with cursor at a particular cell offset within the selected row, but that is straightforward application-level book-keeping.)

Then the next question is whether there exist any exposed (or exposable) APIs from the Table widget that can be used, or if we would have to re-implement it ourselves. The crux is incorporating "global" knowledge (i.e. about all of the items in the list) for the "local" (single) list-item display to determine column widths. Bonus points if it can consider just the subset of rows that are visible in the current viewport.

To help with this, I've refactored Table.hs to separate the table cell layout from the border rendering in a way that exposes the results of the table layout algorithm in a form that could then be subjected to per-row list item rendering. The result is on this branch; see the new Low-level API section of the export list in Table.hs. That work isn't the end of the story for how to get what you want, but it's a necessary step on the path. If you look at the code, you can see now that renderTable is simply a composition of the two phases of processing that are now exposed in the API.

[jtdaugherty]

If you actually tried to use that branch to start hacking on a table-list-like-thing, the first problem you'd run into is that the table cell layout produces results in RenderM, which is essentially too late for the purposes of being composed with List, which is constructed prior to rendering time. I'm going to put more thought into how to deal with that.

[jtdaugherty]

(N.B.: producing results in RenderM is necessary and can't be done earlier, so it's possible that a hacked version of List will still be needed here.)

[jtdaugherty]

Thinking about this more, a fundamental challenge in doing this efficiently (i.e. only showing as much as is needed to fill the viewport) is that for table layout in general, the entire table must be rendered in order to know the widths of all of the columns. This is the "global knowledge" you mentioned. That makes it impossible to choose only a subset of rows to render; even if their heights are known in advance (via the List constraint that the height must be known and must be identical for all rows), the widths of the columns can't be known until their contents are rendered. That means we can't get the benefit of the hack that the List uses for viewport efficiency, which is to choose 2*N + 1 rows to render, for viewport height N, and then translate the result. To render those rows we'd still need to render all rows in the table regardless of the viewport height.

That makes me think that rather than using the Table machinery, which gets column width information from the totality of the table, by far the simpler thing to do will be to just write your list rendering function with some foreknowledge of the "table" columns and their widths. I still think the refactoring of the table machinery that I did has some value since it allows people to do what they want with just the result of the cell layout.

Since the table machinery still does other valuable things like dealing with column alignment, I could expose some of that functionality in a way that would make it usable if you knew your desired column widths. That could make it possible to write list row rendering functions that take advantage of the alignment features without using the full-blown table layout implementation.

[jtdaugherty]

Another thing I'll throw out here, just in case it helps: if, in practice, your 100 rows or so will render quickly enough, then it's also possible to skip using a List at all and use an approach like the one in this demo program. That program does something very similar to a table-like list but uses neither List nor Table. Its main drawback is that the approach used there is ignorant of viewport size (and cannot be otherwise), making it the simplest to implement but also the least efficient as the viewport contents grow larger. Using that approach with the new table API would essentially involve wrapping the cell contents of the "selected" cell with visible as is done in that demo. The viewport translation logic takes care of the rest.

[jtdaugherty]

That branch now has an alignColumns function to help with what I described above.

[kostmo]

Perhaps to reframe the problem a bit, the current type signature of renderList is:

:: (Traversable t, Splittable t, Ord n, Show n) |  
=> (Bool -> e -> Widget n) -- ^ Rendering function
-> Bool
-> GenericList n t e
-> Widget n

I could imagine a tabular-capable API looking something like this:

data ColumnWidthStability
  = DynamicForViewport
  | StaticAcrossAllRows

:: (Traversable t, Splittable t, Ord n, Show n) |  
=> ColumnWidthStability
-> ([e] -> a) -- ^ Summary function
-> (a -> Bool -> e -> Widget n) -- ^ Rendering function
-> Bool
-> GenericList n t e
-> Widget n

Internally, the rendering machinery would invoke the "summary function" just once per display cycle, with many (or all) rows as input. This user-defined function would typically compute the max width of the contents of each cell, storing this result in a record of type a. If the ColumnWidthStability is DynamicForViewport, then only those rows visible in the current viewport are supplied to the summary function. If StaticAcrossAllRows, then all of the rows of the table are included.

Then, the rendering function for individual rows would have access to the summary a and use it to render column widths of each cell.

The effect of scrolling through the table in DynamicForViewport mode may be for the columns to jump around in position, depending on whether particularly long cells pop into view.

[jtdaugherty]

The effect of scrolling through the table in DynamicForViewport mode may be for the columns to jump around in position, depending on whether particularly long cells pop into view.

It'll also jump around before they come into view since some of the rows consulted to prepare the rendering will be off-screen. (I could go into more detail about why, but if you are curious then you can read the comments in List.hs that go into detail about how that works and why it's necessary.)

The above approach also only works if the foreknowledge a can be obtained from the values [e] without rendering them. If that's true, then today's List API could probably be used more or less as-is by using the alignColumns function as part of the body of the list rendering function; the foreknowledge could be computed and stored in your application state whenever the "table" contents change and could be consulted by the function that renders the list items.

[jtdaugherty]

I just pushed a patch to add a proof-of-concept tabular list demo program using some of the ideas I talked about above; see c95baf63e625f7caace78752dd369e7e9cc3e3d9.

[kostmo]

Tried it, very cool. FYI for my use case I had not intended to have navigation across individual cells; I plan only to have navigation between rows, and only care about proper alignment across columns.

[jtdaugherty]

Hah, well, in that case we might have saved some time by clarifying that up front! ;) In that case I am even more convinced that the approach taken in the demo (further simplified) could be adequate for what you want to do without using Table at all.

[jtdaugherty]

It's also possible I read too much into the ticket title along with @amano-kenji's comments and went to thinking about cell-by-cell navigation. In any case, even without that requirement, I think the changes I made along the way are useful enough to merge to master and release.

[jtdaugherty]

I want to point out that this is always the pitfall with solution-first discussion rather than problem-first; the idea to use Table ended up being a bit of a diversion given that it sounds like the basic need was much simpler. :)

The demo program provides a solution that gets the performance benefits of List while also giving the rendering appearance of some aspects of Table. Let me know if there is more to sort out in that demo to work in your use case!

[jtdaugherty]

@amano-kenji at this point it sounds like your use case is somewhat different. If you agree, would you be willing to open a new issue where we can focus on that?

[amano-kenji]

After I read brick user guide, I may or may not open a new issue.

[amano-kenji]

It took a while to learn beam, lens, generic-lens, optics (which has built-in generic support), and brick. I read brick user guide and gained a rough understanding of tabular list demo.

I seriously think that a spreadsheet UI should be designed and used as a tabular list. The tabular list demo gets close, but I don't think it's complete.

As new rows are added, maximum width for each cell can be computed and saved in the spreadsheet state. Some people may choose fixed widths for columns and truncate long column contents.

However, a row can have multiple lines. I don't know whether list can handle rendering only visible rows that span variable numbers of lines.

There can be millions of rows. It can be slow to draw them all.

We have to assume that columns can and will overflow the width of a viewport. If columns don't overflow the width of a viewport, then the last column should fill the gap greedily. If users reduce the size of terminal, columns will overflow the spreadsheet viewport. To use a spreadsheet as a tabular list, one can just scroll horizontally over fixed-width or variable-width columns instead of navigating cells.

I suggest changing this issue to design of spreadsheet UI. Let's solve spreadsheet problem permanently, and you get tabular list as well. I'm going to focus solely on spreadsheet problem until it's solved.

[amano-kenji]

Why don't we just draw incrementally and see whether the result outgrows the viewport?

I don't know whether visibility request can be used.

[jtdaugherty]

I suggest changing this issue to design of spreadsheet UI. Let's solve spreadsheet problem permanently, and you get tabular list as well.

I'm not sure what you mean by "solve the spreadsheet problem permanently." Implementing a spreadsheet in brick will be a significant undertaking. It will not be provided by a built-in component in the library since that vastly exceeds the scope of what I want the library to provide. Many implementation issues will arise, and I'll be willing to help with some of them as they relate to proper and effective use of the library.

As far as this issue is concerned, let's keep it focused on @kostmo's original request. I believe I've satisfied that and I'm going to rely on @kostmo to close this issue if that's true. @amano-kenji - if you would like to implement a spreadsheet application and would like assistance, please open a new issue specifically for that.

Thanks!

[amano-kenji]

Okay, let's discuss in a new issue.

[amano-kenji]

In the latest iteration of my tabular list design, I got two kinds of tabular lists.

• Grid tabular list
  • Every row has the same columns.
  • You must navigate column by column.
  • Column widths are fixed, but they can be modified in EventM. The type for column widths is simply Seq Int.
  • The widget tries to display the current column in the center of viewport. Center rendering algorithm is statelss and doesn't require scroll offset. Center rendering algorithm turned out to be very easy to implement in RenderM. It is also fast.
  • If the current column is wider than the viewport, the current column width is made to equal the viewport width.
• Mixed tabular list
  • There can be more than one row kind.
  • Each row kind can have a different number of columns than the other row kinds.
  • A row can be of a row kind or another.
  • Navigating columns is not possible.
  • All columns must fit in the viewport. Thus, column widths for each row kind are dynamically calculated in RenderM with Context n -> Map rowKind [Maybe Int]. The type for calculating column widths dynamically is Map rowKind [Maybe Int] -> row -> [Maybe Int]. Some columns may be omitted if the viewport is too narrow to show them.

I think I largely finished designing data structures and the API. I just have to fully flesh out the widgets. I think mixed tabular list is what you are looking for.

I got this covered.

Grid tabular list can handle spreadsheets and database UIs, but it can't handle merging two or more columns. I think merging rows or columns requires a fundamentally different design that doesn't build on top of GenericList. And, I don't need to merge rows or columns.

Mixed tabular list is just a list with tabular display formats.

They were designed to be as fast, ergonomic, and efficient in memory usage as reasonably possible.

I think you can close this issue. The name of the widget will be brick-tabular-list on hackage.

[jtdaugherty]

I think you can close this issue.

Thanks @amano-kenji but did you mean #422? I'm leaving this one open to get confirmation from @kostmo.

[amano-kenji]

I would keep #422 open just in case until I finish implementing the widgets.

[amano-kenji]

I finished implementing GridTabularList and MixedTabularList. Now, I just need to add documentation and publish brick-tabular-list to hackage. I'm learning haddock now.

GridTabularList

MixedTabularList

Row headers(red) and column headers(blue) are optional. You can navigate cell by cell in GridTabularList. All columns on MixedTabularList are supposed to fit in the visible viewport.

MixedTabularList doesn't support cell-by-cell navigation. It is just a list with each row in tabular display format. Column widths on MixedTabularList are calculated dynamically during rendering from the available width given to the list and optionally the visible rows.

Column widths on GridTabularList are fixed in rendering, but you can change them in event handlers.

Implementing them was hard, but after fleshing them out fully, they turned out to be simple. Making them simple was hard. My tabular list widgets handle narrow terminal widths gracefully.

I think this issue should be closed after I publish brick-tabular-list on hackage. Agreed?

[amano-kenji]

https://hackage.haskell.org/package/brick-tabular-list has been published.

[jtdaugherty]

@kostmo I'd still like to get confirmation from you that you have what you need, but I'm going to close this due to inactivity. Please re-open it if you need something else on this topic!

[amano-kenji]

My library is already nearly perfect. When GHC allows DeriveGeneric for existential data types and GADTs, I will hide a type variable or two inside a data type. Apart from that, I think it's complete. I think brick uses GADTs and doesn't use Generic much. My library uses Generic from top to bottom.

[jtdaugherty]

@amano-kenji that's great, but please note that I am specifically wanting to hear from the ticket's original author as to whether their needs are met.