Moving from less to curses in order to display wide tables?

[jackmaney]

Note: This is just a question, and implementing it would require a complete rewrite of the draw functionality. So, if you're not interested, I understand.

Since so many Excel files and CSVs are too wide to fit reasonably in one screen (without wrapping), I believe we could scrap the piped out call to less and replace the view portion of the app with a curses application. I have no experience with curses directly, but glancing through this tutorial for the curses package, it appears as though we can:

• Replace the termsize function with a call to getmaxyx on a curses window (testing on my system gave me the same sizes).
• Load the entire table into a pad, and keep the window coordinates of each cell, so that, eg:
  • Pressing the right or left arrows will move right or left one column (respectively)
  • Pressing the up or down arrows will move up or down one row (respectively)
  • Pressing the space bar will move down one page
• I believe we can even keep track of the portion of the header that needs to be displayed, and automatically display it on the top of the screen with each window refresh.

One drawback is that it would be a non-trivial endeavor to implement the rest of the less functionality (eg regex searches). However, since the aim of this app is to get a basic view of data, that loss might be worth it in exchange for scrolling to the left and right.

Thoughts?

[wcastello]

I thought about doing this the moment I saw this program but I had so many things going on that I couldn't start. Definitely I would like to see it made using curses, the view would be far superior from what less provides IMHO. One thing that could be done nicely using courses is creating new windows and using them to view workbooks with multiple worksheets. You could even keep the header static and refresh only the other rows. I personally wouldn't mind about the regex searches of less for now.

[kristianperkins]

Sounds like a great idea if you are up to it :). Less just happens to be an easy way to get paging working.

You should be able to scroll left and right with less though as the -S option is being used to force less not to wrap lines. You should be able to scroll left or right to see the rest of the line unless you have a version of less that doesn't support -S fully?

Having said all that I think curses solution would be worth it just to have the headers fixed at the top of the screen rather than printing them once per page.

[jackmaney]

*facepalm* I don't know why I didn't try scrolling to the left or right in x_x as is.

I agree, though, that it would be better to have a fixed header (and universally-sized columns). And, as has been pointed out by @wcastello, a curses approach would allow for switching between worksheets when viewing Excel files.

I've spent some of today reading up a bit and building a few various base classes that allow for more cleanly printing an entire table ("table" being either an entire worksheet or CSV file).

I also want to emulate what seems to be a feature in less -S horizontally "snapping" to the next column (eg if the column just to the right of what's displayed is fairly wide, I want to display all of it when hitting the right arrow key, not a few characters at a time).

I'll keep pushing my changes to a curses branch and will make a PR once I'm ready.

[kristianperkins]

btw, it may be helpful to give some history on the current implementation :smile:...

asciitable.py was written by @jaysw to display database result sets for the ipython extention ipydb. This should explain why the XCursor class exists in this project, which is to emulate a database cursor. All we really needed was an iterable that also knew what the column headings were (the keys() function) and that's what XCursor does. Feel free to change that up as you see fit.

[kristianperkins]

Also, from the ipydb perspective, one advantage of using less is that is doesn't read everything into memory straight away. If you run ipydb and do select * from all_the_things then less will read all it needs from the db cursor and will only read more when you page down. So less only reads as much as it needs to display (plus maybe a K or two).

What I mean to say is that this project doesn't require buffering at all so we can just read all rows into memory for x_x. To push this change back to ipydb it may be necessary to only read the cursor/iterable what needs to be displayed.

FYI: I haven't talked to @jaysw about this issue and only assume he agreees :smile:

[jackmaney]

I haven't forgotten about this, by the way...just been busy with work and life.

I'll see if I can make some progress later this week.