RFE: Create a terminal widget that isn't dependent on Fl_Text_Buffer

[erco77]

Opening this issue for development of a new terminal widget that is self contained, which in this fork is in branch "Fl_Terminal", specifically src/Fl_Terminal.cxx and FL/Fl_Terminal.H and the development test program test/testterm.cxx

For dev it's currently named Fl_Terminal just to not conflict with Fl_Simple_Terminal. But it's possible it could be used to replace Fl_Simple_Terminal.

[erco77]

Current state of things: it should build and run the test program.

To try it out, first build fltk with the usual configure/make. Do that just once.

Then go into the examples directory; my edit/build cycle has been:

vi ../*/Fl_Terminal.{cxx,H}
touch testterm.cxx && make testterm && ./testterm

So as I add commits, you should only need that touch/make to rebuild the app, without needing to rebuild fltk.

What works so far is:

• Printing ascii and/or utf8 text
• The only ctrl characters that work so far are '\n'. Tab, backspace, etc to come.
• Colors and attributes for text currently only work via method setting. (Esc control to come)
• The only attributes implemented are Bold and Normal. Underline etc. are to come
• In the test app, hitting up/dn/lt/rt arrow keys moves the cursor around to test its drawing over unicode. (There is an issue around the Chinese characters, as they are wider than the other chars, so moving up/dn into/out of them jumps around the more to the right of those strings you test)
• In the test app hitting 's' runs a 'speed test'. This repeatedly fills the screen with random ascii chars/colors using the single character printing routine, then tripping redraw() after each screenfull. It repeats 50 times as fast as it can, then stops and redraws the test screen when done. Hopefully that level of snappiness can be maintained to final release. I imagine redraw()s will have to be tripped by a timer to ensure we don't per-character redraw, as that would be silly and slow things down esp. when there's high output volume.

Currently I'm working on the Esc sequence stuff; my next commits will be fleshing out as much Esc support as possible and adding tests for simple color/attribute control by Esc sequences, as well as cursor movement, and other features common to e.g. curses apps.

I also want to flesh out the as yet unimplemented attributes for Dim, Strike, Underline, and others. Currently only the Bold attribute is supported, and colors.

[Albrecht-S]

Thanks, builds fine and works. Something I'd like to see for testing is CMake support which needs only one line in examples/CMakeLists.txt:

diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt
index 2bc33557a..63ff4fefb 100644
--- a/examples/CMakeLists.txt
+++ b/examples/CMakeLists.txt
@@ -57,6 +57,7 @@ set (SIMPLE_SOURCES
   table-with-keynav
   table-with-right-column-stretch-fit
   tabs-simple
+  testterm
   textdisplay-with-colors
   texteditor-simple
   texteditor-with-dynamic-colors

This requires the option FLTK_BUILD_EXAMPLES:BOOL=ON to build but that's fine for me. An alternative would be CREATE_EXAMPLE (testterm ../examples/testterm.cxx fltk) somewhere in test/CMakeLists.txt which would build w/o an extra CMake option but both would be fine (select only one though).

:+1: Thanks for all this, looking forward to your further development... ;-)

[erco77]

Note that testterm is just a hacky early dev thing that will either go away, or reincarnate into a unittest or regression tester. So I wouldn't expect it to be official.

It's in the examples dir only because I'm lazy and know I can run 'make anything' in that dir, and it'll build anything.cxx.

[Albrecht-S]

Note that testterm is just a hacky early dev thing that will either go away, or ...

Sure, but I like using CMake much more than make. As long as it is in your fork it's clear that it can go away at any time. How difficult would it be to remove that single line later? ;-)

BTW, my "build cycle" is to change the relevant file(s) in my editor (VS Code) and then to type F5 to "build and run" (and optionally debug) once you have added the .cxx file to CMake. Since I'm using ninja to build it doesn't matter if you "build" the FLTK lib, if nothing is to do it's instantaneously finished.

But if you don't like to add it, never mind, I have it in my local clone anyway.

[erco77]

Ya, didn't want to add it to keep the diffs just the files I'm working on, so I don't forget anything later.

Progress -- added some multistep tests that will grow as I continue to test + dev. To walk thru the Escape codes test, hit 'e' repeatedly. As of this writing there's 3 Esc test pages:

Implemented italic, underline, "dim" and strikeout attributes. Also added a test for cursor positioning, to make sure ESC[#;#H codes were 1 based and still hitting their predicted marks.

[erco77]

More progress; with commit cde39860d there's now background colors, and both fg/bg colors are affected by Dim/Norm/Bold attributes:

Only 773 lines of code according to cloc(1), so it's staying trim so far.

TODO:

• Screen history, vert scrollbar
• Other esc codes - insert/delete? scroll regions? Window titling (Esc]0;..)?
• Screen resizing
• Deal with all the XXX and TODO and TBD items in the source
• can't think of what else at the moment

[erco77]

Update: with commit 138113c71, did some housekeeping on the code so Fl_Terminal builds as module that's part of the FLTK library; most method code moved to .cxx file, .H no longer includes .cxx, etc.

Now that I'm past the initial development stage, needed to clean that up.

[erco77]

Update: with commit 326a85c9b, added onscreen mouse selection (left-click + drag):

No pastebuffer yet, need to add ^C and a popup menu for a "Copy" option. Todo: Shift-Click to extend existing selection.

Also, with commit 4c47f88a4 added Inverse attribute:

[erco77]

[EDIT: Looks like github support fixed this issue] Hmm, after syncing my fork to FLTK's master using github's '🗘 Sync fork' button, this corrupted my fork on github in a way I can't push/pull or even clone the repo now:

$ git clone git@github.com:erco77/fltk.git erco77-fltk
Cloning into 'erco77-fltk'...
remote: fatal: bad tree object fb52949e082864f043cb3d2a258a825c27827678
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed

Everything I could find about this indicates a corruption on the server side (github).

When I've seen this problem on my side, a git gc --aggressive fixed it, but I'm not aware of how to do such things on the github server. Actually opened a support ticket with github for this, guess I'll see what happens.

[erco77]

Added scrolling in commit 8e4b2d707 (wasn't available while repo was being repaired, seems fixed now)

The testterm speed test ('s') has been modified to exercise scrolling; it now has a second part after the random color blocks that cat's the .cxx file's contents into the terminal a line-at-a-time, showing how fast lines can be added by triggering a redraw() after each new line is added. I'm happy with the speed (faster than I can read), this screenshot showing the last few lines of the file at the end of the test:

No optimizations for damage() yet; the only kind of redraw right now redraws the entire visible buffer. Speed seems pretty good regardless, but I imagine that kind of optimization can be saved for LAST.

In the final release, there may need to be a timer to trigger redraws so they don't happen too fast. For instance, doing a per-char redraw() would be slow in a high-volume printout. I expect a 'reasonable' 30 fps timer default (configurable by the user) would limit "automated" redraws, and the user could also force immediate redraws by calling redraw() after every print command they care about, which would clear such a timer. Basically similar to how we use fflush(stdout) to force output to appear. I might also need to add a flag to control how automated redrawing occurs, the default being a 30fps timer, but other options could be 'per line' or 'per character' with a warning about the latter being slow. The redraw timer would only run if there's new text in the display that hasn't been redrawn yet.

[erco77]

In commit ea1b707c1, mouse selection can now be copied to the paste buffer with (^C) and select all with (^A).

Looks like github support fixed the corruption -- woohoo, I can push again.

PS. Also added middle mouse paste buffer handling in 42683d2d2.

[erco77]

The thing I'm dreading doing is the scrollback history, as there be dragons:

• Mouse selection crossing the boundary between backscroll and 'active' display will be tricky
• Scrolling back while new data is coming in shouldn't cause the scrollback to 'jump' or do weird stuff to mouse selection/scrollbar. Probably need a pre-buffer (see below)
• An in-progress mouse selection will have to properly handle incoming data in a reasonable way

Perhaps while a mouse selection or scrollback is 'in progress', new data coming in will have to be "pre-buffered", and the buffer flushed into the display only when we scroll all the way down to the first few lines of the 'display area'.

Not sure, but probably something like this:

               __________________________       <-- scroll_history_max
              |                          | \
              |                          |  |
              |                          |  |
              |       S c r o l l        |  |
              |                          |  |
              |      H i s t o r y       |  |__ new history_chars[] array
              :                          :  |   (Utf8Chars)
                                            |
              :                          :  |
              |                          |  |
              |                          |  |
              |                          | /
              |__________________________|  _
              |                          |   |
              |       A c t i v e        |   |
              |                          |   |__ current screen_chars[] array
              |      D i s p l a y       |   |   (Utf8Chars)
              |                          |   |
              |__________________________|  _|

Only the 'active display' can be changed by new input from the application, e.g. escape codes/cursor movement followed by text overwriting would only affect the 'active display'. Scrollback buffer is a frozen history that cannot be modified by incoming data. But to the user scrolling around, the history and active display should be seamlessly connected.

I suppose for efficiently the scroll history should be a linked list, although that gets tricky for things like mouse selection which want indexing via row/column concept. Mouse selection doesn't need as much efficiency as scrollback history management for incoming data, so it makes sense to optimize for incoming data, and let mouse selection do the more cpu intensive conversion from row/col indexing into the linked list. For consistency I think I'll keep the screen_history[] array exactly the same as the active screen's screen_chars[] buffer.

In gnome-terminal I noted this behavior:

• If a program is printing one line per second, scrolling up just one line 'freezes' the display, and that output doesn't appear until you scroll back down ALL THE WAY. So I'm guessing while there's a nonzero scrollback, it's buffering the continued output separately, and then when the scroll goes back to the very bottom, the buffer is flushed into the display.
• Interestingly, mouse selection dragged off the top of the screen (causing automatic scrollback) gets cleared whenever there's new output. You /can/ mouse select within the display, whether in the display or in the scrollback buffer, just not during an autoscroll mouse selection. Interesting. Might be a reason for that, or more likely leftover behavior from before the buffering was added.
• To prevent visible 'jumping' of the display when you scroll to the very bottom, I think as you approach the bottom line by doing single line scrolls, single lines are pulled from the buffer and added, causing the scrollbar to reposition so you have to continue scrolling down one line to 'chase' the new data. But when you drag the scroller and yank it down to the bottom, the entire pre-buffer is flushed, and the display jumps up.
• An active mouse selection is adjusted whenever the display scrolls up, to keep it in sync with scrolling. I don't do this yet. This can be seen in the current testterm program if you (a) make a mouse selection, (b) hit the down arrow moving the cursor and inducing a scroll; the text 'scrolls through' the mouse selection, instead of the selection tracking it. Need to modify scroll_up() to handle this. Not sure, but maybe mouse selection should set a special 'attribute bit' in Utf8Char so that mouse selection is on a per-character basis, and therefore easily tracks with scrolling. In which case text_selection() (e.g. Ctrl-C) would have to walk the entire buffer looking for Utf8Chars that have this bit set to build the copy buffer. That's pretty different from what I have now, a start/end row/col paradigm.
• It might be beneficial for the whole screen_chars[] array to be an array of pointers, rather than instances, so pointers can be NULL to skip over drawing trailing white space, keeping the history more memory efficient. Would also speed up drawing to skip these over completely. But actually, thinking about it, that would mean EVERY CHARACTER would have to be checked for NULL to prevent walking into a crash. It'd be better if every unused character was a space, even in the history.

[erco77]

I can tell the "last 10%" of this project is gonna be the hardest, lol.

[Albrecht-S]

OK, here's something for (a) further work and (b) a little relaxation. ;-)

As you may know there are some VT100 "animations" consisting of pure text with escape sequences that "play" some kind of video. I wondered if these animations could be "playable" in Fl_Terminal. The last time I tested it "worked somehow" - not perfect but you could see the basics of the "graphics". If you are interested in more test files: I linked two sites with demo files in my source file (for more see below).

I wrote a tiny program that lets you "play" such an animation in a standard xterm or other terminal program. It's a single C source file (slow.c) that creates a program that can work as a pipe to slow down the input/output to get a speed like an old VT100 terminal connected to a serial line. I'm thinking about building a test program with Fl_Terminal that could load such a text file in the same "slow mode" as in my slow.c demo program. This could be helpful if you want to test even more escape sequences. Anyway, enough for today.

I used 'xmas3.txt' to create this video:

xmas.webm

Merry Christmas!

[erco77]

Ha, that's definitely some old ANSI ASCII art.. used to see that around xmas time on the old VT100 terminals. And Merry Christmas to you BTW!

It shouldn't be hard to get that showing inside Fl_Terminal so that it animates; I imagine reading the file ~8 chars at a time with a small usleep() and redraw() should do the job. Similar to what I'm using to show the .cxx file in the speed test, but in 8 char chunks instead of line-at-a-time with fgets(). Should work..!

[Albrecht-S]

It shouldn't be hard to get that showing inside Fl_Terminal so that it animates; I imagine ...

Yep, as long as all escape sequences are known, hence this would be a good test tool because you can directly open a file and see how it works. I noticed that in my normal "terminal" program on Linux double width and double height characters are not supported, but in xterm I can switch them on as options, as well as line drawing and maybe more.

To implement this I would use Fl::add_timeout() to start, read some chars in the callback, append them to the Fl_Terminal, call redraw(), and then Fl::repeat_timeout() until everything is done. A slider would be provided to adjust the "speed" (i.e. the timeout interval), an Fl_Check_Button to switch unknown escape sequence display on/off, a file chooser to select/open a file, and so on. A nice spare time project. ;-)

[Albrecht-S]

that's definitely some old ANSI ASCII art.. used to see that around xmas time on the old VT100 terminals.

Technically that's VTxxx art. At the time it was created the VT100 escape sequences were (IIRC) still DEC private and were later provided (opened) as an ANSI standard.

But anyway, since I worked with DEC (systems, not in the DEC company) in the '80s and later I had these files too and wrote a script to display this file at login time for all our users around christmas time - as you say. That's why I started this 'slow' project when your escape sequence stuff reminded me of the good old days.

[erco77]

Looks like I have more work to do lol. It mostly looks right, but the shape characters aren't working. xmas3.txt is using some kind of escape codes (or ctrl codes) that I'm not familiar with to shift the font.

I noticed if I hit ^C while cat'ing it to the screen in gnome terminal, it works, but if I interrupt it, it leaves the terminal in a weird state where all ASCII text shows as 'something else', some weird "other font", which is maybe how the graphics are being done, not as utf8 but "something else"..? Even ESC[0m doesn't reset my gnome terminal out of that weird mode -- had to use reset(1) to get things working again. There are a lot of ^N and ^O characters that I know I'm not handling, so that might be a reason it doesn't work -- will have to look that up.

[erco77]

Yeah I don't plan to support things like double width/height.. not goin there, lol. Not even sure I want to do blink text, at least not without an optimization to have to rewalk the buffer looking for things to blink.

Looks like the numerous ^O / ^N that switch to "alternate character sets".. prolly not gonna support that either. Looks like there's a "line drawing character set" on the DEC terminals I'd never heard of or used, (kinda similar to what the PC graphics character set provides, or a subset thereof): https://en.wikipedia.org/wiki/DEC_Special_Graphics I could translate that to utf8 equivalents I suppose (Unicode line drawing chars), but not sure I'm gonna go there either. What I'd prolly do (if it's not out on the net already) is create a new utf8-friendly .txt file, and then use that, lol.

[Albrecht-S]

Yeah, I didn't think that you want to implement all possible, maybe DEC specific code. I see that my different terminal environments also don't do everything perfectly. Double width/height doesn't seem to be really necessary (I believe it's <esc> # 3 and <esc> # 4 or similar). Switching to the line drawing characters and implementing them would probably be hard to do. The reason I posted it was just for fun (seeing it in action) and to point you to several existing test files. But don't try to implement all this!

And yes, killing the cat while it runs can leave the terminal in a bad state. As you found out, I'm also using reset but there's also an option in the Gnome Terminal app (if that's what I use here too): hamburger menu + Advanced + Reset (and Clear). I should have warned you, sorry.

Did you try my slow.c program? It's easy to compile with gcc and use as a pipe.

[erco77]

Heh, yes was already familiar with playing with animating ascii art, and all the fun modes one could crash the terminal with. These days terminal support seems pretty good -- I'm guessing I had a much less fancy terminal emulator when I last played with that xmas file.

Back in '83 or so, when our college let us use the "new" IBM PC's (actual first generation PCs from IBM with the green phosphor screens and floppy drives), I caught on to ANSI.SYS/DOS, and wrote a small program in IBM Basic that let one interactively draw an ANSI animation using PC graphics; you'd use the arrow keys to move to where you want a character and hit + to plot a box, and space to plot a space. The cursor keys made the ANSI equivalent cursor moves in a file. My first use of it was to make it look like the screen was filling with liquid by pouring in from the side, using the boxes to make the "liquid". You'd then 'type' the .txt file and those machines were slow enough you didn't need to slow it down,lol.

I later worked at a company in 1986 that had DEC VT3xx's that could do black+white pixel graphics, so I rendered some mandelbrot fractals on that. There was also a tek4010 that could do vector graphics, but I never could get time on that machine. It was only recently, in the last few years I got to play with a Tek emulator (which I found is also built into Xterm, lol) so one could make pretty neato vector graphics, which I used as output from a vector renderer I wrote last year that could render Tek4010 line drawing, or to FLTK/cairo. Here's an MP4 animation I made with that renderer, which was the final goal of that project.

[erco77]

Another ASCII art project is this tetris game I wrote back in the 90s which runs in the terminal and uses ESC codes to draw the pieces. It's fun to play and the code is tiny. Had to keep "porting" it over the years to handle the various termio stuff. %^/

[erco77]

Did you try my slow.c?

I was using pv -L 400 -q xmas3.txt to view the animation, which I think does the same thing as slow.c. Plays at a nice speed and with the right graphics (I think) in gnome-terminal.

Commit 76fc7b7878 is an unstable work in progress for handling screen resizing. Basically an 'end of day but it ain't done yet' commit, but good proof of concept.

What works:

• Hit Shift-H it generates a large numbered report that scrolls into the scroll history (which you can't scroll into with the scrollbar yet)
• Resizing the window /larger/ in the vertical direction only recalls lines that scrolled off to the history buffer, and adds them to the top of the display. (looks simple enough, but there's a lot going on behind the scenes to make that happen in the resize_screen_chars() code.
• Hitting lowercase 'h' dumps the history buffer to the tty
• The scrollbar tab changes size correctly based on the history use

What doesn't work/todo:

• Some wanky graphics appearing briefly during resize
• Resizing /smaller/ needs to work too (pushes lines back into the history buffer)
• Moving the scrollbar doesn't do anything yet. The draw() code will have to mix drawing lines from the history and lines from the active display.
• Cursor and mouse selections need to be adjusted when rows are juggled around
• Cursor and mouse selections ALSO need to adjust when scrolling of any kind occurs (up or down)

Will continue over the next day or so to stabilize.

[Albrecht-S]

I was using pv -L 400 -q xmas3.txt to view the animation, which I think does the same thing as slow.c. Plays at a nice speed and with the right graphics (I think) in gnome-terminal.

I didn't know pv, thanks for the hint (installed, and works). My Gnome terminal doesn't seem to support double width/height text, I see "Merry Christmas" at the top in two consecutive lines in single width (which is "normal" since the double height text must be encoded in two lines of text: top + bottom half).

I see that you're making good progress. I'm not going to test each step though but I'm still looking forward to the final result. Thanks for the great work.

[erco77]

Yeah, it's moving along. This is just my record keeping of the design process, as this is a bit of work. The design is still clean IMHO, and am happy with it. Maintainable and not hacky.

Commit cac1ef8b3 finally adds working scrollbar, so when text scrolls off screen, you can scroll up and down to see into the history. Hitting Shift-H to generate 100 lines of text, you can then use the scrollbar to scroll back+forth, seamless handling of the history/active screen transition. Resizing the screen /larger/ when there's history also seems to adjust the scrollbar tab correctly, as lines are pulled from the history into the screen, the tab adjusts as the history gets smaller.

The default history size is currently small (20 lines) on purpose to make it easy to look for problems.

[erco77]

Trivial Shift-Click mouse selection extension is added in commit eb59ff491. Only per-character selection supported for now. And while you can extend left and right of the selection to get expected behavior, shift-click /within/ a selection doesn't always "extend" the way one should expect. Will have to add that logic at some point in the future.

Also a future feature: double-click would start "per-word" selection, and triple-click would start "per-line" selection, and Shift-Click would extend in those modes. For that I'd need to add a 'mode' bitfield to the Selection class to support char|word|line select modes.

So for now, only character-selection.

Todo: mouse selection should be able to operate in the screen history too, and be able to autoscroll when dragging up and off the screen bottom/top.

[erco77]

In 635216aa2 + 607f11abb, mouse selection now operates in screen history, and text scroll moves selection. Retooled internals so that history and display are one monolithic buffer instead of separate buffers.

Broke resizing because I rewrote it; fixed in 9e639d7a1.

[erco77]

With 2427e3245, 796b089fc, and some fixes in 25a278e6e, there's now 24bit RGB escape sequence colors using <ESC>[38;R;G;Bm (for foreground) and <ESC>[48;R;G;Bm (for background). That first commit is the large one that retools the color system to support RGB. This meant keeping RGB fg and bg colors on a per-utf8 character basis, which changed the API a bit to allow both 3bit and 24bit color management.

There's a new test page to test the new ESC[38;.. and ESC[48;.. support:

[erco77]

After seeing some scrolling slowness kick in when I enlarged the history and supported RGB, I realize I can heavily optimize scrolling + history management by using a ring buffer.

At 2am last night I realized managing "scrolling" and "scroll history" could be as easy as just incrementing indexes into the ring, and not moving characters around at all.

Made this simple proof of concept mock-cxx.txt which shows how scrolling works nicely with a history and display combined into a ring buffer, showing the ring buffer/history buffer/display buffer as scrolling occurs, all by just incrementing a couple of indexes.

In the screenshot:

• Left Column: the entire ring buffer. Includes "scroll history" (cyan) and "active display" (yellow). The buffer starts with the history buffer filled with 'H', and the display buffer filled with 'D'.

• Center column: just the 'history' part of the buffer (cyan)

• Right Column: just the 'display' part of the buffer (yellow)

As you hit "Enter", the "display buffer" scrolls up, with numbered lines going in where blank lines would go, to show how the buffer is consumed. Each 'scroll' (hitting ENTER) causes the lines scrolled off the display to roll into the 'history'. All of this is done without moving strings of data around, just adjusting the two integers for the start of history and the start of the display that point into the ring buffer.

So 'scrolling' is as fast as incrementing two integers -- Nice! Can't get more efficient than that.

So that should I think greatly simplify and speed up the widget. So I'll be doing a retooling for that.

[erco77]

Retooled internals for ring buffer in 98b117b0f, so now history can be huge. Set the default to 5000 lines, and still fast as hell.

Added new 'r' command test in 0a9c4e05 which runs ls -la /usr/bin with colors turned on, which should generate at least 2500 lines or so of output.

Pretty happy with the speed, and that will become 'instant' with large reports when I add the timer stuff for redraws.

I may have broke some stuff in subtle ways, not sure yet, as the retool touched a lot of code. (for instance I see escape test page 0006 got broken; it's not erasing correctly. That was working before the retool [EDIT: Fixed in 5dc137093]. I just wanted to commit as soon as things were working, for backup purposes.

Doing a redraw every crlf will slow things down for large reports; speed still too fast to read, but a command generating large output will be slowed down. For instance the 2500 line 'ls' output above takes less than a second to run, but if I redraw every crlf the command takes many seconds to scroll up because it gets log jammed doing all those redraws. But if I rate limit redraws to every 30 lines (which is what the new 'r' command uses), it goes really fast.

And if I used a timer to rate limit, since ls takes no time at all, the entire output would just slam on the screen and into the history buffer "instantly", because buffering the output isn't what's the bottleneck now, it's just drawing text.

No terminals redraw every crlf -- they all seem to rate limit redraws.. I'll probably use something like 25 fps. Then commands that spit out large output quickly will just slap up on the screen instantly.

[erco77]

OK, the current head at 1fc6c2327 is in pretty good shape now.. a good point for looking at progress.

You can resize the display, run through the tests, make text selections and copy/paste. No memory leaks, very fast. cloc(1) shows 1580 lines of code, which IMHO qualifies as still being "light". (Fl_Term is at 1541 lines, but the designs are completely different, so I guess that's about what it takes to make such a widget.)

I'd honestly thought before starting I would need many more lines of code, esp. for the text drawing stuff. But actually drawing utf8 text in different colors/attributes was the EASY part, lol. And what was tough in the early designs was scrolling due to memory movement, but when I redesigned with a ring buffer, that shortened the code by 200 lines, and improved speed to the point where it doesn't even matter how big the display/history is, when new lines are added, text scrolling is super fast and involves no memory movement/reallocs.

There's still some stuff to do, but I'd say the main aspects of the widget are operational. Window dressing stuff like autoscroll during selection off the edges and whatnot have yet to be done (one can still scroll manually and use Shift-Click to extend selections beyond screen edges)

[erco77]

Added a test/terminal program (fluid app) to start fleshing out/exercising the public API for the widget. Similar to the test/tree program, the idea is to be able to exercise the public API for problems. Commit 669719d67 is a "work in progress" for this tool; in the only thing that works is typing a unix command and hitting ENTER to run it and the output appears in the terminal window. None of the spinners/radio buttons work yet. The output in the terminal will be "slow" because currently I'm having to redraw() per line, in order for slow commands like 'ping google' to show up without being buffered. When I add timer redraw() as a default, that slowness won't be a problem.

[Albrecht-S]

Great progress, works well. Thank you.

I was curious and tested the command pv -q -L 3000 xmas.txt: worked fine - not perfect because there are obviously some unimplemented escape sequences, but that's OK. Awesome. You can watch the train "driving through the window". ;-)

One thing I found surprised me though: window resizing is reeeeaaaallllyyyy sloooooooow. Did you notice this as well?

I didn't test anything else but basic commands like the preset ls command, and it works as expected.

[erco77]

Interesting; currently redraws would be needed to "see" the animation, and those redraws are line oriented (fgets), so I wouldn't have expected the train to have CRLFs, so perhaps the train is "animating" in blocks of whatever max string size I'm using for fgets().

Window resizing slowness -- I think I have the screen history cranked to 2000 5000 lines to look for problems like this; haven't checked.

Meanwhile: a long 'ls -la' listing in the 'terminal' app will be slow because I currently am redrawing per-line, which will definitely be "slow". But if you try hitting 'r' or 's' in the testterm program, the 'ls --color -la' report in 'r', and the code listing at the end of 's' will come up really fast, because IIRC I redraw only every 30 lines or so.

That should all be a non-issue once I implement redraw on a timer, which will be the default, so that redraws will be rate limited to e.g. 10 or 20 fps, so that if text comes in quickly, redraws won't happen more frequently than that (preventing slowness), while slow output will continue to update quickly.

I'm imagining there will be at least four redraw options: Timed, per-line, per-character, and 'none' (where the app handles all redraw triggers), with Timed the default.

[erco77]

Right now I'm syncing my fork to fltk.org, and now merging it to my branch, as I haven't synced in a while after that github corruption (which they're still unsure as to the cause of, documented above in an earlier comment). FWIW, to sync my local branch to fltk/fltk master:

1) On github sync the master branch of my fork (switch to Master branch, hit Sync fork)
2) On my local machine:

    git checkout master
    git pull
    git checkout <mybranch>
    git merge master            <-- verbose merge (creates a "knuckle" in gitk)
                                    or 'git rebase master' for a "squashed" merge

I'm actually in progress on the "redraw style" option, and definitely want to get the timer based redraw in place, so that whenever new text comes in, the timer starts triggering redraws, and stops when there's no new changes to the buffer.

[erco77]

Give 398b008be a try to see if that solves the slowness on resize issue. Should at least be "better".

You will see changing the width of the display (to be larger than it was) will be slow, because it's enlarging the entire history, which necessarily involves copying the previous "narrow" history into the "wider" history. So if the history is 5000 lines, it has to move that entire utf8 buffer over and copy.

But making the window smaller or taller/shorter should be fast.

Even making it larger in width should be fast if you've already sized to that width before. (It doesn't shrink the width of the buffer when you make the window smaller, only enlarges. This is to prevent data loss, as I'm not "rewrapping" text yet -- maybe someday. That's a bit complicated).

It's possible I could hold off on doing the updates of the buffer size DURING window resizing, so that it only does one buffer sizing at the time the resize has completed. I'm not sure I know if there's a way to know the user is in the middle of resizing a window or not, vs. "completed'. But there be dragons on that path.

[Albrecht-S]

Yep, resizing is now quick if you have once enlarged the width to a certain (maximal) point.

git merge master <-- verbose merge (creates a "knuckle" in gitk) ... or 'git rebase master'

I'm in favor of git rebase master because it doesn't create such a confusing history ("knuckle"). Some git deelopment guidelines say "never merge downstream", and what you did was exactly thst: a downstream merge (merge upstream/master into your development branch). It works but has drawbacks WRT history. Please never merge your development branch back into upstream/master because this would fix all single commits in the (upstream) history.

You can probably still use git rebase master to do a clean rebase. What I prefer to do is git rebase -i master and squash individual commits. But details would be OT here.

[erco77]

Ya, I kinda like those details, as I can see e.g. manolo's fix wrt my previous releases. Or well, at least in gitk where it shows this nicely. In git log not so much, lol. So I can see your point there.

When I submit Fl_Terminal, I'll probably just update my local fltk/fltk repo, copy the new Fl_Terminal files, and merge the Makefile/CMake stuff with xdiff, commit and git push, avoiding the whole fork/merge worries.

I've gotten into weird states with rebase between my local and remote repos, which is a pain I just want to avoid. I find it really easy to forget the weird states git can get into, and rather than try to stop work I'm doing to figure it all out, just reclone and copy files to power forward. I can only handle so much bs from git, lol.

[erco77]

Actually too, Manolo mentioned that I don't have to cache character widths, which was the cause of that last slowness issue. So if I rip the cached width out of my utf8char class, that'll make history smaller by 4 bytes per char, which would be nice. And not having to update that cache saves a pass at the whole buffer whenever the font or font size is changed. So I'll try to work in that simplification soon.

[EDIT: This was done in ae304899f and b296a70648. Seems OK so far. And the per-character cache of the width was actually a double, so removing that got rid of 8 bytes /per character/, which should help memory size and movement quite a bit. In the case of a 80 x 5000 char history, that'd be a difference of 3.2 million bytes, or five 1985 era 640k IBM PC's!]

[erco77]

@Albrecht-S Regarding merge commits, it looks like github provides a way to block them; I noticed my fork's branch kept asking me about if I wanted to "Protect your branch" or some such, and sick of seeing it, I checked it out, and it appears the owner can force github to only accept linear merges. Thought this and other "protection" options might interest you if you hadn't seen them.

[Albrecht-S]

Thanks for this hint.

This is OT here, but anyway: I don't want to be too restrictive with protections like this one. In particular, sometimes I'd like to have a merge commit (a "knuckle", as you say) if there are a few commits that I want to merge but not add as single commits to the main (master) branch. In our small team the devs with push access should hopefully know how to deal with this situation. I'd only restrict merge commits in general if it happened too often (by accident) or if the merge history became too confusing. The only protections I activated so far is to prevent force pushes and deletes - both are default (not checked) if the branch is set "protected".

[erco77]

In commit 6f19e9829 I retooled test/tree to use Fl_Terminal. No slowness at all when you open "500 Items" and then "Select all". The output snaps right into place. And you can deselect/reselect without any slowness.

Also: I still have the default history scrollback buffer initialized to 5000 lines, so the fact it's still snappy is a good sign.

Retooling was easy; I just edited test/tree.fl and changed Fl_Simple_Terminal -> Fl_Terminal throughout, and recompiled.

Still need to implement timer redraws; currently each printf() causes a redraw, but in test/tree the term window is so small, redraws are really fast. Redraws can get slow if the font is small and the window is large, so redraws need to be rate limited.

[Albrecht-S]

Awesome, that's good news! And congrats (that your idea seems to work out well)!

[erco77]

Thanks. Once the test/terminal app works fully and has all the options so I can test them, and add the dox docs, I figure it'd be ready for release.

[erco77]

OK, added some of the last needed features in commit f786baed, namely:

• Autoscroll during mouse selection; drag off top/bottom causes auto-scrolling
• Automatic management of redraws with rate limiting
• Resizing window should retain content and act predictably
• Expanded the test/terminal app to test more API features (colors, rate limiting speed, etc)
• Updated the README design doc for the new RingBuffer class. Doc still a WIP, but has critical design elements

Getting pretty close to "done" I'd say, feature wise.

A few small items on the TODO list, but doing the doxygen docs should probably be next. I plan to keep those light, as I want to get this thing released soon.

Since 1.4.x hasn't released and therefore Fl_Simple_Terminal never officially released, I imagine the new Fl_Terminal can make Fl_Simple_Terminal obsolete, and perhaps simply remove Fl_Simple_Terminal altogether, recommending people who've been using 1.4.x to retool their apps to use the newer widget.

I'll try to keep the API as compatible as possible with Fl_Simple_Terminal, but necessarily there are some differences, such as access to the Fl_Text_Editor style buffer stuff -- that's no longer relevant in Fl_Terminal.

[Albrecht-S]

... the new Fl_Terminal can make Fl_Simple_Terminal obsolete, and perhaps simply remove Fl_Simple_Terminal altogether ...

I agree to replace Fl_Simple_Terminal with Fl_Terminal, i.e. add Fl_Terminal and remove Fl_Simple_Terminal as soon as we're sure Fl_Terminal works in our test and demo apps. Should really be done before the 1.4.0 release.

[erco77]

Updated unittests in commit 80ade6e26 to use Fl_Terminal, will do others in a burst.

[erco77]

In 7b63afb4b all the test programs use Fl_Terminal now.

[wcout]

Great work!

I played a bit with the test/terminal program, issuing different commands and there is one thing I noted:

With some outputs my system console displays messages Pango-WARNING **: Invalid UTF-8 string passed to pango_layout_set_text(). One of this commands was man exec. There is a visible corruption in one of the tables when displayed in terminal - that must be the cause of these error messages. It does not matter if I redirect the command on my console to a file: man exec >xxx and then in terminal do only a cat xxx. And the file seems ok - because the FLTK editor demo displays it correctly (with no Pango warnings), so this must be something within Fl_Terminal. If I edit the xxx file and remove characters around the spot where the corruption occurs, then the corruption moves on to a later spot in the file! This suggests it has nothing to with the characters, but something in memory is getting corrupted after a while. On the other hand this does not happen with every content, there seems to be a trigger too.

[erco77]

Thanks for testing -- it's kept me busy the last few weeks, lol

Two things I can think of:

• Use append(), not printf()
• Make sure strings you send to the terminal don't include partial utf-8 sequences (e.g. cropped by block reads on pipes)

I ran into the problem you're describing when byte-wise reading pipes in blocks (instead of with fgets()), and the byte stream include utf-8 sequences. The problem being utf-8 character's byte sequences split by the block boundaries.

Currently the append() and printf() methods (and friends) assume complete utf-8 character sequences are being passed in. And that comes up if one trivially calls append() to write the blocks, possibly passing partial utf8 characters at the beginning/end of block boundaries.

The solution would be for Fl_Terminal (or your app) to check if the trailing character at the end of the block is a partial utf-8 char, and if so, crop off those bytes and buffer them to insert into the next block that is written, rinse & repeat for each block.

I'm not sure if Fl_Terminal should handle this, or the app. Hmm.

Anyway, I'm working on the doxygen docs now, and I know I need to cover that for the current implementation of append().

I'm pretty sure there is an fl_utf8_xxx() function that can walk a string in reverse to detect if the string ends in a partial utf8 character, and leaves you pointing to the beginning of the partial character, or perhaps the end of the previous complete utf8 char. I'll take a look.

Perhaps what I can do is provide a separate method for block writing to Fl_Terminal, so that this fl_utf8_xxx function doesn't have to get involved in ALL append()/printf() operations. This would allow the app not to have to care about the issue, but wouldn't have to burden all writes to the terminal. Shouldn't be hard to write a method like Fl_Terminal::block_write(), or some such, that handles this. The 'buffer' for those extra bytes would only have to be about 5 or 10 chars max. The insert operation can surely be done without having to do a costly byte shuffle.

Anyway, for right now, the app has to detect this, and prevent sending incomplete utf-8 chars to append() (et al).

An easy way to prevent this is to use fgets() to read your pipe (or write your own) that only sends strings that end in a "\n", to ensure utf8 chars don't get split.