Fl_Simple_Terminal::ansi() mode may be enhanced by parsing ";" charactor.

[rageworx]

I'm using Fl_Simple_Terminal to emulate remote shell terminal monitor, and found Fl_Simple_Terminal has some strange color parsing with \033 escape. For example, remote shell returns these buffer to Fl_Simple_Terminal,

-rw-r--r-- 1 linaro linaro 4065 Dec  5 09:34 Makefile
-rw-r--r-- 1 linaro linaro  417 Mar  8  2022 README.md
drwxr-xr-x 2 linaro linaro 4096 Dec  2 02:02 \033[0m\033[01;34masm\033[0m
drwxr-xr-x 2 linaro linaro 4096 Dec  5 09:34 \033[01;34mbin\033[0m
-rwxr-xr-x 1 linaro linaro 1413 Mar  8  2022 \033[01;32mconfigure\033[0m
-rwxr-xr-x 1 linaro linaro  141 Aug  2 23:34 \033[01;32mgv.sh\033[0m
drwxr-xr-x 5 linaro linaro 4096 Jun 17 07:25 \033[01;34mlib\033[0m
drwxr-xr-x 2 linaro linaro 4096 Dec  5 09:34 \033[01;34mobj\033[0m
-rwxr-xr-x 1 linaro linaro  384 May  4  2022 \033[01;32mpackaging.sh\033[0m
drwxr-xr-x 2 linaro linaro 4096 Sep 30 09:03 \033[01;34mrefs\033[0m
drwxr-xr-x 2 linaro linaro 4096 Nov 30 00:48 \033[01;34msrc\033[0m

When ansi() enabled Fl_Simple_Terminal displays just RED color with \033[01;32m as parsed color is "1" == RED. I think it should better process checking ";" in Fl_Simple_Terminal by like this.

                            switch(*sp)
                            {
                                case ';':         // numeric separator
                                    ++sp;
                                    continue;

to

                            switch(*sp)
                            {
                                case ';':         // numeric separator
                                    ++sp;
                                    if ( tv > 0 ) tv = 0;
                                    continue;

Fl_Simple_Terminal cannot display all escape colors, but it should express similar color as I thought. Here's my forked & branched commit here.

• https://github.com/rageworx/fltk-origin/commit/c834f45aa777e8622f01f45a06c0d0c10fa2799e

I hope it may fix strange color parsing for Fl_Simple_Terminal.

[rageworx]

Here's difference,

[MatthiasWM]

Thanks for your suggestion. I am pretty sure I can apply this fix in the next days.

[rageworx]

Thanks for your suggestion. I am pretty sure I can apply this fix in the next days.

Thank you so much !

[erco77]

[I email replied to this issue and it looked all wrong, so doing it again inside github..]

> drwxr-xr-x 2 linaro linaro 4096 Dec  2 02:02 \033[0m\033[01;34masm\033[0m
> drwxr-xr-x 2 linaro linaro 4096 Dec  5 09:34 \033[01;34mbin\033[0m
> -rwxr-xr-x 1 linaro linaro 1413 Mar  8  2022 \033[01;32mconfigure\033[0m

Those are xterm sequences that Fl_Simple_Terminal doesn't support. Supporting full xterm is beyond the capabilities of the base class Fl_Text_Display that Fl_Simple_Terminal derives from.

Fl_Simple_Terminal currently only supports very specific ESC[#m color settings where '#' must be a single digit and can be 0, 30-37, 40-47.

If you want ls --color to work correctly inside Fl_Simple_Terminal, set the LS_COLORS environment variable to match what Fl_Simple_Terminal expects.

So if you want normal files to be white, directories to be yellow, executables to be cyan, you would use:

        export LS_COLORS='no=47:di=43:ex=46'
        ls -l --color

..and that should work to generate the single-digit ESC[#m values that Fl_Simple_Terminal wants. Breaking that down:

            File type      LS_COLORS  Escape Code    Fl_Simple_Terminal color
            ------------   ---------  -----------    ------------------------
            normal files   no=47      ESC[47m        Bright White
            directories    di=43      ESC[43m        Bright Yellow
            executables    ex=46      ESC[46m        Bright Cyan

Note there are many other LS_COLOR setting keywords for all kinds of file types (pipes, block devices, setuid, etc) and even *.ext filename extensions.

For more info on how the LS_COLORS environment variable works, see: http://www.bigsoft.co.uk/blog/2008/04/11/configuring-ls_colors

                        switch(*sp)
                        {
                            case ';':         // numeric separator
                                ++sp;
                                if ( tv > 0 ) tv = 0;     // could probably just be: tv = 0;
                                continue;

The above change may "work" for your case and the codes you're getting, but that's not a good solution for the distribution code, as that just short circuits multi-digit ESC sequences (e.g. ESC[#;#m ) which might be supported someday, especially if Fl_Text_Display were to support setting background colors (currently it does not, AFAIK).

If you're doing a terminal emulator, I'd suggest either defining a terminal environment that supports only what Fl_Simple_Terminal does, or take a copy of Fl_Simple_Terminal into your code and modify it to handle all the escape codes you want. But be warned: the base class, Fl_Text_Display, is very limited.

[rageworx]

Yes I know limitation of Fl_Text_Display, And I know why simple terminal handles only for last colors. But there's example to expand color tables for more colors. I know some kind of \033[R;G;Bm also not works too, but I think any case of expression of legacy escape color, it should always last 34m color draws right color instead [01;34m as parse to 1m (red) color. I hope Fl_Simple_Terminal more extended support for xterm 16 colors.

[erco77]

I can probably modify the code to also handle ESC [ 01;#m similarly to how ESC[#m currently is -- would that help? I can follow up with a fix for that, as that shouldn't break anything.

Still though, I imagine if your TERM variable is set to xterm (or similar), you're going to encounter trouble, as there's so many ESC codes for xterm that Fl_Simple_Terminal (FST) doesn't support.

Depending on what you're doing, you might want to create your own termcap for FST so that all programs only use the codes you specify, so as to map highlighting, underline, etc. to nops, so that at least tools like man and grep -color will work correctly, without having to specify GREP_COLORS, LS_COLORS, etc.

It used to be easy to create termcaps (syntax was similar to the LS_COLORS variable), but it's been decades since I've done that, and haven't kept up with the new binary format which I think is now in /usr/share/terminfo/[a-z] or somesuch. I have a vague memory of using a termcap compiler (tic?) that converted the text format to binary.

[erco77]

For reference to myself regarding the handling of # in the long ESC [ #;#;#..#m sequences: https://superuser.com/questions/1684478/the-ansi-escape-code-esc131m-sets-terminal-foreground-color-as-red-what-doe

Looks like ESC[1;31m is equivalent to ESC[1mESC[31m, which I believe means 'enable bold' (1), and enable red foreground color (31). So I believe I can just look at the parsed array of #'s and only handle the values 0, 30-37,40-47, and ignore the rest because FST doesn't features like 'bold' and 'underline' (yet).

Will follow up here with a commit to handle this.

[rageworx]

Dear @erco77 ,

I just thought this case , "\033[xy;ZZm" is support range for Fl_Simple_Terminal by "[xy;" is decorating "ZZm;" as single color index ZZ. For example, "\033[01;32mGREEN" is little brighter green than normal 32m, and [02;32m is darker than normal 32m green. I just thought Fl_Simple_Terminal should ignore decorating cases, and just express only color by "ZZm" as GREEN.

If you don't think it is not a necessary issue to be fixed, or not a bug, or not a suggestion to be little more better, is should be closed. But I thought this case is acceptable case for express only for color with "ZZm".

I believe this case should not much breaking Fl_Simple_Terminal basic sequence. Thank you again for following this issue.

Regards, Raph.

[erco77]

Here's what I feel is safe to do in Fl_Simple_Terminal:

Here's a patch that implements this: fix-FST-patch.txt That patch parses the #;#;#.. sequence without short-circuiting it. The patch also adds testing the ESC[1;#m sequence to the test/unittests program, to make sure both ESC[#m and e.g ESC[1;#m work the same way.

[rageworx]

Here's what I feel is safe to do in Fl_Simple_Terminal:

Here's a patch that implements this: fix-FST-patch.txt That patch parses the #;#;#.. sequence without short-circuiting it. The patch also adds testing the ESC[1;#m sequence to the test/unittests program, to make sure both ESC[#m and e.g ESC[1;#m work the same way.

Dear @erco77 , It looks perfect ! Thank you so much.

ps am I need to close this issue by myself ?

[erco77]

Great -- I'll close the issue when I commit the patch (after I finish testing).

[rageworx]

Update,

Sorry for delete comment, it was my fault :) Now it looks working well for unittest.

plus,

[rageworx]

Here's my customized fltk repository changes,

• https://github.com/rageworx/fltk-custom/commit/fda9372e47343f9f09f10e5dde3f85c07a7abf59

[MatthiasWM]

Sorry for not replying sooner. Our entire part of the city was and still is without internet, and using GitHub on the phone is not may favourite. Thanks, @erco77 for continuing this. Is my help still needed here?

[erco77]

Hi Matt, I've got this issue, thanks. (wow, no internet.. In a pinch you can use your phone as a "hotspot" for your computer. When I had DSL my internet was actually faster over the cellphone..!)

[MatthiasWM]

Yeah, I did that for a bit, but German contracts really suck, and I get only 12GB a month (used to be two until half a year ago). The DSL in the whole area is down for three days now in the middle of Düsseldorf with hundreds of businesses in the street. And no fiber in sight.

[Albrecht-S]

Please see also PR #588 which may be related.

[spectrum70]

Hi, sorry, couldn't see there was job ongoing on this.

Anyway, i fixed it in PR 588 with a simple for loop, processing all the fields found in the "m" sequence as \x1b[0;1;33m

It's a fast fix, maybe not the best, but actually systemd colors are displayed correctly.

[erco77]

Yes, PR 588 is the easier way to apply the fix for standard ESC[#;#;#..m sequences. My patch is a little longer, supporting the strange 'missing #' format where e.g. ESC[;m is the same as ESC[0m, a strange variant I have not seen in the wild, but is apparently "allowed" according to the spec. I find that to be a pathological case, but by changing the parsing loop slightly, I was able to support it (I think) which is why I've been taking so long; need to carefully regression test this new mod. When I first submitted Fl_Simple_Terminal's ANSI support, I had a reg tester, but had no where to commit it. We really need a place to put regression testing code.

[Albrecht-S]

supporting the strange 'missing #' format where e.g. ESC[;m is the same as ESC[0m, ... apparently "allowed" according to the spec.

In my "earlier life" ;-) I worked with DEC systems and terminals (DEC = Digital Equipment Corporation = the "inventor" of VT-100 which later became an "ansi standard" IIRC) and I'm pretty sure I saw this type of Escape sequences. If it's not too complicated (and you seem to have done it already) then I'd say, keep it. Then we could just close PR #588 as resolved, is this correct?

[MoAlyousef]

Should the ansi mode also be able to handle \x7f and \x08 which are used for backspace and deletion?

[erco77]

I'd say backspace surely should be supported (kinda surprised it isn't, but it might be "hard" for some reason -- I'll check). Del, I'm kinda 'meh'. (I think many actual terminals don't even support Del for printing)

Offhand there's surely "a lot" Fl_Simple_Terminal "could" do but probably won't because of how limited the base widget is. The intention for Fl_Simple_Terminal is definitely NOT to be a full xterm compatible widget, but more a simple tty-like widget to allow messages from an application to be accumulated more easily than Fl_Text_Display.

[erco77]

Should the ansi mode also be able to handle \x7f and \x08 which are used for backspace and deletion?

OK, added backspace processing (\x08) in commit 118bf55, which (currently) only works when ansi() mode is enabled. Let me know if that works for you.

I'm still holding back my ESC[#;#;#..m parsing; will follow up when I've chosen the best route.

[spectrum70]

Great thanks, i need it too :)

On Fri, 16 Dec 2022, erco77 wrote:

  Should the ansi mode also be able to handle \x7f and \x08 which are used for backspace and deletion?

OK, added backspace processing (\x08) in commit 118bf55, which (currently) only works when ansi() mode is enabled. Let me know if that works for you.

I'm still holding back my ESC[#;#;#..m parsing; will follow up when I've chosen the best route.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because youcommented.[ATFKXTU325ZS3W5NM2KE7ATWNS4EJA5CNFSM6AAAAAASXTXV4OWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSQZJOAY .gif] Message ID: @.***>

[erco77]

Of interest I also realize ESC processing currently does not work if an ESC sequence is split across append() currently. So for instance:

tty->append("\033[");
tty->append("1mThis is red");

..would currently not work. This is because the widget was mainly designed for writing complete strings from the app, and not designed as a full terminal emulator where reading a pipe might easily split an ESC sequence across multiple pipe read(2) operations, and therefore split across one or more calls to append().

Will see if I can address that next, as that too will greatly affect the ESC parser, so that the sequencing parsing state information is retained outside of the append() loop.

[spectrum70]

yes, sounds a static variable that keeps a state may be used.

On Fri, 16 Dec 2022, erco77 wrote:

Of interest I also realize ESC processing currently does not work if an ESC sequence is split across append() currently. So for instance:

tty->append("\033["); tty->append("1mThis is red");

..would currently not work. This is because the widget was mainly designed for writing complete strings from the app, and not designed as a full terminal emulator, where reading a pipe might easily split an ESC sequence across reads.

Will see if I can address that next, as that too will greatly affect the parser, so that the ESC sequence parsing state information is retained outside of the loop in append().

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because youcommented.[ATFKXTW2ARYNY3MSNFQTX7DWNS5VFA5CNFSM6AAAAAASXTXV4OWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSQZKPNM .gif] Message ID: @.***>

[erco77]

Well, maybe not static -- I was thinking more like private members in the class.

[spectrum70]

oh, yeah, sure it's c++ :), i am stuck with my mind in C.

I am wondering if thinking a total new widget to have a good real terminal that can compete with gtk, that seems quite fast (see gtkterm) may be a good idea.

Also, a separate vt100 object to handle the sequences.

On Fri, 16 Dec 2022, erco77 wrote:

Well, maybe not static -- I was thinking more like private members in the class.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because youcommented.[ATFKXTUNNKN5U7PURWLM4TLWNS6EFA5CNFSM6AAAAAASXTXV4OWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSQZK6RO .gif] Message ID: @.***>

[erco77]

Yes, I purposefully didn't use the name Fl_Terminal so that if someone came along and wrote a full terminal emulator widget, that name could be used.

But honestly, a full cross platform terminal emulator widget would be a large undertaking, and would likely be rejected as "not fast-and-light". It might be fast, but it would be 'heavy'. A good design would start with a base class that could handle text in a way that per-character colors and styles could be subclassed by higher level HTML, Text Editor, and Terminal Emulator widgets to prevent redundant "rich text" buffer handling code.

Terminal emulators have also become quite complex. And any real support would likely involve integrating with pseudo ttys to handle character cooking. And the pty interfaces vary greatly across Linux vs MacOS, and I don't even want to think about Windows. The pty and termio API for MacOS vs Linux differs quite a bit, IIRC, due to SysV vs BSD differences. I also know MSFT changed a lot of things in WIN32 to support Linux terminal integration and pseudo ttys, and I'll bet that has a "flavor" all its own, lol.

As it stands, FLTK has a kind of old but "serviceable" text editor, Fl_Text_Editor (and friends), which shows its age, and is a beast of code, and Fl_Simple_Terminal necessarily derives from it, inheriting its limitations.

[Albrecht-S]

a full cross platform terminal emulator widget would be a large undertaking, and would likely be rejected as "not fast-and-light".

I agree. Such a terminal emulator would be something for a project based on FLTK but not for the FLTK core as a widget. IMHO.

[erco77]

OK, so I have a branch pending with changes mentioned above regarding saving state information, so that escape sequences can span calls to append():

https://github.com/erco77/fltk/tree/issue_577_ansi_append

See recent commit comments on that branch. In short this involved a rewrite of the ANSI parser, using a private class to handle the state information. Also includes:

• recent additions for backspace handling
• fixes this issue (#577) by handling multi-value sequences like ESC[0;1m
• handles weird sequences like ESC[;m as if it were ESC[0m

So when this commits to master, that'll all be fixed together.

I've tested it pretty well, but want to give it a "Denny's Code Audit" which involves me taking the code to a coffee shop and just staring at it, looking for mistakes and/or any unfinished business. Should be usable, as I was able to run test/tree and test/table and test/unittests.

[wcout]

a full cross platform terminal emulator widget would be a large undertaking, and would likely be rejected as "not fast-and-light".

I agree. Such a terminal emulator would be something for a project based on FLTK but not for the FLTK core as a widget. IMHO.

This one looks quite interesting:

https://github.com/yongchaofan/FLTerm

It does not rely on Fl_Text_Display or other widgets but draws everything on its own. (Tested shortly using a 'Shell' connection).

[erco77]

Re: FLTerm -- very nice!

I cloned and built it -- seems snappy, has good text highlighting, colors, seems to maybe have full xterm support. Was able to run vi and page through nicely. Pretty impressive! And it's all self contained, doesn't seem to derive from other FLTK widgets to do its work, doesn't use STL (other than for locks). Yet the code for FLTerm.{cxx,h} is only 1500 lines.. how the heck, lol.

Only problems I noticed, which are surely surmountable:

• GPL3 license (can perhaps request from the author(s) if it can be folded into our LGPL2 exceptions)
• Doesn't seem to handle Unicode/UTF8
• a few mild bugs in the xterm support (noticed during shell editing)
• sometimes screen history gets mangled, but not sure what triggers that (window resize? vi?)

If this turns out to be a good solution, I'd dump Fl_Simple_Terminal in a new york minute, lol.

[erco77]

Gotta admit, tempted to decouple Fl_Simple_Terminal from Fl_Text_Display and just do the text handling/scrolling myself, as Fl_Text_Display just seems embarrassingly slow to do simple things like append 400 lines or so at a time to a growing buffer.

It occurs to me that all the screen drawing stuff only has to happen on the /visible/ screen, which could simply be a malloc()ed buffer for all chars and attributes on the screen. Unicode makes that a bit tricky I suppose. Then that all just gets fl_draw()n as things change on the screen. The backscroll buffer just a read-only array of what scrolled off the top, so cursor movement/editing wouldn't be involved in the backscroll, just mouse selection. Seems kinda reasonable.

[spectrum70]

Seem very nice, will check critical parts of it.

My terminal is more embedded-field oriented, will add file transfer as z/x/ymodem and other things that may occour.

No offense to the nice Fl_Simple_Terminal, but very likely i am going to use a Fl_Text_Display handling whatever sequence, even colors out of the vt100, and all kind of special chars for arrow up, tab, etc.

Thanks all for now.

On Fri, 16 Dec 2022, erco77 wrote:

Very nice!

I cloned and built it -- seems snappy, has good text highlighting, colors, seems to maybe have full xterm support. Was able to run vi and page through nicely. Pretty impressive! And it's all self contained, doesn't seem to derive from other FLTK widgets to do its work. Yet the code is only 1500 lines.. how the heck, lol.

Only problems I noticed, which are surely surmountable:

• GPL3 license
• Doesn't seem to handle Unicode/UTF8
• a few mild bugs in the xterm support (noticed during shell editing)
• sometimes screen history gets mangled, but not sure what triggers that

If this turns out to be a good solution, I'd dump Fl_Simple_Terminal in a new york minute, lol.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because youcommented.[ATFKXTTTHJ6OEMOW3VXRYHTWNVVZTA5CNFSM6AAAAAASXTXV4OWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTSQ2R47S .gif] Message ID: @.***>

[rageworx]

Dear, @erco77  How about "Fl_Simple_Termial_Display", maybe it is the right widget name because it implemented Fl_Text_Display.

And for all people helping FLTK makes better, I am a medical embedded system programmer - makes whole embedded system through client SDK for multiple platform programmers, so I need to make many kinds of GUI applications for multiple platforms to something precision calculation results from my embedded system uboot, kernel, rootfs that made for exclusive operation as like something mysterious USB device by user side. And this time was made a special application to emulate ANSI-terminal for check shell results for maintenance features to access my embedded system by user account level what check system status, or activates disabled services, or many things with my non-standard security access certification methods - because these features are not for end-users.

I'd recognized Fl_Simple_Terminal is better use than normal logging visual something for people like UNIX experienced since 1980's from UNIX system-V, after Apple IIe - through Slackware, to using escapee expressions like this,

loggerWin->Printf( "\033[43mWarning : No device found ( retry = %u/%u )\033[0m\n", refindtrials, 10 );

Simply, I can make each separated words for different colors by using traditional echo " ... " in POSIX terminal by default Fl_Simple_Terminal widget. But some requirement was belonged to printing buffers from remote Linux shell ( actually it transferred by highly encrypted buffer ) result looks strange like very old terminal applications, and debugged by reason of multiple ";" separator makes strange color result - so I think it should be fixed or enhanced by simply changing parse of ";".

I know FLTK, it is basically Fast-Light-ToolKit aimed for lightness as name, So I always used FLTK since 2009, and made applications for Linux, macOS and Windows by less porting layers - just less platform-dependent C++ codes makes me happy. (even I like to make armhf application with minimal x11 even kernel provider not supported official x11, with FLTK)   I was believed this feature may better, a little bit more right-colored terminal result for modern Linux terminal color escape expressions - as xterm-color, as a different kind of Fl_Text_Display.

Regards, Raph.K.

[erco77]

Some of you guys should really take a look at that Fl_Term application wcout linked to; if you need a widget that is basically running a unix shell/os command prompt and needs full xterm support (like for running a terminal text editor like vi/emacs and having it work right), Fl_Term appears to be the right solution for that. It really has pretty decent full xterm support, and the default app is basically a full on terminal that you can even run terminal editors inside, and it displays quite well. It seems like its design is more parallel with the use case some of you folks in this thread need. It also has ssh support, and even mentioned something about xmodem (and friends) in the code.

Fl_Term's code also mentioned something about being "scriptable" (perhaps like expect(1)?), which I assume means you can take control of the shell within the terminal and have a script run commands and inspect output, or somesuch?

Fl_Simple_Terminal (FST) is really exactly designed only for the purpose of an application's output with colors, like what @rageworx wrote above, using FST's printf() and/or append() to output text that might normally go to a terminal, but viewable within the application.

The main reason FST was made was to allow the FLTK test programs a way to output text that can be seen (e.g. test/table, test/tree) so that events/callbacks can be shown clearly. We had trouble sending this to a 'real' terminal because most people would run these test programs by "clicking" on them, where no terminal is involved, and would loose that output. So FST has been widely deployed into the test programs.

Anyway, I'll see if I can improve Fl_Simple_Terminal a bit. My main concern at this point is the slowness (issue #300) which has to do with the base widget Fl_Text_Display, which I think Matt is looking into with recent issues #596 and #603.

[MatthiasWM]

I learned that whatever single purpose option we write, someone will eventually repurpose. If Fl_Term is as universal as described and can be made to fit our license, we should absolutely do that. I am afraid that adding UTF-8 support is not an easy task though. Is it needed though? Also, Fl_Simple_Term and Fl_Term can coexist, just like Fl_Text_Editor and Fl_Multiline_Input. It adds 1MB of binary code though, if I read that right.

I can accelerate the line wrapping by a factor of four or so, but it will still remain relatively slow. The Fl_Text_Buffer optimisation is not needed, but users can accelerate things here by setting a larger preferred gap size in the constructor.

[erco77]

@MatthiasWM Yes, that was my thinking as well, they (FST and Fl_Term) could coexist. If we used Fl_Term (and got the OK from it's author), we'd have to decouple the ssh, crypto, and xmodem stuff I think, and possibly make the termio/pty stuff optional, so that it can be used like FST as just a text output widget, but can also be used as a shell terminal if the termio/pty stuff is turned on.

I did not test on Windows or Mac, but it seems to say it works there too; in what way I'm not sure (does it emulate a DOS/cmd shell too?) It certainly handles the termio/pty on linux pretty well. I just don't understand how the code is so short, lol.

The unicode support would be a big change though, yeah. I did some experiments using Mac's terminal, mixing chinese text (which is 2 chars wide) with ascii to see how it handled it. It seems like what they're doing is treating each line of text as an entity, and the complexities of e.g. cursor up/down motion is calculated on the fly, to figure out what character is 'above' when characters have different widths. In my tests, line height was unaffected by the presence of chinese characters, but perhaps there are some Unicode characters that affect line height too, which would affect vertical scroll calculations I imagine, getting us back into the kind of problems Fl_Text_Display and friends have.

[rageworx]

Dear @erco77,

FLTK UTF-8 input/expression may OK on macOS since FLTK1.4, specially Korean improved. I was tested my FLTK TTF render engine with last FLTK 1.4 ( my cloned, but almost same at base ). Old FLTK1.3 was some issues at writing Korean or Japanese words on input area, but current version seems Ok. I can write and speak them as well, and testing for multiple languages for drawing with my fl_imgtk engine. Fl_Text_Display/Edit also works fine on macOS 12.

Regards, Raph.

[MoAlyousef]

I tried FLTerm on windows, for some reason things don't work out for me. The default shell is set to ipconfig, if changed to cmd.exe or pwsh.exe, the shell starts, but it doesn't accept input. On wsl, it works well.

[erco77]

@rageworx Yes -- I know FLTK 1.4 and Fl_Simple_Terminal/Fl_Text_Display handles Unicode/utf-8 well, thanks to Manolo's and other dev's recent improvements. But in Fl_Term it does not. (This is an interesting app/widget by a third party that has good xterm support and really good terminal emulation, but does not seem to support Unicode).

@MoAlyousef Interesting -- yeah, I imagine the Windows equivalent would be quite tricky when it comes to the equivalent of termio/ptys. I recall watching a MSFT tech video back in 2016 or so about how they had to rewrite the DOS window code to support Linux terminal integration, and found a way to do it cleanly. As a result, DOS windows now support ANSI codes (just like the old DOS days of ANSI.SYS). A random link on modern windows ANSI colors.

[erco77]

OK, just pushed a bunch of fixes to master as commit 993b7da3b which should fix this issue.

• Multivalue ESC sequences, e.g. ESC[#;#;#m
• Backspace support includes backspacing over Unicode characters
• ESC sequences can now straddle calls to append() / printf() etc, and still work normally

Gonna close this -- if anyone has any followups related to this issue, I'll reopen.

[MoAlyousef]

The recent changes work nicely. I have an fltk terminal widget in Rust based on Fl_Simple_Terminal:

https://user-images.githubusercontent.com/37966791/208316330-33a898d3-d8d7-42bf-9d03-cb27898b7df1.mp4

This is using a crossplatform library for handling pty. Fl_Simple_Terminal does most of the heavy lifting, I had to handle the sticky escape sequence \x1b]0;...BEL (can't remember what it's called). For my purposes it works nicely. It needs some extra escape code handling with cmd.exe on windows, but input works fine. curses tui apps don't work well but at least they won't hang the terminal and can be ctrl+ced.

[erco77]

Cool -- thanks for the follow up. Ya, for that use case, a shell, should be fine as long as ESC codes are limited to colors and clear screen (ESC[2J). Anything else is ignored. Wondering if I should print '¿' for ignored ESC sequences, or perhaps it's better to just silently show nothing. Perhaps that should be an option flag.

[MoAlyousef]

Personally I would prefer for them to be ignored since that would probably mean a lot of unhandled sequences :)

[erco77]

@MoAlyousef understood.

I made sure that was the default behavior for the new method ansi_show_unknown(bool) (commit e7630e045 just added now).

The default for this flag is false, so unknown escape sequences are silently ignored. If enabled by the app to true, the error character '¿' is shown for each unknown sequence.

[MoAlyousef]

Thank you

[rageworx]

Appreciate for all people to handle this issue !