New Cool Stuff for ELKS

[ghaerr]

I'm opening this issue to talk about various projects I'm working on, most of which are running, but not fully integrated into the ELKS build yet. I'm interested to know how interested people might be in each. Here's the first three projects:

• ELKS remote console display
• D-Flat Text UI windowing toolkit
• Debug stack traces for ELKS executables

I'll write up each of these separately.

[toncho11]

blink16 works only on contemporary computers and it is 32/64 bit compiled, right? It is compiled with the standard gcc (not ELKS gcc16). For a second I had this idea where blink runs inside ELKS ... and it allows for the execution of DOS executables. @ghaerr this all came as I was reading that it "translates system calls". The idea is not completely crazy ... :). If it is going to be insanely slow then it could be used only on 386 and 486. If it saves you the job of having DOS installed and restarting, it's still worth it.

[tyama501]

Thank you @ghaerr ,

I see. Here is the screen after "D" and ./elksmori

[tyama501]

cat morifile/newselks works. The previous image I posted is trying to show this file.

[ghaerr]

Hello @toncho11,

blink16 works only on contemporary computers and it is 32/64 bit compiled, right? It is compiled with the standard gcc (not ELKS gcc16).

Yes - blink16 is a 32/64 bit application that uses a large amount of memory to emulate the 8086 instruction set, as well as save instruction history for "rewinding" execution (by using the scroll wheel). This takes a lot of RAM and won't ever be able to run on ELKS, nor any PC with less than 32MB of RAM.

this all came as I was reading that it "translates system calls". The idea is not completely crazy ... :).

The ELKS ("guest") system calls have to be translated to the operating system blink16 is running on ("host").

If it is going to be insanely slow then it could be used only on 386 and 486.

Not sure what you're referring to here, but if its blink16 execution speed, the purpose is for debugging, not just to emulate an 8086 like QEMU does. The capital C) and D) commands allow fast execution without instruction display.

If it saves you the job of having DOS installed and restarting, it's still worth it.

The idea is to run some DOS executables, also by translating the DOS system calls to Linux/macOS/FreeBSD, etc, but blink16 doesn't try to emulate a full DOS environment, so I'm sure many programs may not fully work.

The intent is mostly to visually see a program's execution. Blink16 also supports P)rofiling, which shows which procedures are bottlenecks in execution time.

Thank you!

[ghaerr]

Hello @tyama501,

Here is the screen after "D" and ./elksmori

I think you may be getting confused here on how to run blink16, please re-read my post above. That is, if you run:

./blink16 fd1440.img

This will execute blink16 in "system emulation mode", where the ELKS kernel (in this case) is booted from the .img file. Trying to run ./elksmori from within system emulation mode won't work well - as blink16 is primarily emulating an 8086 system, rather than trying to run a particular ELKS program. In order to directly debug elksmori, you would want to type something like:

./blink16 elksmori

This would load elksmori directly into blink16 memory, then attempt running it and translate the ELKS application system calls to Linux/macOS/FreeBSD, etc. However, this will not yet work, as the blink16 loader won't load medium-model ELKS applications (yet). So you should test with another program. I have included banner and echo, built with -maout-symtab, which includes the symbol table, which really helps display.

The reason why, in the first case, ELKS won't run ./elksmori is that the fd1440.img image has on it a very specially-prepared ELKS kernel which is very small, and is built without medium-model support. This was done because I've just barely got ELKS running in blink16. Since the ELKS kernel didn't have MM support compiled in, it actually thinks the executable is a shell script, which you can see eventually displays "Syntax error: ')' expected. That's the ash shell trying to run ./elksmoria!

I hope this all makes sense. If not, please ask more questions.

Thank you!

[tyama501]

Hello @ghaerr ,

I understand the blink16 does not support medium model yet and if I do blink16 elksmori then it says "not supported medium model" .

Also the reason why I am using the system mode is that I need to put morifiles in root directory. Is there way to put files when direct mode?

The banner is working on the direct mode.

Thank you.

[ghaerr]

Also the reason why I am using the system mode is that I need to put morifiles in root directory. Is there way to put files when direct mode?

Currently, the blink16 system call translation just translates open and doesn't change the pathname - this would mean you'd need the files in the root directory of your host system, which is not a good idea. I will have to think more about how to best handle this, and have made a note of it. For the time being, elksmoria won't run anyways since the medium model loader isn't done yet. Sounds like we still have lots of work to do to get blink16 usable, it just isn't ready yet for larger programs nor programs that use fixed pathnames. For that reason, I would suggest testing with small programs, as there is so much left to get working.

[charlysan]

Sorry if this is not related to the main topic, but is there any alternative for having an assembler and debugger within ELKS? So that we could generate binaries from x86 asm code?

[ghaerr]

is there any alternative for having an assembler and debugger within ELKS? So that we could generate binaries from x86 asm code?

Do you mean operating from ELKS itself, or just an alternative to assemble ASM programs from your host to run on ELKS?

We don't have much that can run on ELKS itself at this time (see #1443 for C compiler discussions). But nasm or another assembler could be used to generate binaries for ELKS. We would need to generate the output ELKS-compatible a.out header from an ASM file as well for it to be loaded. Of course, our current toolchain will assemble source into a.out ELKS binaries, using AT&T format input.

[charlysan]

Thanks @ghaerr. I meant operating from ELKS itself. I saw the "C compiler" discussion and I understand the difficulties, but I thought that maybe for the case of an assembler operating from ELKS it could be easier than the C compiler (?).

[cocus]

Hi all, this looks so cool! In fact, it's so cool, that I dusted off my 8018x and flashed it with today's tip of master. The text-ui came pre-enabled by default, but that doesn't fit on my ROMFS (I think it's because the actual kernel is placed right after the ROMFS and I should move it a little bit); but I was able to copy the binaries to an SD and run them. I was able to run the fm fine, although really slow (a little bit faster over ethernet for sure!), but cons doesn't do anything interesting, except writing colored-characters all over the screen (either via serial or telnet, with putty or picocom): I would like to run the memopad or something cool, and feel the "d-flat" experience (which I was looking for like 4 months ago, never knew the name of this "windowing" scheme). Do I have to do something special in order for this to work? Thank you!

[ghaerr]

Hello @cocus,

Nice to hear from you, how have you been doing?

but cons doesn't do anything interesting, except writing colored-characters all over the screen

The reason cons doesn't work on your 8018x is that it reads the IBM PC BIOS display screen area at B800:0, which appears to be either random memory or non-existent on your system. It would be possible to hack the direct console with some work, which could then run headless and then use cons to display the pseudo-console, but that's probably not worth the effort, and the keyboard would have to be dealt with separately.

I would like to run the memopad or something cool, and feel the "d-flat" experience

Because of its size, D-Flat and memopad are built in a seperate repository, for both UNIX and ELKS. You can take a look at it over here and build the ELKS version using make -f Makefile.elks.

The text-ui came pre-enabled by default

What are you referring to here?

Thank you!

[cocus]

Nice to hear from you, how have you been doing?

Pretty good! Thank you! How are you doing? I see you were really busy with elks!

The reason cons doesn't work on your 8018x is that it reads the IBM PC BIOS display screen area at B800:0, which appears to be either random memory or non-existent on your system. It would be possible to hack the direct console with some work, which could then run headless and then use cons to display the pseudo-console, but that's probably not worth the effort, and the keyboard would have to be dealt with separately.

Ah! I thought cons was required for memopad to run. I was mistaken. Yes, I agree, completely useless to do this on the headless targets

What are you referring to here?

Oh, if I copy the 8018x.config as my main .config and run the menuconfig, I see that the text-ui was enabled by default on the userland.

Because of its size, D-Flat and memopad are built in a seperate repository, for both UNIX and ELKS. You can take a look at it over here and build the ELKS version using make -f Makefile.elks.

Ah! cool. I just built it (I had to change the path to my elks TOPDIR under the Makefile.elks) but the binary doesn't run on my board, I always get a

# ./memopad
0@�@^: not found
./memopad: 2: Syntax error: ")" unexpected

the file seems to be okay. I copied it to the SD card because it won't fit on the rootfs romfs.

[ghaerr]

the binary doesn't run on my board,

Oh, memopad is built in medium model, since its code size > 64k. That means the kernel needs to have the a.out medium model extension enabled: set CONFIG_EXEC_MMODEL=y and rebuild the kernel. (Sorry if that's a pain to reflash!)

I see that the text-ui was enabled by default on the userland.

I'm sorry I'm being a little dense here, but I still don't know what text-ui is referring to, I don't recall having ever seen that term used within ELKS.

[cocus]

Oh, memopad is built in medium model, since its code size > 64k. That means the kernel needs to have the a.out medium model extension enabled: set CONFIG_EXEC_MMODEL=y and rebuild the kernel. (Sorry if that's a pain to reflash!)

Okay, I'll test this. What are the downsides of enabling this? can I do it and keep it as the default?

I'm sorry I'm being a little dense here, but I still don't know what text-ui is referring to, I don't recall having ever seen that term used within ELKS.

It's the name set on the menuconfig for tui:

[ghaerr]

What are the downsides of enabling this? can I do it and keep it as the default?

There are no downsides. Its only a config option because of the extra space taken up in the kernel, which is small. Perhaps it should be on by default. (Though not many programs are medium model at this point, I think memopad is the only one :)

It's the name set on the menuconfig for tui:

Geez, thanks. Yes, I named that and promptly forgot what I called it. Are you thinking it should be off by default for 8018x?

[cocus]

There are no downsides. Its only a config option because of the extra space taken up in the kernel, which is small. Perhaps it should be on by default. (Though not many programs are medium model at this point, I think memopad is the only one :)

Well, I enabled it, but it still doesn't run. Could it be because I don't have enough memory? meminfo reports 360kb free.

# ./memopad 
# 
# ./memopad 
# 

Geez, thanks. Yes, I named that and promptly forgot what I called it. Are you thinking it should be off by default for 8018x?

Well, I think it might be a good idea to turn it off by default, just for the sake of reclaiming some bytes on the romfs

[ghaerr]

Does trying to run memopad act like a shell script command failure, or does the shell return an exec error? If the former, the a.out magic number is not correct, which means either no kernel support or bad executable. Post your memopad and I’ll run it when I get home.

Push text ui off on your open PR and we’ll commit it there.

On May 20, 2023, at 4:46 PM, Santiago Hormazabal @.***> wrote:



There are no downsides. Its only a config option because of the extra space taken up in the kernel, which is small. Perhaps it should be on by default. (Though not many programs are medium model at this point, I think memopad is the only one :)

Well, I enabled it, but it still doesn't run. Could it be because I don't have enough memory? meminfo reports 360kb free.

Geez, thanks. Yes, I named that and promptly forgot what I called it. Are you thinking it should be off by default for 8018x?

Well, I think it might be a good idea to turn it off by default, just for the sake of reclaiming some bytes on the romfs

— Reply to this email directly, view it on GitHubhttps://github.com/jbruchon/elks/issues/1366#issuecomment-1556043856, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC3OFZJ4X4JL7K3HNIGTBZDXHFJUDANCNFSM53JQFRHQ. You are receiving this because you were mentioned.Message ID: @.***>

[cocus]

Does trying to run memopad act like a shell script command failure, or does the shell return an exec error? If the former, the a.out magic number is not correct, which means either no kernel support or bad executable. Post your memopad and I’ll run it when I get home. Push text ui off on your open PR and we’ll commit it there.

I just pushed the config with the tui disabled, but I need to enable the medium model config. I'll do that in a second.

When I run it, no error is shown:

# cd /mnt
# ./memopad
# echo $?
0

[ghaerr]

Ok it is no longer the shell script syntax error, so the kernel is or should be loading the executable. It could be failing due to no memory, but what you are showing is the exit code, not the exec errno.

It could also be trying to write to IBM PC console memory instead of emitting ANSI sequences. I think there may be some ifdef ELKS in the source code that forces that. I will have a look in a couple hours.

On May 20, 2023, at 4:56 PM, Santiago Hormazabal @.***> wrote:



Does trying to run memopad act like a shell script command failure, or does the shell return an exec error? If the former, the a.out magic number is not correct, which means either no kernel support or bad executable. Post your memopad and I’ll run it when I get home. Push text ui off on your open PR and we’ll commit it there.

I just pushed the config with the tui disabled, but I need to enable the medium model config. I'll do that in a second.

When I run it, no error is shown:

cd /mnt

./memopad

echo $?

0

— Reply to this email directly, view it on GitHubhttps://github.com/jbruchon/elks/issues/1366#issuecomment-1556044915, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC3OFZKUFXDPNA7DXKIESEDXHFKZPANCNFSM53JQFRHQ. You are receiving this because you were mentioned.Message ID: @.***>

[ghaerr]

Hello @cocus,

I enabled it, but it still doesn't run. Could it be because I don't have enough memory? meminfo reports 360kb free.

It looks like memopad needs an 84k code segment and is requesting (and getting) a 65k data segment (see second screenshot for these), which matches the 147k or so shown by ps:

Can you post a meminfo of your system after boot and login? That will show what the free memory looks like.

I'm also trying to track down whether sh or sash will display an error when errno=ENOMEM is returned from exec, which should be happening, but likely is not.

(I'd like to add an option to meminfo that would make it easier to show just application used/available memory, rather than all the other stuff... its getting kind of long).

Thank you!

[cocus]

Can you post a meminfo of your system after boot and login? That will show what the free memory looks like.

# meminfo
  HEAP   TYPE  SIZE    SEG   TYPE    SIZE  CNT
  ab7e   SEG     16   1080   CSEG    7056    1
  ab9a   INT      6
  abac   INT      6
  abbe   INT      6
  abd0   INT      6
  abe2   BUFH   216
  acc6   TTY     80
  ad22   TTY     80
  ad7e   SEG     16   1239   DSEG    5808    1
  ad9a   SEG     16   13a4   free   24864    0
  adb6   SEG     16   25e6   DSEG   24864    1
  add2   SEG     16   2bf8   DSEG    2352    1
  adee   SEG     16   19b6   CSEG    3632    1
  ae0a   SEG     16   1a99   free    2320    0
  ae26   SEG     16   1b2a   CSEG   43968    1
  ae42   SEG     16   2c8b   free  341840    0
  ae5e   free 20898
  Heap/free   21646/20898 Total mem  456704
  Memory usage  446KB total,   86KB used,  360KB free
# /mnt/memopad
# 

[ghaerr]

Thanks @cocus. You're running this from the external headless console to Putty, right? Can you do a ps and tell me what the TTY number is you're running on ? memopad does try to do something slightly different, by not translating all the CP 437 characters to ANSI Unicode sequences, when running on /dev/tty1 but that shouldn't affecting this. I can't yet determine whether its running, crashing, or the kernel is rejecting it with ENOMEM and /bin/sh isn't saying so. The meminfo above shows plenty of room for the executable. If you'd like to post memopad here, I'll try running it on QEMU (or you can try that also - the D-Flat Makefile.elks copies it to ELKS /root, I happen to notice).

[cocus]

Thanks @cocus. You're running this from the external headless console to Putty, right? Can you do a ps and tell me what the TTY number is you're running on ? Yes, this is through the headless console, The same happens when I run it through telnet, although with less available RAM. I'm using picocom, but the same happens with Putty.

# ps
PID   GRP  TTY USER STAT CPU  HEAP  FREE   SIZE COMMAND
1     0      root    S   0  3072  2016  12864 /bin/init 
6     6    1 root    S   1  1150 13234  68832 -/bin/sh 
11     6    1 root    R   0  1024  1176  11264 ps

memopad does try to do something slightly different, by not translating all the CP 437 characters to ANSI Unicode sequences, when running on /dev/tty1 but that shouldn't affecting this. I can't yet determine whether its running, crashing, or the kernel is rejecting it with ENOMEM and /bin/sh isn't saying so. The meminfo above shows plenty of room for the executable. If you'd like to post memopad here, I'll try running it on QEMU (or you can try that also - the D-Flat Makefile.elks copies it to ELKS /root, I happen to notice).

Here it is: https://www.dropbox.com/s/fhb81xqzvkhk2oe/memopad?dl=0

[ghaerr]

Thanks, your version runs fine on QEMU, and is the same as mine.

I see that your system is running headless on /dev/tty1. That means memopad (the D-Flat library) is thinking the system is running on the ELKS console.

Try rebuilding memopad with the following patch and rerunning it:

diff --git a/tty.c b/tty.c
index a519134..030a84f 100644
--- a/tty.c
+++ b/tty.c
@@ -110,6 +110,7 @@ int tty_init(enum ttyflags f)
         once = 1;
         memcpy(&t, &oldterm, sizeof(t));
         iselksconsole = tty_iselksconsole(1);
+        iselksconsole = 0;
     }
     tty_enable_unikey();
     if (flags & FullBuffer)

[cocus]

I changed that, re build it, and tested it again, but this time we're back with the old errors:

# /mnt/memopad
0@�@^: not found
/mnt/memopad: 2: Syntax error: ")" unexpected

I'm not sure why, the medium model thing is enabled on my kernel.

[ghaerr]

Well, something is goofed up for sure. To determine if you've got a valid executable you can run chmem:

ELKS 0.6.0

login: root
# chmem ./memopad 
 TEXT  FTEXT   DATA    BSS   HEAP  STACK  TOTDATA  TOTAL
42736  41520  24128   6944  65535      0    65520 149776 ./memopad
# 

If it shows as valid (with approx 84k text and heap set to 65K which means max heap), then it means that your kernel isn't running CONFIG_EXEC_MMODEL. Otherwise, we've got a build issue with make elks over at D-Flat.

[cocus]

If it shows as valid (with approx 84k text and heap set to 65K which means max heap), then it means that your kernel isn't running CONFIG_EXEC_MMODEL. Otherwise, we've got a build issue with make elks over at D-Flat.

i've got the same output as you, so the MMODEL is not set then? let me clean everything on the elks topdir and re-build the images

# /mnt/chmem /mnt/memopad 
 TEXT  FTEXT   DATA    BSS   HEAP  STACK  TOTDATA  TOTAL
42736  41520  24128   6944  65535      0    65520 149776 /mnt/memopad
# /mnt/memopad
0@�@^: not found
/mnt/memopad: 2: Syntax error: ")" unexpected

[ghaerr]

Sounds like CONFIG_EXEC_MMODEL is not set. However, your chmem should be slightly different than mine with the iselksconsole = 0; patch included.

[cocus]

hey hey! this is so painfully slow over serial, that I might die! but it works!!!! how cool is this!

I see the uart TX led constantly on, even while not doing absolutely anything on the terminal... it might be expected I guess

[ghaerr]

this is so painfully slow over serial, that I might die! but it works!!!! how cool is this!

Well it does look cool, even it it's hellishly slow?! Kind of neat that your ROM based 8018x is computing all this :)

What baud rate are you running? I'm not yet certain whether the bottleneck is just the 80c188eb calculating D-Flat stuff, or perhaps a dumb virtual screen to ANSI sequence conversion routine I wrote. I haven't actually ever run this over actual serial, only emulated on a fast laptop over pseudo terminals!

I see the uart TX led constantly on, even while not doing absolutely anything on the terminal...

I just happened to notice something similar when coming up with the iselksconsole patch. I'll investigate that some more and also try to come up with a mechanism other than the current patch for allowing D-Flat to run on ELKS console and serial. It is also possible the continuous transmitting could be part of the speed problem you're seeing.

[cocus]

What baud rate are you running? I'm not yet certain whether the bottleneck is just the 80c188eb calculating D-Flat stuff, or perhaps a dumb virtual screen to ANSI sequence conversion routine I wrote. I haven't actually ever run this over actual serial, only emulated on a fast laptop over pseudo terminals!

I'd say it's both the CPU and the serial. It's running at 9600 by default (since the headless driver configures it by default, even if the actual driver supports faster baudrates). I've tried 38400 bps before, and it works, but the driver always initializes it at 9600 :(.

Running it thru telnet is "somewhat" faster, but still slow, so you can get an idea that the problem is the CPU more or less. Even if it runs at 16MHz.

I just happened to notice something similar when coming up with the iselksconsole patch. I'll investigate that some more and also try to come up with a mechanism other than the current patch for allowing D-Flat to run on ELKS console and serial. It is also possible the continuous transmitting could be part of the speed problem you're seeing.

Yeah, no problems, just seeing this is amazing. Thank you for your efforts!

[ghaerr]

I've tried 38400 bps before, and it works, but the driver always initializes it at 9600 :(.

You're welcome to submit a patch for CONFIG_ARCH_8018X to default serial baud rate to 38400. I'd do it for IBM PC except I think others may guess/expect 9600.

I'd be interested to know more accurately how much the baud rate affects the results here. Perhaps even running 57600 would help. If the speedup is significant, its not CPU bound. Another idea would be to implement interrupt-driven transmit, instead of busying looping for output. And memopad shouldn't be looping or continuously outputting byte streams, I'll look into that on QEMU.

Now you need a mouse (somehow). Is that possible on that system (there's only one serial port, right)?

[cocus]

You're welcome to submit a patch for CONFIG_ARCH_8018X to default serial baud rate to 38400. I'd do it for IBM PC except I think others may guess/expect 9600.

I'd be interested to know more accurately how much the baud rate affects the results here. Perhaps even running 57600 would help. If the speedup is significant, its not CPU bound. Another idea would be to implement interrupt-driven transmit, instead of busying looping for output. And memopad shouldn't be looping or continuously outputting byte streams, I'll look into that on QEMU.

It'd be nice to know how to force the default speed to 38400, haven't figured how to do that myself.

Now you need a mouse (somehow). Is that possible on that system (there's only one serial port, right)?

It has two serial ports, but the secondary one was never enabled, nor any drivers added to it. However, using the main serial port and a console emulator, I can use the mouse without any issues! It works fine! I was even able to drag and drop a window, slowly as hell, but they moved!

[ghaerr]

It'd be nice to know how to force the default speed to 38400, haven't figured how to do that myself.

The 8018x port is using a special console implemented in elks/arch/i86/drivers/char/console-serial-8018x.c. In that file, the console_init function calls update_port which references the associatedstruct tty found in ntty.c. In ntty.c the struct termios def_vals is initialized with B9600 | CS8 which could be ifdefed CONFIG_ARCH_8018X and set to B38400 | CS8 (or perhaps set a DEFAULT_BAUD earlier in the file with an ifdef there).

[EDIT: Try 57600 or more, it would be interesting to see what the highest value the 8018x might support. At some point too high a baud rate may case loss of data from the mouse stream... ].

using the main serial port and a console emulator, I can use the mouse without any issues!

You mean your terminal emulator allows using a mouse, which is interpreted and then sent out to the serial input stream by D-Flat and moves the mouse? That's pretty cool, since I initially designed D-Flat to take mouse movements from the macOS Terminal, but didn't realize this could be used with a serial-based terminal emulator that interfaced with a mouse and does the same thing! Very nice!

[cocus]

The 8018x port is using a special console implemented in elks/arch/i86/drivers/char/console-serial-8018x.c. In that file, the console_init function calls update_port which references the associatedstruct tty found in ntty.c. In ntty.c the struct termios def_vals is initialized with B9600 | CS8 which could be ifdefed CONFIG_ARCH_8018X and set to B38400 | CS8 (or perhaps set a DEFAULT_BAUD earlier in the file with an ifdef there).

Aha, I'll try this! Good idea.

[EDIT: Try 57600 or more, it would be interesting to see what the highest value the 8018x might support. At some point too high a baud rate may case loss of data from the mouse stream... ].

I'd love to use even more than that, but the problem is the FCPU that I have (which is set by the external crystal). The baudrate generator is really off when going above 38400. Usually my FTDI USB-serial doesn't even read anything, and I've seen this on the logic analyzer that the timings are really off, as the theory said.

It's funny because some other user with a different clock could easily use the higher baudrates without issues!

You mean your terminal emulator allows using a mouse, which is interpreted and then sent out to the serial input stream by D-Flat and moves the mouse? That's pretty cool, since I initially designed D-Flat to take mouse movements from the macOS Terminal, but didn't realize this could be used with a serial-based terminal emulator that interfaced with a mouse and does the same thing! Very nice!

Yep, I did exactly that. It worked on my telnet and picocom, didn't test PuTTY but it usually works as well.

[tyama501]

(I'd like to add an option to meminfo that would make it easier to show just application used/available memory, rather than all the other stuff... its getting kind of long).

I think so too. It would be nice if meminfo can show names of applications like FreeDOS mem. Is it hard to do that?

FreeDOS mem

[ghaerr]

It would be nice if meminfo can show names of applications like FreeDOS mem. Is it hard to do that?

It isn't that straightforward, but certainly could be done. I've been wanting the same feature as well for a while. Let me take a look at how we might get this working :)