Questions about Fiwix

[ghaerr]

Hi @mikaku,

This isn't an issue, but rather some quick questions about your project. First off, wow, what a nice project!! Very impressive how you've built such a straightforwardly-coded UNIX-like kernel, I find the sources easy to read and understand, which isn't always the case for complex kernels :)

I'm the maintainer over at ELKS, which is a Linux kernel, C library and applications for 8086 and compatible CPUs in real mode, running a segmented architecture. I was thinking it might be fun to jump in and contribute to Fiwix. However, my development environment is macOS. Do you think that might be much of an issue, substituting say x86_64-linux-musl-gcc for CC? I thought to ask before diving in to see what you think might be other gotcha's in the kernel build. We had the same issue at ELKS 3 years ago, but now the entire kernel, C lib, applications and images can all be built on macOS.

It's pretty cool how you've got a framebuffer console and /dev/fb running. I was thinking it might be fun to port over Microwindows or Nano-X over, which, now that Fiwix has UNIX sockets, should run easily directly on top of 16/24/32bpp framebuffer, and could use serial mouse rather than a dedicated kernel mouse driver. Any interest in that?

At ELKS, we have a nicely-working TCP/IP stack and application set, although in our case it runs in userland due to size constraints, which I wouldn't recommend here. But it has a nice state machine, which could possibly be somewhat easily inserted under the Fiwix socket code. Looking closer, I see it probably has the wrong license though.

Finally, is there a build script for the FiwixOS binaries, or is that all magic for the time being?

Thank you for your Fiwix project, I'm having fun reading the kernel code. Nicely done!

[mikaku]

First off, wow, what a nice project!! Very impressive how you've built such a straightforwardly-coded UNIX-like kernel, I find the sources easy to read and understand, which isn't always the case for complex kernels :)

Thank you very much!, I really appreciate your words.

I'm the maintainer over at ELKS, which is a Linux kernel, C library and applications for 8086 and compatible CPUs in real mode, running a segmented architecture. I was thinking it might be fun to jump in and contribute to Fiwix.

Yes, I know the ELKS project. It helped me a lot when I implemented the UNIX socket interface in Fiwix. As you can see, the code is heavily influenced by ELKS and by very early Linux kernel versions as well.

I was thinking it might be fun to jump in and contribute to Fiwix.

You are very welcome.

Do you think that might be much of an issue, substituting say x86_64-linux-musl-gcc for CC

First of all, and as stated here, it's not expected that you use the compiler of your system to compile a totally different kernel. I recommend you to build a generic cross-compiler (I use GCC 4.7.4) and use that compiler to build the Fiwix kernel.

That said, the Fiwix code is pretty strict to ANSI C89, so as long as the macOS C compiler complies with ANSI C89, it shouldn't have much problems. Perhaps the inline assembly parts are not supported in the macOS CC?. In these cases we can include #ifdef preprocessor directives to help to circumvent small problems. But I don't like the idea to fill the code with excessive #ifdef, as this could reduce readability.

In fact, we already have included some #ifdef preprocessor directives when we added support to build the Fiwix kernel using the tcc in #63. This was necessary as part of the live-bootstrap project by the bootstrapping community. I recommend you to take a look at this project. They have their own IRC channel at Libera.Chat.

It's pretty cool how you've got a framebuffer console and /dev/fb running. I was thinking it might be fun to port over Microwindows or Nano-X over, which, now that Fiwix has UNIX sockets, should run easily directly on top of 16/24/32bpp framebuffer, and could use serial mouse rather than a dedicated kernel mouse driver. Any interest in that?

I'm really interested. One of my goals is to have X11 running on FiwixOS. As you might know, I'm targeting to run FiwixOS on old PCs (386, 486 and Pentium) with networking capabilities. After completing the UNIX socket implementation, I wanted to start including a simple NIC driver (RTL8139?), and a simple TCP/IP stack. Unfortunately, I have to be sincere here, despite I'm an syadmin, I'm not an expert in low level networking at kernel level. That's why I thought this would be a good exercise to increase my knowledge on TCP, IP, ARP, ICMP, etc.

At ELKS, we have a nicely-working TCP/IP stack and application set, although in our case it runs in userland due to size constraints, which I wouldn't recommend here. But it has a nice state machine, which could possibly be somewhat easily inserted under the Fiwix socket code.

I think Fiwix could benefit from your current TCP/IP stack implementation on ELKS. We'll have to include a list of ioctl() options to keep Linux 2.0 compatibility and be able to use GNU network tools, like Linux does. Please, feel free to investigate this further and let me know if I can be of help for you on this process.

Looking closer, I see it probably has the wrong license though.

I'm not an expert in licensing. I'm open to change the Fiwix kernel license if this can attract people to this project, as long as this doesn't affect other projects like Boostrappable that depends of Fiwix. I believe @rick-masters might bring more information on this.

Finally, is there a build script for the FiwixOS binaries, or is that all magic for the time being?

You mean the FiwixOS user land packages?. If so, it's all in the Installation CDROM. Mount that CDROM and you will find a script called install/pkgs/src/makeall.sh to build all the packages, and another one called install/pkgs/src/toolchain/make-toolchain.sh to build the GNU Toolchain (Binutils, GCC and Newlib C library).

Thank you for your Fiwix project, I'm having fun reading the kernel code. Nicely done!

Is a pleasure for me to see people enjoying Fiwix as much as I do.

[ghaerr]

Do you think that might be much of an issue, substituting say x86_64-linux-musl-gcc for CC it's not expected that you use the compiler of your system to compile a totally different kernel. I recommend you to build a generic cross-compiler (I use GCC 4.7.4) and use that compiler to build the Fiwix kernel.

Sorry - x86_64-linux-musl-gcc is a cross compiler, that is exactly what I was suggesting - using (in this case), the statically-compiled gcc running on macOS to compile the Fiwix kernel. x86_64-linux-musl-gcc is available as a download so it doesn't have to be built on macOS to be used (like we have to with ia16-gcc-elf on ELKS).

Perhaps the inline assembly parts are not supported in the macos CC?.

All the inline ASM should compile fine. I was more worried about the other aspects of the build mechanism, including other non-gcc/binutil tools, of which I haven't looked at closely yet.

But I don't like the idea to fill the code with excessive #ifdef, as this could reduce readability.

I completely agree! It is quite refreshing reading Fiwix sources with very few #ifdefs. I have been working for years trying to remove as many from ELKS as possible, but sometimes contributed platforms set that process backwards a bit.

With regards to Fiwix #ifdefs, I notice quite a few #ifdef __DEBUG__ around printk. How about the following idea, which would remove most all of them, thus returning the source readability to very high:

Instead of:

#ifdef __DEBUG__
    printk("\t(inode = %d)\n", i ? i->inode : -1);
#endif /*__DEBUG__ */

use

/* in a header file */
#ifdef __DEBUG__
void dprintk(const char *, ...);        /* printk when debugging on */
#else
#define dprintk(...)
#endif
...
    dprintk("\t(inode = %d)\n", i ? i->inode : -1);

Then modify printk to be vprintk(const char *fmt, va_list p) and have printk and dprintk call it. This also could have the advantage of a runtime or command line boot switch to turn dprintk output on or off without recompilation.

[mikaku]

How about the following idea, which would remove most all of them, thus returning the source readability to very high:

This is a great idea, I've never had satisfied with the excessive amount of #ifdef __DEBUG__ in the code.

The only thing that worries me with this new implementation, is if I'll be able to continue activating debug for a particular code. I mean, now I just add the line #define __DEBUG__ on particular system call and I can see the debug messages of that system call only.

Also, I don't understand the syntax of the line #define dprintk(...). Does that mean that it won't me compiled at all?

[ghaerr]

I don't understand the syntax of the line #define dprintk(...). Does that mean that it won't me compiled at all?

Yes. According to the rules of variadic macros, this particular method effectively replaces the entire line of code with nothing, resulting in no code produced when debug is off.

The only thing that worries me with this new implementation, is if I'll be able to continue activating debug for a particular code.

I see. I had thought, seeing the master Makefile that -D__DEBUG__ was only turned on for the entire system at once, rather than per system portion. I solved this in ELKS by creating a series of debug_xxx macros, where each can be individually turned on/off for finer precision debugging without #ifdefs in the main source. Here's debug.h using something like the following which might work for Fiwix:

#define DEBUG_FILE      0               /* sys open and file i/o*/
#define DEBUG_NET       1               /* networking*/
...
#if DEBUG_FILE
#define debug_file      dprintk
#else
#define debug_file(...)
#endif
#if DEBUG_NET
#define debug_net       dprintk
#else
#define debug_net(...)
#endif

Then, instead of using dprintk directly in the kernel, debug_file would be used in the file I/O code, debug_net for networking, etc. It's a little messy in debug.h, but keeps the rest of the kernel without #ifdefs. I had another thought where a bit mask of subsystems to display debugging information would be passed into dprintk as part of the debug_xxx macro allowing for runtime changing of debug settings without kernel recompilation, but for 16-bit ELKS, it was deemed a bit too expensive in terms of code size.

All-in-all, I like your idea of keeping the kernel sources easy to read, with very few explicit #ifdefs.

I'm going to continue to play with Fiwix and will look further into cross-compiling on macOS, thank you!

[ghaerr]

now I just add the line #define DEBUG on particular system call and I can see the debug messages of that system call only.

Yes, that could be done without the more complex debug.h described above by just passing -D__DEBUG__ to a particular file being compiled and using a single #ifndef __DEBUG__ / dprintk(...) macro in a kernel header file. dprintk could be renamed debug to keep things very simple.

[mikaku]

Yes. According to the rules of variadic macros, this particular method effectively replaces the entire line of code with nothing, resulting in no code produced when debug is off.

This is interesting, I've never heard of it before.

The problem is that the page says "Variadic macros became a standard part of the C language with C99.", which presumably GCC will compile regardless of -std=c89 because they include Extensions, but other compilers like tccmight not accept it.

[mikaku]

If you are using variadic macros, does that mean that GCC and macOS CC are the only compilers you use on ELKS and your code is not C89 strict?

[ghaerr]

but other compilers like tcc might not accept it.

Good point, I had forgotten Fiwix can be compiled with a compiler other than gcc. I'll have to experiment with tcc to see what it accepts. It's kind of nice that Fiwix keeps things simple enough that the complex gcc inline asm isn't used, thus increasing portability.

Speaking of which, I just added CROSS_COMPILE = x86_64-linux-musl- to Makefile, and the kernel compiles without error! It is dying on the final link stage, where my cross-compiler is complaining about an invalid format libgcc.a. I'll try another cross compiler and report results.

[mikaku]

Speaking of which, I just added CROSS_COMPILE = x86_64-linux-musl- to Makefile, and the kernel compiles without error!

You don't need to touch the Makefile to use your cross compiler.

I use the following command:

export CROSS_COMPILE=~/Fiwix/toolchain/4.7.4-elf/bin/i686-elf- ; make clean ; make

[ghaerr]

If you are using variadic macros, does that mean that GCC and macos CC are the only compilers you use on ELKS and your code is not C89 strict?

ELKS can't use macOS cc/clang as it only produces 64-bit code. The portability problem is much worse, as we now have to use a specialized port of GCC known as ia16-elf-gcc, which has all sorts of 8086-style segment extensions (__far pointers, etc.) This was discussed heavily a couple of years back when switching from BCC, a very old compiler, and for ELKS it was deemed ok as the project needed a stronger modern compiler that was able to generate tight code, for speed and space reasons.

That said, its nice to not have a project depend on a specific compiler. Given that Fiwix doesn't use GCC extensions and keeps x86 ASM in (very few) .S files, it seems that all that is needed is a compiler that produces ELF .o output, along with a few (incompatible) linker options.

For ELKS, we have to build the GCC ia16-elf-gcc cross compiler which is quite time consuming, although its only done once and isn't part of the standard "make all" build.

[ghaerr]

It does look like TCC supports variadic macros as a C99 extension. I'm not sure whether using variadic macros breaks the -std=c89 option or not on GCC.

[mikaku]

It is dying on the final link stage, where my cross-compiler is complaining about an invalid format libgcc.a. I'll try another cross compiler and report results.

If you are having problems linking the kernel with libgcc, you might try to #undef the options CONFIG_OFFSET64 and CONFIG_PRINTK64 in include/fiwix/config.h, and then remove the part -L$(LIBGCC) -lgcc in the Makefile.

You'll not have 64bit offsets (EXT2 filesystems bigger than 4GB) but you'll be able to play with Fiwix quickly.

[ghaerr]

If you are having problems linking the kernel with libgcc, you might try

Very cool - that worked! (Using x86_64-linux-musl-gcc). Now I need to figure out how to add the newly-built fiwix into the flat ATA disk raw image I'm using for QEMU.

I'm also downloading i686-elf-gcc and will try that without removing -lgcc to see whether that works. Thanks for the help, I hadn't realized that only the libgcc.a routines for 64-bit arithmetic were being used by gcc output.

[ghaerr]

I'm assuming the ATA raw disk image is EXT2 format, right? We can't write/mount EXT2 on macOS.

We got around this problem on ELKS by writing the mfs (Minix Filesystem) tool that allows creating MINIX filesystems from scratch on any OS. (I didn't say it was easy to build operating systems on macOS, right?! :)

[mikaku]

Now I need to figure out how to add the newly-built fiwix into the flat ATA disk raw image I'm using for QEMU.

You don't need to update the flat ATA disk raw image every time you changed the Fiwix kernel. Just use something like this:

qemu-system-i386 \
       -kernel fiwix -append "ro root=/dev/hda2 bga=1024x768x32" \
       -drive file=FiwixOS-3.3-i386.raw,format=raw,if=ide,cache=writeback,index=0 \
       -net none \
       -enable-kvm \
       -machine pc \
       -cpu 486 \
       -serial pty

[mikaku]

I'm assuming the ATA raw disk image is EXT2 format, right? We can't write/mount EXT2 on macOS.

Yes, it's EXT2, but beware because it uses revision level 0. That is the original EXT2 without any kind of features.

I didn't say it was easy to build operating systems on macOS, right?!

Heh :smiley:

[ghaerr]

-kernel fiwix -append "ro root=/dev/hda2 bga=1024x768x32" \

Fiwix is booting, built on macOS! Very nice. How does the QEMU -kernel option work, does QEMU know about EXT2 filesystems and thus able to redirect file reads to the host OS image, or is it more tricky than that?

Using the i686-elf-gcc toolchain isn't working, i686-elf-ld is failing not understanding the -nostartfiles and other options. So I'll just stay with the working x86_64-linux-musl-gcc toolchain for now, with 64-bit operations turned off.

I understand the Fiwix applications are all being built on Fiwix itself? And then is the output EXT2 image created through QEMU running makeall.sh then kept as the ATA raw disk image?

I suppose in order to give the Microwindows port a test run I'll still need to figure a way to get its source files (or cross-compiled binaries) onto a compatible Fiwix filesystem... I suppose I could try using my mfs to write a MINIX v1 filesystem and boot from that?

Thanks for your help. Getting Fiwix running using macOS is proving lots easier than I had thought.

[ghaerr]

How does the QEMU -kernel option work,

I think I get it now, since Fiwix doesn't depend on any non-standard boot mechanism (and doesn't include its own boot code), the QEMU -kernel and -append options just use a GRUB-style or standard boot to load the kernel, and pass the -append options to the boot loader. Nice.

I haven't found the script that adds a boot loader to the ATA raw disk image, I'll look at that stuff in a bit.

[mikaku]

How does the QEMU -kernel option work, does QEMU know about EXT2 filesystems and thus able to redirect file reads to the host OS image, or is it more tricky than that?

Fiwix is a Multiboot Specification v1 compliant (GRUB loader v1 also called GRUB legacy). When you use the -kernel parameter, QEMU boots the provided kernel and passes it a compliant Multiboot Spec. Fiwix then knows the memory map and other machine features.

I understand the Fiwix applications are all being built on Fiwix itself?

Yes, FiwixOS is what it's called a self-hosted operating system. All the packages has been built under FiwixOS (including the kernel, of course).

And then is the output EXT2 image created through QEMU running makeall.sh then kept as the ATA raw disk image?

The ATA raw disk image is created on my Host OS (Fedora Linux) using the simple command truncate -s 1GB FiwixOS-3.3-i386.raw. Then I use the Installation CDROM to install the FiwixOS there.

[mikaku]

I suppose in order to give the Microwindows port a test run I'll still need to figure a way to get its source files (or cross-compiled binaries) onto a compatible Fiwix filesystem... I suppose I could try using my mfs to write a MINIX v1 filesystem and boot from that?

First, I don't think porting your Microwindows will be trivial. I'd be really surprised if it is.

Also, Fiwix supports Minix v1 and v2 filesystems. Just need to activate the option CONFIG_MINIX in config.h.

[mikaku]

I haven't found the script that adds a boot loader to the ATA raw disk image, I'll look at that stuff in a bit.

The ATA raw disk image was created using the Installation CDROM and installing FiwixOS there. I think that you should try install FiwixOS on your own ATA raw disk to have a better idea of how interact all the pieces.

[ghaerr]

FiwixOS is what it's called a self-hosted operating system. All the packages has been built under FiwixOS (including the kernel, of course).

That's pretty amazing. Especially building gcc and binutils on Fiwix itself.

using the simple command truncate -s 1GB FiwixOS-3.3-i386.raw.

I see. So that command actually only creates a 1GB hole (read as effectively zeros) and then the Installation CDROM actually writes the filesystem onto it in one of Fiwix's kernel-supported formats.

Then I use the Installation CDROM to install the FiwixOS there. The ATA raw disk image was created using the Installation CDROM and installing FiwixOS there.

I see, truly self-hosted, including building the images themselves. Wow.

I think that you should try install FiwixOS on your own ATA raw disk to have a better idea of how interact all the pieces.

Ok, I'll try that.

I don't think porting your Microwindows will be trivial. I'd be really surprised if it is.

Microwindows can run without a keyboard or mouse by configuring it run by only requiring /dev/fb0 in 32/24/16 bpp, with almost no dependencies. I definitely want to try that. Do you support the Linux framebuffer ioctls to get the framebuffer size and address for mmap? If so, then possibly no modifications at all will be required for /dev/fb0 support.

Microwindows has a couple of drivers for the keyboard (scancode mode for full keyboard control w/shift keys etc) and a basic ASCII mode for systems without scan code keyboards. It's got its own self-contained MS or Logitech mouse driver that can run on the serial port, which I think QEMU can possibly redirect into from the host mouse. So I think we can get it running quickly.

Fiwix supports Minix v1 and v2 filesystems.

ELKS only supports Minix v1. The only real benefit here would be to allow creating Fiwix images directly from macOS without running Fiwix itself on QEMU to create the images. I'll look at your suggested approach and learn how you've done it. The other problem with Minix v1 could be the 14-character max filename, which sometimes causes problems these days.

[mikaku]

Do you support the Linux framebuffer ioctls?

Not yet, basically because I've not found a software that can use the framebuffer and do not requires a basic TCP/IP stack.

[ghaerr]

Do you support the Linux framebuffer ioctls?

How can an application get the size (width, height, stride) and address of the VESAFB framebuffer? The Microwindows driver then just mmaps that address into the process and the rest is handled internally.

All we really need is the hardware address of the framebuffer, the rest could be provided manually.

[mikaku]

How can an application get the size (width, height, stride) and address of the VESAFB framebuffer?

No application is using the framebuffer. I only use it to have a better console experience.

The only graphic applications that run on FiwixOS (like DOOM) are using the SVGAlib.

[mikaku]

All we really need is the hardware address of the framebuffer, the rest could be provided manually.

You can use /dev/fb0 to write anything into the framebuffer.

[ghaerr]

I've not found a software that can use the framebuffer and do not requires a basic TCP/IP stack.

Microwindows implements two APIs, a Win32-compatible and an X11-like (Nano-X). The former doesn't require any sockets as the application main loop is compiled into the "win32" app. Nano-X is normally compiled separately using a named FIFO to communicate between multiple applications and Nano-X but we also support linking the application into the server.

Thus, both APIs can work by drawing directly into a mmap-ed framebuffer.

The only graphic applications that run on FiwixOS (like DOOM) are using the SVGAlib.

I see. I'll just have to dig up how SVGAlib gets the framebuffer address, unless it's not using the framebuffer.

You can use /dev/fb0 to write anything to the framebuffer.

Yes - but the application normally opens /dev/fb0, then does a single ioctl to get the framebuffer address. Otherwise, one would need to use the open/read/write/lseek in order to draw, which would be very slow.

[mikaku]

The other problem with Minix v1 could be the 14-character max filename, which sometimes causes problems these days.

Minix v2 supports up to 30-character filenames, but that's not much either.

[ghaerr]

Here's a sample of how Microwindows uses /dev/fb0 to gain access:

    /* open framebuffer file for mmap*/
    if((env = getenv("FRAMEBUFFER")) == NULL)
        env = "/dev/fb0";
    fb = open(env, O_RDWR);
    if (fb >= 0)
    {
        /* mmap framebuffer into this address space*/
        int extra = getpagesize() - 1;
        psd->size = (psd->size + extra) & ~extra;   /* extend to page boundary*/
        psd->addr = mmap(NULL, psd->size, PROT_READ|PROT_WRITE, MAP_SHARED, fb, 0);
        if (psd->addr == NULL || psd->addr == (unsigned char *)-1)
        {
            EPRINTF("Error mmaping shared framebuffer %s: %m\n", env);
            close(fb);
            return NULL;
        }
    }

After this, psd->addr can be directly used as a pointer to the framebuffer, after which 32-bit RGBA can be read/written to.

[EDIT: Oops! I was incorrect: an ioctl is only used (previous to this listing) to get the framebuffer size. The mmap call then takes the framebuffer fd and maps it into the process address space. Will that work on Fiwix?]

[mikaku]

I see. I'll just have to dig up how SVGAlib gets the framebuffer address, unless it's not using the framebuffer.

SVGAlib doesn't use the framebuffer, and if you want to try the software using this library, you cannot boot the kernel using the framebuffer console, but the VGA text console. That is, without the parameter bga=.

Yes - but the application normally opens /dev/fb0, then does a single ioctl to get the framebuffer address. Otherwise, one would need to use the open/read/write/lseek in order to draw, which would be very slow.

I'm sorry, there are no ioctls implemented for the framebuffer.

[ghaerr]

In any case, if there's a way to map the framebuffer address into a process, when booted using framebuffer console, then we can use that method in a Microwindows screen driver to gain access and draw directly. That would be cool. If not done, I will look further into how this might be done, if you're interested.

[mikaku]

The mmap call then takes the framebuffer fd and maps it into the process address space. Will that work on Fiwix?]

Since there is no ioctl calls implemented for the framebuffer, you don't know it's size. But you can open /dev/fb0 and read or write there. You don't need mmap() here, but if you want it you can use it.

Please, check the fb driver to get an idea of what happens when you read or write to /dev/fb0.

[mikaku]

In any case, if there's a way to map the framebuffer address into a process, when booted using framebuffer console, then we can use that method in a Microwindows screen driver to gain access and draw directly.

With the current framebuffer implementation (without ioctl calls), you won't be able to get the memory address of the framebuffer. But you don't need it to read/write there. Just use the /dev/fb0 device.

You can test it with a simple command:

cat /dev/random > /dev/fb0

And see what happens.

[ghaerr]

But you can open /dev/fb0 and read or write there. check the fb driver to get an idea of what happens when you read or write to /dev/fb0.

Yes, I see. This requires that a graphics application wrap all direct memory accesses to a framebuffer with read/write system calls, which will be exceedingly slow.

What is needed would be to somehow map the fb driver's video.address into a process space. To get the width and height, one would probably want ioctls to the frambuffer fd, something like:

width = ioctl(framebuffer_fd, IO_VID_WIDTH, 0);
height = ioctl(framebuffer_fd, IO_VID_HEIGHT, 0);

cat /dev/random > /dev/fb0

Yes, I get that. But low-level framebuffer reading writing is typically done (for speed) with something like this:

/* Draw horizontal line from x1,y to x2,y including final point */
static void
linear32_drawhorzline(PSD psd, int x1, int x2, int y, uint32_t c)
{
    unsigned char *addr = psd->addr + y * psd->pitch + (x1 << 2);
    int width = x2-x1+1;

        int w = width;
        while (--w >= 0)
        {
            *((uint32_t *)addr) = c;
            addr += 4;
        }
}

psd->addr is the framebuffer address, psd->pitch is the stride (bytes per scanline).

[ghaerr]

BTW, once there is direct framebuffer access to a process on Fiwix, X11 can also be ported.

There is a special version of X11 that was built for embedded systems that doesn't use SVGAlib-style drivers, but direct framebuffer access very similar to the above Microwindows code.

So this would be the first step in getting any windowing system running on Fiwix. I can look up the X11 version for you if you're interested, but X11 is very much more complicated to port.

[mikaku]

I think Linux uses some trick when mmap() receives a file descriptor that belongs to the framebuffer, but if you cat /dev/random to /dev/fb0 under Linux, I think you are not writing the framebuffer memory address directly, so it's slower.

Fiwix needs the same trick that Linux does when you call mmap() with a framebuffer file descriptor, to map the framebuffer video memory into the process address space.

[mikaku]

BTW, once there is direct framebuffer access to a process on Fiwix, X11 can also be ported.

I know, that's the reason why I wanted to focus on implementing a TCP/IP stack, which is needed by X11.

[mikaku]

There is a special version of X11 that was built for embedded systems that doesn't use SVGAlib-style drivers, but direct framebuffer access very similar to the above Microwindows code.

I'm really excited to see Microwindows or Nano-X or X11 or any other similar graphic system running under Fiwix. This is the only way to implement the ioctl call to put the keyboard into raw mode, create a mouse driver and also, the pty driver.

[ghaerr]

I think Linux uses some trick when mmap() receives a file descriptor that belongs to the framebuffer

Yes - I believe what happens is that each mmap-able file device has a special fops entry point that can be called to do that for the specific device. So the kernel mmap gets the address by calling the function table entry that ends up calling a fb.c function to get the hardware address to map.

We wouldn't need to go to that extreme initially though, as that requires another entry point for most all drivers. Instead, perhaps an ioctl to fb.c that ends up mapping the hardware address into the process space and returns the virtual address. That would allow getting framebuffer versions of graphics libraries working without waiting for a full TCP/IP stack.

I wanted to focus on implementing a TCP/IP stack, which is needed by X11.

I think that only UNIX sockets are actually required, if all applications are local. Yes, this does require a pretty full higher-level socket implementation, but no TCP/IP underneath.

This is the only way to implement the ioctl call to put the keyboard into raw mode, create a mouse driver and also, the pty driver.

Having ported Microwindows to many OSes, I would suggest doing this in steps:

• Add ioctl to fb driver to map framebuffer address to process and return virtual address
• Add ioctls to return width, height, format and stride of framebuffer
• Port Microwindows and/or other simple framebuffer-based graphics libraries (no keyboard, no mouse)
• Get mouse driver working over serial port, and/or directly into kernel
• Implement /dev/tty keyboard in raw mode (no scan codes)
• Add /dev/tty scan code support to Fiwix kbd driver
• Possibly update fb driver to return Linux-compatible framebuffer ioctl formats
• Add PTY drivers to allow terminal emulators to work
• Port X11 framebuffer version

[ghaerr]

Add ioctl to fb driver to map framebuffer address to process and return virtual address

This should be fairly straightforward, ultimately a few consecutive page table entries need to be created, mapping the hardware address video.address for video.memsize bytes. I haven't looked at how Fiwix handles page mapping, and am interested to learn more about how you do it.

Add ioctls to return width, height, format and stride of framebuffer

I think most of these values are already in the struct video_parms as fb_width, fb_height and fb_pitch or fb_linesize.

[mikaku]

Yes - I believe what happens is that each mmap-able file device has a special fops entry point that can be called to do that for the specific device. So the kernel mmap gets the address by calling the function table entry that ends up calling a fb.c function to get the hardware address to map.

This is exactly what I was trying to say, but my English is not as good as I wanted.

We wouldn't need to go to that extreme initially though, as that requires another entry point for most all drivers. Instead, perhaps an ioctl to fb.c that ends up mapping the hardware address into the process space and returns the virtual address. That would allow getting framebuffer versions of graphics libraries working without waiting for a full TCP/IP stack.

I completely agree.

I think that only UNIX sockets are actually required, if all applications are local. Yes, this does require a pretty full higher-level socket implementation, but no TCP/IP underneath.

That would save a lot of time of implementation.

Having ported Microwindows to many OSes, I would suggest doing this in steps:

Looks like a good plan.

[ghaerr]

Looks like a good plan.

I'll play around with getting the framebuffer mapped in the fb driver. Given that macOS doesn't have the ability to read or write EXT2 virtual filesystems at all, how do you recommend getting source (or binary) files onto and off the root or other mounted Fiwix filesystem easily?

Also, is there a sample Makefile that might be used for cross-compiling user mode executables outside of a running Fiwix? Or is the idea to always use Fiwix for compiling all applications?

[mikaku]

how do you recommend getting source (or binary) files onto and off the root or other mounted Fiwix filesystem easily?

Since Fiwix also supports the ISO9660 filesystem, you might want to consider using that media to import software to your FiwixOS system, by using macOS tools to create CDROM images. And then you might use the Minix filesystem to export them out. Minix v2 would be specially useful as it supports 30-character filenames and up to 1GB of filesystem size.

Another alternative is to use the Paragon Software driver.

Also, is there a sample Makefile that might be used for cross-compiling user mode executables outside of a running Fiwix? Or is the idea to always use Fiwix for compiling all applications?

I used cross-compiling long time ago, in the very early stages of the FiwixOS, when I was creating a self-hosting operating system. Once the system was self-hosting capable, I no longer used that path.

For best results build your software under FiwixOS 3.3, or under an old Linux distro with kernels 2.0 or 2.2 since Fiwix is system call mostly compatible with those kernels. I regularly test the compatibility using the old RedHat Linux 4.2 and 5.2 distributions (both with Linux 2.0.x kernels), and also RedHat Linux 6.2 with kernel 2.2.x. I recommend you to statically link your binaries, to avoid dragging a lot of Linux libraries to FiwixOS.

[rick-masters]

Given that macOS doesn't have the ability to read or write EXT2 virtual filesystems at all, how do you recommend getting source (or binary) files onto and off the root or other mounted Fiwix filesystem easily?

@ghaerr Not sure if this will help you but there is this project for creating and populating ext2,3,4 file systems from a program: https://github.com/gkostka/lwext4

You'll want to use my fork which supports ext2 rev 0 and a couple other fixes/features: https://github.com/rick-masters/lwext4/commits/live-bootstrap

Downloadable here: https://github.com/rick-masters/lwext4/releases/download/v1.0.0-lb1/lwext4-1.0.0-lb1.tar.gz

We use this in live-bootstrap to build an ext2 for booting Fiwix: https://github.com/fosslinux/live-bootstrap/tree/master/steps/lwext4-1.0.0-lb1

This is the program that creates the file system: https://github.com/fosslinux/live-bootstrap/blob/master/steps/lwext4-1.0.0-lb1/files/make_fiwix_initrd.c

[ghaerr]

Thanks @rick-masters, with a little hack to the CMakeLists.txt file, I was able to get your version of lwext4 compiled on macOS! Also got make_fiwix_initrd to build.

I'd like to use the lwext4-mkfs (or possibly lwext4-generic?) tools to read/write from EXT2 V0, as then I could write a script to move things back and forth. I'm a bit concerned about specifying the EXT2 V0 version though: would lwext4-mkfs -i ext_image -e 1 work (where -e 1 specifies the value of F_SET_EXT2_V0 from your branch), or do I possibly need to modify the lwext4-mkfs.c source?

Is lwext4-generic just a test tool or is there a built program that can be used to read/write EXT2 files directly from the command line? I'm trying to determine whether there's anything in the lwext4 source that can be used as command line programs to list, read or write EXT2.

[ghaerr]

Another alternative is to use the Paragon Software driver.

This could potentially be very easy to use, but its a paid program; it says to support EXT2 but I am bit worried about exactly the difference(s) between the V0 and later version of EXT2 filesystem. Is the Fiwix EXT2 V0 incompatible with later versions of EXT2? Unfortunately, I am not much familiar with EXT2, although I am very familiar with MINIX v1.

The idea of doing development on a Fiwix MINIX v1 filesystem is quite possible, except I am not sure how to convert or create the Fiwix 3.3 ATA Raw disk image into a MINIX filesystem, from which I could use my ELKS mfs utility to get files back and forth. If Fiwix requires an image > 65MB, then I suppose I would have to enhance the mfs utility to support V2, of which I think that is almost done, but not tested. Another thought, I suppose, would be to automatically mount a MINIX v1 filesystem with my source code when running Fiwix under QEMU.

In some sense it might be good to figure a way to create a full image of Fiwix 3.3 binaries from a template of such on a host operating system, as then it would be possible to generate a specific distribution based on the users needs or set of source code being worked on. On the other hand, this may not be in the goals of Fiwix.

Sorry for all the trouble; it seems that tools supporting filesystem formats of self-hosted systems are in fact another subtle dependency not always realized when not running Linux.

[ghaerr]

Good news! After fooling around a bit with ELKS mfs and recompiling Fiwix to support MINIX, I was able to easily create EXT2 image from scratch and transfer an entire source distribution of Microwindows onto it, using:

$ cd microwindows
# create 1GB EXT2 filesystem w/30 char names and transfer entire src/ dir to it
$ mfs fwsrc.img genfs -2 -n30 -i32736 -s1000000 src
$ qemu-system-i386 \
    -kernel fiwix -append "ro root=/dev/hda2 bga=1024x768x32" \
    -drive file=FiwixOS-3.3-i386.raw,format=raw,cache=writeback,index=0 \
    -drive file=/Users/greg/net/microwindows/src/fwsrc.img,format=raw,cache=writeback,index=1 \
    -net none \
    -boot d \
    -machine pc \
    -cpu 486
[Then on Fiwix after login]
...
[(root) ~] # mount -t minix /dev/hdb /mnt

This should allow for quick creation of arbitrary source trees to be mounted and compiled within Fiwix. The kernel can be compiled outside and use the QEMU -kernel option for testing. Getting the files off EXT2 will be a bit harder for the moment, as mfs doesn't have a recursive directory copy out, but that can be alleviated for the time being by creating a tar file, copying that out, and running diff -urN on the contents for commit.

[ghaerr]

Also got all Microwindows Win32 API libraries and most applications compiled and linked, without too many issues. Will report small list of unimplemented Fiwix kernel features later.

Rewriting screen driver now for Fiwix, and will attempt mapping framebuffer address into process space hopefully tomorrow.

I'm impressed with the very full development environment all ported and running on Fiwix. Things are a bit slow compiling on QEMU though, but no strange errors at all. Turns out it is quicker to just mount an MINIX v2 filesystem and compile there than to copy source to Fiwix root EXT2 filesystem.

[mikaku]

it says to support EXT2 but I am bit worried about exactly the difference(s) between the V0 and later version of EXT2 filesystem.

The revision level 0 of the EXT2 filesystem is the original EXT2, it don't has any extra feature. For instance, the very next version of EXT2 introduced the sparse superblock after seeing that so many copies of it resulted on a lot of disk space lost on big filesystems. Over time, they have included more and more new features, but Fiwix only supports V0.

I can interact easily with the Fiwix filesystem from my Fedora because any Linux system supports all EXT2 versions. The problem here is that you are using a macOS. But Minix v2 can be of help for you since you made your own driver.

If Fiwix requires an image > 65MB, then I suppose I would have to enhance the mfs utility to support V2,

Yes, with only 64MB of size in the root filesystem you won't be able to do any type of development. GNU Toolchain, Bash, system tools, etc. all them weight 500MB at least. You'll need 1GB which means, you need Minix v2.

[rick-masters]

Thanks @rick-masters, with a little hack to the CMakeLists.txt file, I was able to get your version of lwext4 compiled on macOS! Also got make_fiwix_initrd to build.

I'd like to use the lwext4-mkfs (or possibly lwext4-generic?) tools to read/write from EXT2 V0, as then I could write a script to move things back and forth. I'm a bit concerned about specifying the EXT2 V0 version though: would lwext4-mkfs -i ext_image -e 1 work (where -e 1 specifies the value of F_SET_EXT2_V0 from your branch), or do I possibly need to modify the lwext4-mkfs.c source?

Is lwext4-generic just a test tool or is there a built program that can be used to read/write EXT2 files directly from the command line? I'm trying to determine whether there's anything in the lwext4 source that can be used as command line programs to list, read or write EXT2.

Sounds like you may have found another method, but here are my answers:

Sorry, I don't know about the tools you mentioned. I think they may be test tools. If I remember correctly, I couldn't find any simple command line tools in the project that could be used out-of-the-box. I found that surprising since it seemed like it wouldn't have been too hard for the project to include those.

I just ended up writing my own program. The make_fiwix_initrd.c program is probably your best bet as an example for how to use the library.

Unfortunately, the author of lwext4 does not appear to be maintaining the project. My Issues and PRs haven't received any attention.