Questions about Fiwix

[ghaerr]

Hi @mikaku,

This isn't an issue, but rather some quick questions about your project. First off, wow, what a nice project!! Very impressive how you've built such a straightforwardly-coded UNIX-like kernel, I find the sources easy to read and understand, which isn't always the case for complex kernels :)

I'm the maintainer over at ELKS, which is a Linux kernel, C library and applications for 8086 and compatible CPUs in real mode, running a segmented architecture. I was thinking it might be fun to jump in and contribute to Fiwix. However, my development environment is macOS. Do you think that might be much of an issue, substituting say x86_64-linux-musl-gcc for CC? I thought to ask before diving in to see what you think might be other gotcha's in the kernel build. We had the same issue at ELKS 3 years ago, but now the entire kernel, C lib, applications and images can all be built on macOS.

It's pretty cool how you've got a framebuffer console and /dev/fb running. I was thinking it might be fun to port over Microwindows or Nano-X over, which, now that Fiwix has UNIX sockets, should run easily directly on top of 16/24/32bpp framebuffer, and could use serial mouse rather than a dedicated kernel mouse driver. Any interest in that?

At ELKS, we have a nicely-working TCP/IP stack and application set, although in our case it runs in userland due to size constraints, which I wouldn't recommend here. But it has a nice state machine, which could possibly be somewhat easily inserted under the Fiwix socket code. Looking closer, I see it probably has the wrong license though.

Finally, is there a build script for the FiwixOS binaries, or is that all magic for the time being?

Thank you for your Fiwix project, I'm having fun reading the kernel code. Nicely done!

[ghaerr]

Thanks @mikaku and @rick-masters for all your help. I'm trying to call do_mmap inside fb_ioctl and have possibly run into a problem (bear with me, I'm a bit new at fully understanding kernel page mapping and Fiwix). I'm doing:

unsigned int addr = do_mmap(NULL, (unsigned int)video.address, video.memsize, PROT_READ|PROT_WRITE,
        MAP_PRIVATE|MAP_ANONYMOUS, 0, P_MMAP, 0, NULL);

It seems that the physical (or perhaps already mapped kernel virtual address?) for the framebuffer is 0xfd00000. I'm looking to map that address (or its predecessor with the real VESA hardware address?) into the process address space, at any available address. The do_mmap code following is causing an -EINVAL return, presumably because one can't map addresses higher than PAGE_OFFSET into a user process address space:

    if(start > PAGE_OFFSET || start + length > PAGE_OFFSET) {
        return -EINVAL;
    }

Any thoughts on how to proceed?

Thank you!

[ghaerr]

I also don't yet understand this code in fb_init:

    /*
     * Frame buffer memory must be marked as reserved because its memory
     * range (e.g: 0xFD000000-0xFDFFFFFF) might conflict with the physical
     * memory below 1GB (e.g: 0x3D000000-0x3DFFFFFF + PAGE_OFFSET).
     */
    from = (unsigned int)video.address - PAGE_OFFSET;
    bios_map_reserve(from, from + video.memsize);

Are the kernel virtual addresses already mapped into the user process space, and/or does Fiwix keep seperate page tables for kernel and user mode? I suppose if 0xFD000000 is already mapped into the process space, perhaps it only needs to be given read/write permission for the user page table entry for the same address. I must admit I don't yet understand Fiwix enough to know how the system actually runs regarding virtual/physical address mappings etc.

[ghaerr]

I am guessing here that do_mmap is the wrong function to use - it builds a vma entry and waits for a page fault to map, right?

Instead, something like map_kaddr but for user address space (would that be map_page?) that would basically copy the existing kernel address page table entries for the 16MB at 0xFD000000 to some available user virtual address and mark them as present? I'm not quite able to determine exactly what code should be written for that.

It seems strange that 0xFD000000 would always be the VESA FB address; does multiboot setup protected mode page table entries for a kernel mode framebuffer address before passing control to Fiwix? Because the kernel/multiboot.c code just copies the address, which always seems to be 0xFD000000 to video.address, and doesn't create a Fiwix page table entry for it that I can see.

Sorry for all the questions! I hope to learn a bit more and will then get a graphical windowing environment running on Fiwix.

[mikaku]

Yes, Fiwix memory subsystem is not as good as we all would want, and there is no documentation either. Please, keep in mind that this is a kernel made by only one person (mostly).

I 'd like to rewrite the memory subsystem and fix some pending problems with RAMdisk drives and other aspects, but sometimes it's difficult to have a good window of spare time to do it with calm.

The framebuffer is mapped in a way that is not expected to be in the user address space. As I stated above, the only way for a user program to write to the /dev/fb0 is using the basic semantics (open, write and close). I know that this slows down the access to the framebuffer. Your Microwindows and X11 will want to access the framebuffer memory directly.

I think the function do_mmap() won't be of much help here. As you already noticed, you need something more like map_page but for a range of addresses instead of a single page.

I cannot answer all your questions quickly because there are some parts of the kernel that I forgot how I implemented them and it forces me to re-read and test de code to refresh my memory, which wants time.

[ghaerr]

Please, keep in mind that this is a kernel made by only one person (mostly).

Don't worry about that, I am already very impressed with what you have produced. Excellent job!!! I am really enjoying Fiwix, and find the coding very straightforward and well written.

Fiwix memory subsystem is not as good as we all would want

I have spent a couple more hours looking at code, and actually the memory and paging subsystem seems quite understandable, although of course complex. It is just taking me a bit of time myself to come to speed with 32-bit protected mode paging and memory management.

The framebuffer is mapped in a way that is not expected to be in the user address space.

Yes, it seems so, but I have a couple ideas below on how this might get done.

you need something more like map_page but for a range of addresses instead of a single page.

I am thinking of the following: it seems that normally, the kernel virtual addresses (KVA) start at 0xC0000000 (PAGE_OFFSET), and user addresses at 0x10000 up to 0x4000000 (MMAP_START) which includes the heap, and mmaps mapped from there up to the bottom of stack, where the stack starts growing downwards at PAGE_OFFSET.

Thus, yes it does seem there is not a good space to map the framebuffer address, but I am thinking it could be placed as a normal mmap (P_MAP) section by calling get_unmapped_vma_region to find a space above MMAP_START, then call vma_add_region to reserve it (like do_mmap does). However, instead of then waiting for a page fault, the kernel page table entries for video.address (0xFD000000) could be copied and new user page table entries created with PAGE_PRESENT flags in the vma region just reserved.

In this way, a vma region would get allocated for the 16MB framebuffer in the normal mmap user address space, but get immediately marked present using the values from the existing kernel framebuffer page table entries.

Thus, something like this:

   unsigned int start = get_unmapped_vma_region(video.memsize);
   if(!(vma = (struct vma *)kmalloc(sizeof(struct vma)))) {
            return -ENOMEM;
    }
    memset_b(vma, 0, sizeof(struct vma));
    vma->start = start;
    vma->end = start + video.memsize;
    vma->prot = PROT_READ|PROT_WRITE;
    vma->flags = MAP_PRIVATE|MAP_ANONYMOUS;
    vma->offset = 0;
    vma->s_type = P_MMAP;
    vma->inode = NULL;
    vma->o_mode = 0;
    vma->object = NULL;
    for (unsigned int addr = vma->start; addr < vma->end; addr += 4096) {
        unsigned int kaddr = get_kernel_pte_addr(video.addr + addr - vma->start);
        map_page(current, addr, kaddr, PROT_READ|PROT_WRITE);
    }

The new function get_kernel_pte_addr would return the physical address mapping from the kernel page table entry for the passed kernel virtual address (starting from 0xFD000000).

I cannot answer all your questions quickly

That's ok, thank you very much for your help! I am enjoying learning by using your project :)

[ghaerr]

Great news! Microwindows running on Fiwix!!

https://github.com/mikaku/Fiwix/assets/11985637/185ced43-cf5a-4ea3-ba7f-801935c3d0e3

I got the framebuffer mapping working using the idea above. At the moment no mouse and no keyboard, but hey, kind of neat :)

The VMA region mapped for the framebuffer needs unmapping before the process exits, and it seems that the kernel fb_close routine is not automatically called on the open framebuffer file descriptor. I've worked around that problem but will report back more issues.

One can also see the fbcon cursor on the left side of things. This can all be worked around with some more ioctls. Continuing to port Microwindows drivers over to Fiwix.

[ghaerr]

[ghaerr]

Status Update

I finished with a full port of Microwindows to Fiwix, and now both Win32 and Nano-X APIs are working, along with the mouse and keyboard. It's only tested on QEMU, and with a few exceptions, most demo programs are working well. I must say, it's been a pleasant experience working with both the Fiwix kernel and very complete development environment!

Things Added

• Added three ioctls to Fiwix framebuffer driver: map the framebuffer address to the current process space, and return X and Y resolution when a VBE 32-bit RGBA framebuffer is specified at boot.
• Framebuffer is unmapped from address space on close of file descriptor.
• config file to build Microwindows on Fiwix from scratch with no dependencies.

Things not finished

• Framebuffer address map only works for a single open call (!).
• It appears that Fiwix process exit does not call sys_close on each open file descriptor, which causes a hash table error on release_page. This is a problem when an application exits abnormally.
• The kernel doesn't remove the fbcon cursor when calling ioctl(ttyfd, KDSETMODE, KD_GRAPHICS). Calling ioctl(ttyfd, KDSETMODE, KD_TEXT) may want to restore the fbcon text console buffer.

Potential Kernel Issues Found

• Last line of console is sometimes not scrolled properly; can be duplicated using vi and scrolling up/down but also seen from shell on rarer occasions.
• No named pipes, so Nano-X applications are linked directly with the NX server, instead of running client-server.
• No struct dirent or DT_DIR using . Not a big deal.
• Running bin/demo-font causes Page Fault then system panic in release_page. Unknown reason, could be NULL pointer in application? (Screenshot available).

Wishlist

• Add FreeType 2.10.4 to FiwixOS distribution; this allows for much better font support.
• Add Microwindows to FiwixOS distribution (will provide commit for this)
• Need C library header to put new framebuffer ioctls - IO_FB_ADDR, IO_FB_XRES, IO_FB_YRES.
• Figure out libgcc.a problem on cross compiler for macOS so 64-bit functions can be used.
• Building Microwindows on Fiwix is quite slow... is this possibly because of MINIX v2 filesystem use?

For testing, QEMU was setup to pass through the host mouse as a MS mouse, which is then decoded again by the Microwindows mouse driver on /dev/ttyS0. This was done using the following QEMU invocation:

qemu-system-i386 \
    -kernel fiwix -append "ro root=/dev/hda2 bga=1024x768x32" \
    -drive file=FiwixOS-3.3-i386.raw,format=raw,cache=writeback,index=0 \
    -drive file=/Users/greg/net/fwsrc3.img,format=raw,cache=writeback,index=1 \
    -chardev msmouse,id=chardev1 -device isa-serial,chardev=chardev1,id=serial1 \
    -net none \
    -boot d \
    -machine pc \
    -cpu 486

With a bit more work, the X11->Nano-X libraries could be built, which would allow for some X11 programs to run. But this isn't very feasible until we get Nano-X client/server running and X11 headers and other X11 client GUI source on Fiwix.

All-in-all, pretty cool! More screenshots and/or screencasts available if wanted.

[mikaku]

Great news! Microwindows running on Fiwix!!

This is ... well, I don't have words to express how I feel right now. Amazing!, really amazing!. Good Job Gregory!

Fiwix is pronounced as 'few weeks' because that should be the average time you need to learn how it works (and start tinkering). Definitely, you and @rick-masters made this name a good choice! :-)

Added three ioctls to Fiwix framebuffer driver: map the framebuffer address to the current process space, and return X and Y resolution when a VBE 32-bit RGBA framebuffer is specified at boot.

Are those ioctls fully compatible with Linux?

It appears that Fiwix process exit does not call sys_close on each open file descriptor, which causes a hash table error on release_page. This is a problem when an application exits abnormally.

Yes it does: https://github.com/mikaku/Fiwix/blob/5bb7de72145dce8fc6acd9c41ac9f7536f22ae6e/kernel/syscalls/exit.c#L73-L77

The kernel doesn't remove the fbcon cursor when calling ioctl(ttyfd, KDSETMODE, KD_GRAPHICS). Calling ioctl(ttyfd, KDSETMODE, KD_TEXT) may want to restore the fbcon text console buffer.

Hmm, you're right. I think this ioctl only works in the VGA console (vgacon) but in fbcon, it doesn't takes the cursor into account.

Last line of console is sometimes not scrolled properly; can be duplicated using vi and scrolling up/down but also seen from shell on rarer occasions.

Yes, I know this issue. There are still some VT100 ANSI sequences not supported.

No named pipes, so Nano-X applications are linked directly with the NX server, instead of running client-server.

FIFOs (named pipes) are fully supported.

No struct dirent or DT_DIR using . Not a big deal.

The header dirent.h exist, of course, otherwise it wouldn't be possible to make a simple directory listing. And regarding the DT_DIR type, check the file include/fiwix/dirent.h.

Running bin/demo-font causes Page Fault then system panic in release_page. Unknown reason, could be NULL pointer in application? (Screenshot available).

I don't know. Where is the screen shot?

Add FreeType 2.10.4 to FiwixOS distribution; this allows for much better font support.

Sure, as long as we can have a graphic environment, the number of packages will increase a lot. It will be just a matter to start porting packages.

Add Microwindows to FiwixOS distribution (will provide commit for this)

Just add a new entry in the script makeall.sh.

Figure out libgcc.a problem on cross compiler for macOS so 64-bit functions can be used.

Perhaps your cross compiler was built without libgcc. I recommend you to go to https://wiki.osdev.org/GCC_Cross-Compiler and rebuild a new one.

Building Microwindows on Fiwix is quite slow... is this possibly because of MINIX v2 filesystem use?

QEMU under Linux includes the option -enable-kvm to speed up emulation. I don't know if there is something similar when QEMU is running under macOS. In any case, block I/O performance in Fiwix is not good, I plan to rewrite it using I/O queues. I'm reading a lot of documentation on the subject. My ToDo list is enlarging it a lot lately. :-)

With a bit more work, the X11->Nano-X libraries could be built, which would allow for some X11 programs to run. But this isn't very feasible until we get Nano-X client/server running and X11 headers and other X11 client GUI source on Fiwix.

I'ld love to see xeyes running on Fiwix. :-)

All-in-all, pretty cool! More screenshots and/or screencasts available if wanted.

I guess that the lack of pty makes impossible to have a terminal under Microwindows/Nano-X. That's another pending feature for the near future.

In all, you did an incredible progress in just a few days. Congratulations!!! Seeing all these graphic screenshots is like a dream come true. Thank you very much!

[ghaerr]

Fiwix is pronounced as 'few weeks'

I was going to ask what Fiwix stood for! :)

Are those ioctls fully compatible with Linux?

No. For the first round, I punted as it was enough work figuring out how to map an existing KA into a user VA. Yes, the ioctls should be made compatible.

That's a bit more of a problem for me as, while adding (header) files to the kernel would be easy since they're in the Fiwix repo, I haven't yet figured out how to add files to the C library and/or non-repo /usr/include distribution, and don't want to go through a full distribution "make" to create a runtime image during a fast development cycle. This will be more of a problem given my preferred non-Linux (macOS) development environment.

IMO, what is needed to solve this problem would be the ability to (somehow) add (header?) files to a host-based repo, from which there could be a way to very quickly generate an entire FiwixOS disk image. This is exacerbated by the need to compile everything on a running Fiwix QEMU installation. I have some ideas on this regarding a dual-mechanism to create Fiwix images (both self-hosted and host-hosted) but I'm not sure that's within the goals of Fiwix. More on this later if your are interested.

It appears that Fiwix process exit does not call sys_close on each open file descriptor, Yes it does:

Hmm, that does seem to. There's still a problem, when SIGINT is generated from ^C to exit Microwindows, it seems fb_close does not get called for some reason. I just checked the kernel/signal.c sources and it seems it should. I'll look further into this.

Yes, I know this issue. There are still some VT100 ANSI sequences not supported.

If you know which ones are required, I can implement them. It would be nice to have vi display properly, since it must be used for development on Fiwix.

FIFOs (named pipes) are fully supported.

Great! I'll change the compilation for Nano-X to client/server and see what happens :) That will be a big improvement.

The header dirent.h exist, of course, otherwise it wouldn't be possible to make a simple directory listing. And regarding the DT_DIR type, check the file include/fiwix/dirent.h.

You're saying both methods of reading directories should be supported... I'll check into that.

Running bin/demo-font causes Page Fault then system panic

I further debugged this, and its failing in the PCF font decoder... not sure exactly why yet, will report more soon. I suspect something to do with malloc incompatibility between systems, but bug is likely in PCF decoder.

Just add a new entry in the script makeall.sh.

I'm looking for that, where is it? I found make_media.sh and install.sh in FiwixOS repo? I haven't yet tried building a system using FiwixOS scripts, not sure that'll work on macOS anyways.

I'ld love to see xeyes running on Fiwix. :-)

Nano-X has xeyes running under X11 compatibility layer... I will compile this up after reviewing X11 issues later.

I guess that the lack of pty makes impossible to have a terminal under Microwindows/Nano-X. That's another pending feature for the near future.

Yes - the terminal demos had to be temporarily removed from the Microwindows build. Not a big deal at the moment, as Nano-X doesn't have a good terminal emulator anyways.

In all, you did an incredible progress in just a few days. Congratulations!!!

Thanks. In years past, I had worked hard on Microwindows such that it could be ported with very few dependencies. After getting a live usermode framebuffer, things progressed quickly.

Microwindows isn't really a desktop GUI environment. It's place in the universe is a full windowing system on small devices or bare hardware. At this point, it is very useful to get Fiwix up to speed with some basics for graphics. I have been thinking of rewriting Microwindows to produce a compositor-based solution. But even then, the problem of what API to bring over is a topic for long discussion, as there are so many, they're all large and very complicated, and mostly incompatible! That might be one of the reasons I like small systems.

[ghaerr]

It appears that Fiwix process exit does not call sys_close on each open file descriptor, Yes it does There's still a problem, when SIGINT is generated from ^C to exit Microwindows, it seems fb_close does not get called for some reason.

Chased this down: yes, fb_close is being called through sys_exit as you indicated. The problem I was seeing had to do with releasing the mapped page table entries for the framebuffer, which I was doing inside the fb_close routine. If that wasn't done before the process exited, the system would crash due to my buggy implementation.

sys_exit unmaps the process' PTEs in release_binary before the file descriptors are closed, which triggered my bug, which is that I misunderstood exactly how Fiwix page tables should be constructed and freed. An upcoming PR will include this proper implementation.

After more reading of Fiwix's memory management, page handling and mmap implementation, I now realize a far better solution will be to use mmap on the framebuffer file descriptor to map the physical framebuffer into user memory, which will match Linux. This gets rid of the incompatible ioctls I initially coded to get things running. It also separates the framebuffer map/unmap from the close handling, which also match the way Linux systems work. If a process doesn't munmap the framebuffer before exit, release_binary will do it automatically via the free_vma_pages and free_vma_regions routines.

I'm in the process of rewriting the Fiwix framebuffer and Microwindows drivers for this now.

I haven't yet figured out how to add files to the C library and/or non-repo /usr/include distribution

I think I can "punt" on this and include the linux-compatible frameuffer header files in the Fiwix source along with the rewritten framebuffer map/unmap handling, and use a MIcrowindows driver that for the time being assumes 1024x768. That will allow a Fiwix distribution to automatically place the updated kernel header files in the proper locations in the FiwixOS distribution. After that, I can update the Microwindows and Fiwix drivers to become fully Linux-compatible so that other framebuffer-based graphics applications or libraries should be portable to Fiwix.

[mikaku]

I'm in the process of rewriting the Fiwix framebuffer and Microwindows drivers for this now.

Yes, Fiwix memory management is not very well designed. Moreover, people from the #boostrappable project have problems because the RAMdisk drives are taking memory from the user space, instead of using memory above PAGE_OFFSET. I mean, Fiwix can have two virtual memory splits 3/1 and 2/2 (user/kernel). All the memory above is not used. If you need a big RAMdisk (e.g. 1500MB), your best path is go to 2/2 and you'll have around 400MB for user space regardless if your system has 4GB of physical memory.

I plan to rewrite how RAMdisk drives memory are used and take advantage of the unused memory. I hope our code will not collide. :-)

After that, I can update the Microwindows and Fiwix drivers to become fully Linux-compatible so that other framebuffer-based graphics applications or libraries should be portable to Fiwix.

That would be fantastic because X11 porting could take advantage of this.

Regarding to add the header files in the C library, this is something I do very often. As you might know FiwixOS uses the Newlib C v4.3 with my changes in sys/fiwix/. You might want to take a look into. You'll find it in the FiwixOS 3.3 Installation CDROM.

After the inclusion the UNIX domain socket and other new system calls, I have added new headers and functions in the C library, so the sources in the current Installation CDROM are already outdated.

Let me know if you want to be able to push PRs into the C library, and I'll create the repository.

[ghaerr]

if you want to be able to push PRs into the C library, and I'll create the repository.

Yes, that would be useful for creating the Linux ioctl compatibility in Fiwix, as otherwise I don't have easy access to any of the C library outside of the running QEMU Fiwix image.

Fiwix memory management is not very well designed. people from the #boostrappable project have problems because the RAMdisk drives are taking memory from the user space, instead of using memory above PAGE_OFFSET.

It seems pretty straight-forwardly designed to me, but I'm still only a week into it :)

Can you explain in a bit more detail what the problem is here? It seems to me that MultiBoot first loads the optional initrd into physical RAM just after the kernel text/data (just before kpage_dir). Then mem_init remaps that initrd RAM above PAGE_OFFSET before loading the final kernel GDT, right?

So how is initrd taking up user space virtual addresses? Aren't all the initrd addresses in KVA?

[mikaku]

Can you explain in a bit more detail what the problem is here? It seems to me that MultiBoot first loads the optional initrd into physical RAM just after the kernel text/data (just before kpage_dir). Then mem_init remaps that initrd RAM above PAGE_OFFSET before loading the final kernel GDT, right?

So how is initrd taking up user space virtual addresses? Aren't all the initrd addresses in KVA?

Initially the Fiwix kernel only had the virtual memory split 3/1 (3GB user / 1GB kernel) and so the maximum available physical memory was 1GB, regardless if your PC had more memory. In this case the RAMdisk drives or the initrd were limited to a maximum of around 950MB, but if you tried to use all the memory, you cannot even login to the system because there would not be enough memory for the user applications.

In #34, @rick-masters suggested a patch to increase the limit of the size in the initrd images to be more than 1GB, which is a requirement for their Bootstrappable project. But somehow we agreed that the patch was a bit tricky.

So, in my to attempt to fix this, I presented a new path, the support to have the virtual memory split 2/2 (2GB user / 2GB kernel) with the new kernel option CONFIG_VM_SPLIT22 (disabled by default). This way you can have easily an initrd of 1.2GB in size and you still have around 750MB for the user space.

Recently though, people in the Bootstrappable project complained that with very large initrd files (let's say 1.5GB) they only have around 450MB for user, even when the system has 4GB of physical memory. They ask if there would be a change to use the unused 2GB of physical memory for the initrd files and the rest for the user space.

I've been testing several (old) Linux kernel versions 2.0, 2.2 and 2.4 and none of these versions is capable to have a RAMdisk drive bigger than 950MB, even when the system has 2GB of physical memory.

I continue thinking how can I implement this in a decent way (no tricky) but so far I'm failing on this.

It's interesting to see that modern Linux kernels have other different memory splits like 2.5GB/1.5GB and 1GB/3GB, to solve this kind of problems.

[mikaku]

If you know which ones are required, I can implement them. It would be nice to have vi display properly, since it must be used for development on Fiwix.

Yes, I know is very annoying. It forces me to keep refreshing the screen with CTRL+L all the time. Sincerely, I'm not sure if this is a lack of support or just a bug in the console.

I'm looking for that, where is it? I found make_media.sh and install.sh in FiwixOS repo? I haven't yet tried building a system using FiwixOS scripts, not sure that'll work on macOS anyways.

The makeall.sh script is used to build all the user space packages. You'll find it in the FiwixOS repository inside the iso/install/pkgs/src/ directory. I recommend you to create a new ATA disk image (4GB) to build the packages and the GNU Toolchain. I have the FiwixOS 3.3 disk image in /dev/hda, and an ATA disk image of 4GB in /dev/hdd that contains all the src/ directory in the FiwixOS repository. Every time I port a new file, I just mount the filesystem /dev/hdd1 into /mnt/disk, then I modify the script makeall.sh with the changes needed by the new package and I test the build.