Implement mapping framebuffer physical address to user space using mmap

[ghaerr]

Implements kernel enhancements required to compile and run Microwindows on Fiwix. Discussed in https://github.com/mikaku/Fiwix/issues/78#issuecomment-1960604730.

This PR adds the following:

• Adds ability to map arbitrary addresses into user space using newly-defined PAGE_NOALLOC bit set in page table entry. When set, this bit specifies that the mapped address is not a kalloc-allocated page but instead a physical memory address outside of the page pool. This bit is one of three available bits allowed in page table entries, reserved for operating system software management.

When page table entries are created, deleted or cloned, the kernel operates slightly differently when a PTE specifies PAGE_NOALLOC, and does not call kalloc or kfree on the associated address, as the address is not an index into the page_table[] array but just a physical address outside of the otherwise managed system RAM.

• Adds mmap file system ops entry into the framebuffer driver and corrects do_mmap operations on it.
• Fixes small bugs in fs/devices.c.
• Adds temporary ioctl to framebuffer driver to return the framebuffer width and height. This will be removed shortly, after adding Linux framebuffer ioctl compatibility in the next commit.
• Adds CROSS_COMPILER= and slightly modifies LIBGCC in Makefile to allow for easier compilation of Fiwix on macOS.

The changes to Microwindows for Fiwix will be posted in the Microwindows repository and a link posted here shortly.

[ghaerr]

Pushed changes, I think I got everything you asked for. Thanks.

[mikaku]

Looks good to me.

[mikaku]

Thank you very much.

[ghaerr]

Thank you! I am enjoying working with your protected mode kernel and refreshing myself on the inner details of setting up and using paging, mmap and page faults.

I'll push the changes to Microwindows and post the commit here, which should compile up and run directly on Fiwix, so that you can see it running for yourself.

[ghaerr]

On another small note, I notice that include/fiwix/version.h is always rebuilt, which then interferes with the diff produced for commits... have you considered removing that file from the repository, since it's always built anyways?

[mikaku]

I'll push the changes to Microwindows and post the commit here, which should compile up and run directly on Fiwix, so that you can see it running for yourself.

Nice! :-)

I'm thinking the best way for you to have a the latest changes I make in the Newlib C to be able to test your portings based on the new changes in the kernel. Perhaps pushing all the files into the current FiwixOS repository would be enough. We'll see.

Meanwhile, I'm replying your questions in #78.

On another small note, I notice that include/fiwix/version.h is always rebuilt, which then interferes with the diff produced for commits... have you considered removing that file from the repository, since it's always built anyways?

Well, the include/fiwix/version.h file is used in different parts of the Fiwix kernel, we cannot omit it. Also, it marks the time and date of every new release of the kernel. I believe we cannot remove it from the repository. Perhaps you could add it in your .gitignore file.

BTW, if you use IRC on your project ELKS, the Fiwix kernel has the channel #fiwix on Libera.Chat. Consider join there if you want to talk with me more directly (my TZ is CET).

[mikaku]

By the way, now I started a new complete rebuild of all packages of the FiwixOS. This is something I always do after applying specific changes in the kernel. Is a way to make sure nothing has been broken, you know.

I'll let you know if I encounter any issues.

[ghaerr]

I'm thinking the best way for you to have a the latest changes I make in the Newlib C to be able to test your portings based on the new changes in the kernel. Perhaps pushing all the files into the current FiwixOS repository would be enough. We'll see.

I will let you know - I am currently running with the FiwixOS 3.3 distribution, and now that we have the kernel changes, just tried recompiling Microwindows using named sockets - and the compilation failed not finding #include <sys/sockets.h>. So I'll push the non-client/server version of Microwindows now and then worry about C library sockets support when there is either an updated distribution ATA HD .raw file, or I'm able to actually create an ISO from macOS from the Fiwix repo.

I still think it would be a good idea to have the source for the C library etc available on a repo, thank you!

[mikaku]

I still think it would be a good idea to have the source for the C library etc available on a repo, thank you!

Yes, of course. Give me some time to upload all the files into the FiwixOS repository.

[mikaku]

I think firmly that we need to talk on IRC. We'll have a more fluid conversation

[mikaku]

I still think it would be a good idea to have the source for the C library etc available on a repo, thank you!

In the meantime, install into your FiwixOS 3.3 the latest Newlib C which includes the UNIX socket functions and header files:

newlib_4.2.0_i386.ipk.gz

# gzip -d newlib_4.2.0_i386.ipk.gz
# opkg install --force-reinstall --force-overwrite newlib_4.2.0_i386.ipk

Then restart your system. I think that this should work without having to install the whole GNU Toolchain.

[ghaerr]

I installed Newlib 4.2.0 as directed above, no problems with the installation, thank you.

After that, I changed Microwindow's src/config file to "LINK_APP_INTO_SERVER=N", to test named pipes. It built fine, but Nano-X doesn't run client/server and hangs. I spent a while debugging it, ultimately there's either something wrong with the returned file descriptors from select, or reading from the named pipe (/tmp/.nano-X) may be hanging. It seems to be returning read available on the named PIPE when there aren't any connections waiting, that is, when running bin/nano-X alone with no clients.

I wasn't able to figure it out and will have to come back to it later.

Meanwhile, I pushed the working (non-client/server configured) port of Microwindows to HEAD. You should be able to follow these instructions to see it work on your system:

git clone https://github.com/ghaerr/microwindows.git
cd microwindows/src
make # 'config' is the configuration file, preset for Fiwix for the time being
bin/mwdemo or
bin/tuxchess
(or other programs in bin/*)
^C to exit

After ^C exit, you'll see the framebuffer console all messed up. I have a fix for that we can talk about.

As a reminder, use the following for attaching an emulated MS mouse to QEMU:

qemu-system-i386 \
    -kernel fiwix -append "ro root=/dev/hda2 bga=1024x768x32" \
    -drive file=FiwixOS-3.3-i386.raw,format=raw,cache=writeback,index=0 \
    -drive file=/Users/greg/net/fwsrc3.img,format=raw,cache=writeback,index=1 \
    -chardev msmouse,id=chardev1 -device isa-serial,chardev=chardev1,id=serial1 \
    -net none \
    -boot d \
    -machine pc \
    -cpu 486 -m 4G

Here's bin/tuxchess:

Try it out and let me know it works on your system, thank you!

[mikaku]

It built fine, but Nano-X doesn't run client/server and hangs. I spent a while debugging it, ultimately there's either something wrong with the returned file descriptors from select, or reading from the named pipe (/tmp/.nano-X) may be hanging. It seems to be returning read available on the named PIPE when there aren't any connections waiting, that is, when running bin/nano-X alone with no clients.

Yes, this is a known issue as stated in the web site. I've been trying to debug it why is happening but I failed. I tried different FIFO and pipe examples with clients and servers and all worked fine. It seems that there is a specific case where it hangs, but I cannot found it.

When it happens I use a simple work-around. I just login to a different tty and it resumes the execution instantaneously. Also, it looks like keeping the tty 'busy' also helps to avoid these hangs (e.g: watch -n 1 cat /proc/meminfo). I'm not sure if the problem is in the select() implementation or in the pipe side. But it's indeed annoying when it happens.

I wasn't able to figure it out and will have to come back to it later.

Take it easy, this is a very long standing bug.

Meanwhile, I pushed the working (non-client/server configured) port of Microwindows to HEAD. You should be able to follow these instructions to see it work on your system:

Try it out and let me know it works on your system, thank you!

Yes, it worked!. Amazing to see all these graphics on Fiwix!, I love it! :-) And the mouse trick also worked!

-m 4G

Heh, you definitely don't need 4GB to boot FiwixOS ;-)

[ghaerr]

I tried different FIFO and pipe examples with clients and servers and all worked fine. It seems that there is a specific case where it hangs, but I cannot found it.

Nano-X uses a simple scheme: hang in select for read ready on the mouse, kbd and socket (named PIPE for incoming connections) file descriptors, followed by if statements for each marked readable file descriptor. I debugged Nano-X running alone and found that the socket descriptor was set as readable after the first select, which is incorrect, as there were no clients running. Then I think nano-X hung on the the attempted socket read.

So I would think the problem is in select, returning bad readable-ready file descriptor. When I commented out the handling of the incorrect socket read, the nano-X server produced a graphics screen and mouse cursor, which worked.

This should be fairly easily duplicated, do you have a test suite for anything like this? I could help debug it.

I just login to a different tty and it resumes the execution instantaneously.

I can't switch consoles when Nano-X is running, as the screen is in graphics mode. Unfortunately its a bit of a pain to undo the graphics portion and still get to the point of select/socket. I suppose writing a simple test routine for this would be the easiest way to debug select for this?

Also, it looks like keeping the tty 'busy' also helps to avoid these hangs

Interestingly, ELKS had a similar issue with the PTY code hanging that also unblocked when another TTY was accessed. But I think the socket problem here is firmly in select because of the incorrect return fd_set value with nothing connected to the socket. I am not convinced this is the same problem as the long-standing hang, unless that problem is also in select.

Yes, it worked!. Amazing to see all these graphics on Fiwix!, I love it! :-)

Nice!! Thanks for testing. If you can add FreeType v2.10.4 to your next ISO build I'll enable that in Microwindows and we'll have truetype font display, which is pretty nice.

Heh, you definitely don't need 4GB to boot FiwixOS ;-)

LOL, that's a remnant from testing Fiwix physical memory use in thinking about the large initrd problem :)

[ghaerr]

Looking at the UNIX socket select code:

int unix_select(struct socket *s, int flag)
{
    struct unix_info *u, *up;

    if(!(s->flags & SO_ACCEPTCONN)) {
        return 0;
    }

    u = &s->u.unix;
    up = s->u.unix.peer;

    switch(flag) {
        case SEL_R:
            if(u->size || s->state != SS_CONNECTED) { // <--- IS THIS CORRECT?
                return 1;
            }
            break;
        case SEL_W:
            if(s->state != SS_CONNECTED) {
                return 1;
            }
            if(up->size < PIPE_BUF) {
                return 1;
            }
            break;
    }
    return 0;
}

Should unix_select return 1 on SEL_R when the socket is not connected? That would answer why nano-X is hanging with no client... ? I'm not yet familiar with all the states in the UNIX socket state transitions but this seems like it could be an issue.

[mikaku]

Should unix_select return 1 on SEL_R when the socket is not connected? That would answer why nano-X is hanging with no client... ? I'm not yet familiar with all the states in the UNIX socket state transitions but this seems like it could be an issue.

The select() with UNIX socket has not been fully tested, sorry. In my previous post I was talking about FIFOs and pipe's problem with select(). It is well possible that select() in UNIX socket could be buggy.

[mikaku]

This should be fairly easily duplicated, do you have a test suite for anything like this? I could help debug it.

Nothing special. I collected some select() examples here and there to test FIFOs and pipes, but I've never tested well the UNIX socket. This implementation is really new and it passed all my tests using read(), write(), recv(), send(), recvfrom() and sendto() , except select() which I haven't had time to do it. I simply postponed for later. Sorry.

UNIX socket implementation is pretty new, just a pair of months. I can even imagine that someone would need that feature so fast!. LOL

If you can add FreeType v2.10.4 to your next ISO build I'll enable that in Microwindows and we'll have truetype font display, which is pretty nice.

Sure, as long as it can be build. Just added it my ToDo list.

[mikaku]

I saw these questions unanswered here, but I think this was already solved by you in #82. So, I think that we can close this discussion. Thanks.