Could Aquila fit on a 1.44MB or 2.88MB floppy?

[informer2016]

Good day, @mohamed-anwar and @nagyation . Please tell, could Aquila fit on a 1.44MB or 2.88MB floppy? Even today the floppies are still being used, for example - as virtual floppies inside the coreboot open source BIOS. Just imagine: your wonderful OS could be a part of someone's BIOS build! _(for coreboot supported motherboard, maybe you have or could get one - see https://www.coreboot.org/Supported_Motherboards )_

If you already have a coreboot-supported motherboard, or a real chance to get one, - wouldn't it be cool to be able to launch your own OS straight from the BIOS chip? ;) With one simple command its possible to add any floppy to coreboot BIOS build - and then you see it as a boot entry! Multiple floppies could be added this way (as long as you have enough space left inside the BIOS flash chip, luckily LZMA compression could be used for the stored floppies to reduce their occupied size)

So far I see ~10MB images at your Aquila releases, but could their contains be compressed for a floppy, maybe not 1.44MB but an 2.88MB one?

[thethumbler]

Hello @informer2016 Aquila is actually very tiny, it can fit on 1.44 MB floppy, but some arrangements have to be done.

• Most of the size in the kernel binary comes from ".boot" section because the linker squashes everything into a single section (including what should go into .bss section), so most of the kernel binary is just filled with zeros, optimizing the link.ld script would solve this problem.
• Most of the size in the ramdisk image is due to static linking, if we strip the binaries to the absolute minimum (or better yet, support dynamic linking) we can achieve much smaller sizes.

I'll be working on these issues, release and notify you if I could fit it on the 1.44 MB image :)

[informer2016]

@mohamed-anwar That's very nice to hear! I will be checking Aquila from time to time, as soon as there's a floppy I could test Aquila as a part of my BIOS image :wink: And if you have any coreboot-related questions I will be happy to help you :nerdface: I'm using AMD A10-5750M quadcore Lenovo G505S laptop with 16GB RAM and as you see there are some of quite modern laptops/PCs among the coreboot-supported list, even servers - e.g. ASUS KGPE-D16 with two AMD Opterons. Personally I prefer AMD-based ones because they don't have Intel ME hardware backdoor inside and also - while being powerful - are old enough not to have AMD PSP backdoor which (compared to ME) got added much later, so the latest "AMD-without-PSP" is significantly more powerful than "Intel-without-ME" P.S. By itself Lenovo G505S is really affordable, possible to get for ~$100-$150 in good condition (although the RAM is expensive nowadays and 16GB of DDR3 cost almost the same as this laptop :stuck_out_tongue_winkingeye: ) Just need to make sure that it has A10-5750M CPU : there were some versions with lower performance A8 or A6, and although the CPU is socket'ed and not soldered, it is about $50 to get it separately

[thethumbler]

Hey @informer2016 you can now fetch a 1.44 MB image of Aquila from http://aquilaos.com/v86/aquila.img, this is a HD image with grub-legacy and a FAT12 partition, the kernel includes all features but the ramdisk has only aqbox and init (only UART console - no VGA). I didn't push the modifications to the repo yet, so you could play with the binaries for now until I find a way to add these hacks to the standard build. As of coreboot, I'm a strong advocate of it and went through the entire code for the love of it. Unfortunately though, I don't own a coreboot supported board, I do intend to get one but not in the mean time. You could be a tester for AquilaOS as a payload to coreboot (and maybe, with some work, run Aquila on coreboot without grub). Keep me posted.

[informer2016]

Hi again @mohamed-anwar , I just checked your floppy but sadly can't boot it even at QEMU... I'm using this command line: qemu-system-x86_64 -L . -m 256 -fda ./aquila.img -usbdevice tablet -vga vmware -localtime The only info printed by Aquila is GRUB Hard Disk Error Maybe we could borrow the successful floppy approach from some other hobby OS, e.g. Fiwix ? https://www.fiwix.org/downloads.html Although Fiwix gets a kernel panic after it boots from GRUB, its' booting/GRUB part works great and can be used. Once we get Aquila floppy working with QEMU / floppy drive, I can test it as QEMU / coreboot+SeaBIOS (have a special build for QEMU) and finally as a part of coreboot+SeaBIOS on real hardware :wink:

[thethumbler]

@informer2016 sorry I didn't make it clear, that image was an HD image (with MBR) so it wouldn't work as a floppy image in qemu (with GrUB). I have created a new image for you http://aquilaos.com/v86/floppy.img that can work with -fda in qemu, it has a single ext2 partition and supports VGA (no wallpaper for size). We use a patched version of GrUB legacy to support VBE for Vesa driver to work. I should add a floppy disk controller so we would be able to use that ext2 partition in the floppy as the root filesystem for aquila, will notify you if I manage to.

P.S. Fiwix is a great OS, I toyed with it for some time and really appreciate the hard work committed by @mikaku. Aquila is in its days of infancy compared to Fiwix. I'd recommend reporting that kernel panic you faced to mikaku so the issue could be investigated further.

[informer2016]

@mohamed-anwar Thank you very much! I just tested your floppy and discovered a weird problem: I boot it at QEMU with qemu-system-x86_64 -L . -m 256 -fda ./floppy.img -usbdevice tablet -vga vmware -localtime command, or even with an extra -enable-kvm flag for acceleration, it boots fine but really slow and also works very slow, like some interrupt is constantly irritating the kernel, but I don't know how to check it because there's no /proc/interrupts file so I can't do something like cat /proc/interrupts to view how many interrupts are there. At VirtualBox the same floppy is quite fast. That's the first time I see such a problem, and don't know how to debug it but hopefully it's reproduceable. Could you please try running it in QEMU as well?

P.S. I also discovered a bug , will post it to a different issue (and already sent the Fiwix bug reports)

[thethumbler]

Hello @informer2016. The performance issue is actually due to the use of fbterm, we don't use any acceleration to draw the screen and rely on libvterm to provide us with the update regions. fbterm needs a serious amount of work to meet performance needs without kvm acceleration (you will notice it's slow when an entire screen update is needed, like scrolling).

As of incomplete terminal output, that also is due to fbterm, it's probably due to some weird bug in libvterm or inadequate handling of special cases. If you use the serial console instead of fbterm you won't face any of these problems, I intended to rewrite fbterm from scratch and drop dependency on libvterm completely, but I don't have much time at my disposal.

There is another point to the performance issue, I deliberately set the 8259 PIT with a lower frequency so it would be easier for me to debug, I forgot to remove that line from code (in arch/x86/chipset/misc/pit.c), but that shouldn't be very noticeable.

As for interrupts, all interrupts are masked early on startup, and only specific interrupts are enabled by device drivers, so no other interrupts reach the kernel anyway.

P.S. VirtualBox uses some acceleration by default, Qemu should be fine with -enable-kvm, my best guess is that KVM failed to initialize properly or some other thing related to Qemu/KVM, it has nothing to do with Aquila itself.

[informer2016]

@mohamed-anwar Recently I tested QEMU's KVM with another OS and it worked fine there, but in case of Aquila I don't see any speed difference between KVM enabled vs disabled. Please could you test this Aquila floppy in QEMU with this command line? qemu-system-x86_64 -L . -m 256 -fda ./floppy.img -usbdevice tablet -vga vmware -localtime -enable-kvm

[thethumbler]

@informer2016 I tried the command with enable-kvm and the performance is highly comparable to non-KVM. I don't know what's the cause of your problem, I use QEMU emulator version 2.11.2(qemu-2.11.2-2.fc28) bundled with Fedora 28. I'm dedicating tomorrow to Aquila and hopefully would be able to make a lot of changes, I'll try to focus on fbterm.

For performance issue, further debugging would be helpful. Aquila uses UART for debugging output, please use -serial stdio in Qemu and check what is causing it to be that slow.

[informer2016]

@mohamed-anwar Here is my QEMU boot log with KVM enabled: aq.txt, running on Linux Mint 19 with QEMU 2.11.1 (almost the same version as yours). Haven't seen anything suspicious except the 8254 PIT: Setting period to 3 ns but maybe this is OK

[thethumbler]

@informer2016 could you please try this image? I have fixed the incomplete output issue (through a hack, but should work for now) and tweaked the performance a bit. This is a special image that runs login on both serial and fbterm (could be configured through grub), could you please test and compare the performance of both?

[informer2016]

@mohamed-anwar I just tested, sadly this QEMU performance problem is still there; maybe you also have it but your PC is much more powerful so you don't notice? Because my QEMU host is another laptop which is like 10 years old (but the other floppy-based OS are working fast there) Compared to the old log, the beginning is the same but this ending part is different:

ata: Scanning PCI bus for IDE controller
ata: Found IDE controller (Bus: 0, Dev: 1, Func: 1)
ata: IDE controller primary channel: (Base: 0x1F0, Control: 0x3F6)
ata: IDE controller secondary channel: (Base: 0x170, Control: 0x376)
ata: drive error: Command aborted
ata2: Signature: 0x0
ata2: Capabilities: 0x0
ata2: Command Sets: 0x0
ata2: Max LBA: 0x0
ata2: Model: 
ata2: Initializing ATA device (PIO)
kdev: Registered blkdev 3: ata
fdc: Initializing
fdc: Reset controller
kdev: Registered blkdev 2: fdc
kernel: Loading ramdisk
kernel: Loading init process
AquilaOS version v0.0.1a
Username:

[thethumbler]

Oh, enter the username and interact with it, that's a UART console. I'm trying to see if fbterm is definitely the problem (the other OSes either use VGA Text Console or a custom terminal with some sort of acceleration).

[informer2016]

@mohamed-anwar the interaction through QEMU's UART is also very slow... Some other news: today I made a new coreboot+SeaBIOS build for G505S and put your Aquila floppy (previous version) inside - so you could be proud your OS got into BIOS :wink: Already flashed, and here is a boot log retyped from a screen:

  Booting 'AquilaOS'

kernel /boot/kernel.elf
   [Multiboot-elf, 0x100000:0x10c:0x102ef4>, <0x203000:0x17ffc:0x0>, <0x21b000
:0x2458:0x307d68>, shtab=0x5261b8, entry=0x100030]
module /boot/initrd.img
   [Multiboot-module @ 0x52c000, 0x78600 bytes]
vbeset 0x142
 Mode 0x142 is not supported.

_ <--- blinking

What is this mode 0x142 ?

[thethumbler]

That's VBE (Vesa BIOS Extensions) video mode, 0x142 is 640x480x32. You can choose a different mode, just hit c when GrUB menu shows up and enter vbeprobe, check which graphics mode you want, record the value then reboot and hit e to edit the boot entry, choose the "vbeset" line, hit e again to edit, press Enter when you finish editing the b to boot.

You could also do it without rebooting if you'd like by entering configfile /boot/grub/menu.lst in the command line.

P.S. I still don't know what's causing the performance issue with UART, I'll try to add command line handling to the kernel and add an argument for manually setting 8254 PIT divisor value (that's the timer used for scheduling)

[thethumbler]

@informer2016 Please try this updated image, in GrUB menu, edit the kernel entry (as above) and edit the 8254.div value. This is the value used for the divisor of 8254 PIT (base freq. 1.193182 MHz), you can use any value from 1 to 65535. The lower the value the higher the frequency (and hence better responsiveness, but too low will overwhelm the scheduler and the system will seem stuck).

Please check if any values improve the performance of Aquila and report back.

[informer2016]

@mohamed-anwar by the way it's possible to return from command line mode (where I entered vbeprobe) back to menu just by pressing the Escape key. Native screen resolution of G505S is a classic 1366x768. Here is my vbeprobe output, united with the testing results:

grub> vbeprobe
 VBE version 3.0
  0x100: Packed pixel, 640x400x8

0x100 - I see the welcome screen ( Welcome to AquilaOS v0.01; the yet-in-development OS :) +newline+ AquilaOS version v0.0.1a +newline+ Username: ) but the letters are very distorted / rainbow colored and OS doesn't react to the typed letters. However it went through the "gray screen" phase preceding this screen very quickly - at QEMU it took a few minutes

  0x101: Packed pixel, 640x480x8 <--- same as 0x100
  0x103: Packed pixel, 800x600x8 <--- same as 0x100, just a bit different resolution
  0x105: Packed pixel, 1024x768x8 <--- same as 0x100, just a bit different resolution
  0x110: Direct Color, 640x480x16

0x110 - same as 0x100, but the letters aren't rainbow color - are various shades of blue/purple/green

  0x111: Direct Color, 640x480x16 <--- = 0x110
  0x113: Direct Color, 800x600x16 <--- = 0x110 but a diff res
  0x114: Direct Color, 800x600x16 <--- = 0x113 
  0x116: Direct Color, 1024x768x16 <--- = 0x110 but a diff res
  0x117: Direct Color, 1024x768x16 <--- = 0x110 but a diff res
  0x121: Direct Color, 640x480x32

0x121 - normal white color, but also stuck at the same Welcome screen. As with the cases above, at "Username: " this is not blinking and can't type the letters

  0x122: Direct Color, 800x600x32 <--- = 0x121 but a diff res
  0x123: Direct Color, 1024x768x32 <--- = 0x121 but a diff res
  0x1d1: Packed pixel, 1366x768x8 <--- = 0x100 but a diff res, same rainbow colors
  0x1d2: Direct Color, 1366x768x16 <--- 0x110 but a diff res, same blue/purple/green colors
  0x1d4: Direct Color, 1366x768x32

0x1d4 - normal white color, just like 0x121 but a diff res

To summarize, only these modes are normal white letter colors:

  0x121: Direct Color, 640x480x32
  0x122: Direct Color, 800x600x32
  0x123: Direct Color, 1024x768x32
  0x1d4: Direct Color, 1366x768x32

But even at this modes Aquila is stuck at Welcome screen and I don't know how to get more debug info?

[informer2016]

@mohamed-anwar Very good ! :+1: It is very fast speed with high responsiveness at 8254.div=10000 :smiley: I'm curious if this value is acceptable for you to also have a great speed/responsiveness. Now the only remaining problem is coreboot one - but it could take me some time to put the updated floppy inside BIOS and the cause of this "freezing" could be different... Do you know how to collect some debug info there?

[thethumbler]

@informer2016 I play with different values for the divisor since Aquila's scheduler is very simple for now (just round-robin with no priorities) so all threads get the same time slice. The value of 1000 could be good for interactive applications but bad for other long running jobs (like daemons, which are not publicly available yet). I'm setting the default to 1000 for now.

I can think of a few things for freezing at this point, either keyboard driver failed to load (UART doesn't use PS/2 keyboard), fbterm crashes - unlikely (fbterm doesn't produce any logs yet, it will be daemonized at some point and the logs would be available), or, if it it doesn't happen all the time then it could be a simple problem with IRQs.

For logs, unfortunately, I disable logging to UART after init is loaded (so we could use UART as a tty), changing that requires recompiling the kernel, I'll add another kernel parameter like debug or something that controls this behavior. I'm overly tired right now so I'll get some sleep first, then we can work on some other things (like debugging and maybe VGA text console support). I'll keep you posted.

[informer2016]

@mohamed-anwar It seems I'm starting to have these speed problems somewhere between 2000 and 3000, haven't found the exact boundary, but could confirm that 3000 works fine - while for 1000 its really slow like the previous cases. I don't know if it'd be a good idea to set it as default 3000 and also leave it a customizable parameter just in case... about Aquila at G505S : I'm unsure if it'd be possible to use UART on this laptop (the only way is through UART-over-USB and while it works for coreboot+SeaBIOS I don't know if it'd work for Aquila) , so it would help if such info would be printed directly on the screen but is it possible without VESA initialization, or at least shortly after it has happened?

[thethumbler]

Hello @informer2016. Sorry for taking so long to reply, I have pushed an updated floppy disk image that has a framebuffer based console for debugging output. The codebase is being considerably refactored (git blame me when I push the big updates :"D). You can set the divisor value as usual, you can choose the earlycon with earlycon=fb or earlycon=uart (if you pass it multiple times, only the first time will be recognized), by default it's set to fb.

This image is tested on real hardware (albeit using the ISO image) and it's known to work smoothly. Let me know if you have updates.

[informer2016]

Good day @mohamed-anwar ! I just tried to run your latest floppy but using only the following modes:

To summarize, only these modes are normal white letter colors:
  0x121: Direct Color, 640x480x32
  0x122: Direct Color, 800x600x32
  0x123: Direct Color, 1024x768x32
  0x1d4: Direct Color, 1366x768x32

123 and 1d4 cannot boot with a similar message:

x86: Initializing PCI
8259 PIC: Initializing [Master: 0x20 (pio), Salve: 0xA0 (pio)]
i8042: Controller not found
8254 PIT: Initializing [0x40 (pio)]
kdev: Initializing
kernel: Loading builin modules
kdev: Registered chrdev 11: kbddev <--- same line for
^^^ 29:fbdev, 4:sttydev, 5:ttydev, 2:ptmdev, 136-143:ptsdev
uart: Registered uart 0: 8250
kdev: Registered chrdev 10: mousedev
kdev: Registered chrdev 1: memdev
kdev: Registered blkdev 1: ramdisk
kernel: Loading ramdisk
kernel: Loading init process
8254 PIT: Setting period to 838 ns (div = 1)
arch_handle_signal(sig=11)
KERNEL PANIC:
proc.c [135] proc_kill: init killed
Bailing out. You are on your own. Good luck.

121 and 122 could boot further, after 8254 PIT: Setting period to 838 ns (div = 1) it becomes:

init: started
init: executing /etc/rc.init script
mounting devfs on /dev
populating static devices in /dev
mounting devpts on /dev/pts
init: starting fbterm
fbterm: starting...
fbterm: initializing framebuffer /dev/fb0
fbterm: /dev/fb0: xres = 800, yres = 600, depth = 32
AquilaOS version v0.0.1a
login: S version v0.0.1a
password:
... a few text/graphical glitches for 0x122 which happen here,
but much more glitches for 0x121 which start happening after "init: starting fbterm" ...
Login failed

Then Aquila starts a new screen where is only

AquilaOS version v0.0.1a
login: _ <--- not blinking _

and I couldn't type anything there. Tried playing with various values of 8254.div, but still can't write

[thethumbler]

@informer2016

• fbterm (and fb earlycon) can handle only 32-bit depth right now, that's a basic limitation but I'm working on a new terminal emulator right now and don't wish to pursue fbterm further.
• You can't start both a terminal over serial (i.e. passing --serial to ramdisk) and use earlycon=ttyS0 at the same time, it will mess things up.
• The cursor actually doesn't blink in fbterm, the actual problem is due to i8042: Controller not found, which means it couldn't find the PS/2 controller and keyboard wouldn't work.. are you testing on real hardware or Qemu?

[informer2016]

@mohamed-anwar These test results are for real hardware; aside from changing vbeset from 142 to other values (and playing with 8254.div for a while) I haven't changed any of those default parameters:

kernel /boot/kernel.elf
module /boot/initrd.img --serial --fbterm
vbeset 0xVALUE

Now I tried leaving only either --serial or --fbterm but it didn't improve the things

[thethumbler]

@informer2016 remove --serial to get rid of these test/graphical glitches. Does the hardware have an i8042 (PS/2) controller? if not, we hit a dead end here, aquila implements only keyboard over PS/2. USB keyboards (without i8042 or PS/2 emulation) don't work.

[informer2016]

@mohamed-anwar I am sure that this laptop's internal keyboard and touchpad are internal PS/2 devices + there is a PS/2 controller ( integrated to embedded controller EC KB9012 chip ) And while a touchpad didn't work at some hobby OS (e.g. Snowdrop) because of some incompatibility (but worked e.g. at KolibriOS), this laptop's keyboard is working at all the hobby OS I've tried so far

[informer2016]

@mohamed-anwar Happy coming New Year ! :christmas_tree: :santa: :gift:

[thethumbler]

@informer2016 Happy new year! I don't know if KolibriOS checks if i8042 is present or not, I have pushed a new image with i8042.nocheck parameter passed to the kernel, it will attempt to initialize the keyboard without checking, let me know if that works. --UPDATE-- I have just tried the image with Bochs, apparently i8042 doesn't get detected there too, so it's mostly a bug, I'll check the specs to see if the way used to check the controller is the proper way -- or maybe just ignore checking entirely.

[informer2016]

@mohamed-anwar I just tested your latest Aquila floppy, with the only difference in boot parameters of changed vbeset 0x142 to something else because no 0x142 mode at my PC :

To summarize, only these modes are normal white letter colors:
  0x121: Direct Color, 640x480x32
  0x122: Direct Color, 800x600x32
  0x123: Direct Color, 1024x768x32
  0x1d4: Direct Color, 1366x768x32

Congratulations! Now your latest Aquila floppy boots better at those 121 / 122 / 123 / 1d4 modes - nearly instantly for 122 / 123 / 1d4 , but much slower for 121 and with some extra graphical glitches - and after the booting completes I could finally type the letters successfully :smiley: Some notes:

1) If I press "Ctrl+C" at the initial login screen to "terminate it", it prints in the lower half of screen

arch_handle_signal(sig=2)
arch_handle_signal(sig=2)
KERNEL PANIC:
proc.c [132] proc_kill: init killed
Bailing out. You are on your own. Good luck.

and have to force shutdown with laptop's power button 2) Even the incorrect password for root user works to login to the console 3) Nice ASCII art at "Welcome to AquilaOS!" screen :smile: :eagle: 4) ls -al at / directory always successfully prints everything at once, both at real hardware and in QEMU, so I guess #24 issue could be closed 5) However, at /bin directory ls -al always prints with a glitchv:

... everything ok ...
lrwxrwxrwx 1 root root 10 Dec 31 2018 11:34 sh -> /bin/aqbox
lrwxrwxrwx 1 root root 10 Dec 31 2018 1lrwxrwxrwx 1 root root lrwxrwxrwx 1 root root 10 Dec 31 2018 11:34 env -> /bin/aqbox
lrwxrwxrwx 1 root root 10 Dec 31 2018 11:34 uname -> /bin/aqbox
... everything ok ...

Will be nice if there would be commands like reboot and shutdown, and maybe some cool demo apps

[thethumbler]

Hey @informer2016, I'm glad to hear that! It's still a long road from here for aquila, but hopefully we can make it! Here are the notes to your notes

• 1. init is very basic as of now, it doesn't handle signals or reap zombies, I'll be working on it soon.
• 2. There is no "correct" password per se, the auth syscall permits any password, I still didn't come up with a proper way to handle logging and authentication, so will leave it there until I finally settle on something.
• 5. As discussed in #24 this limitation is due to fbterm and not ls itself, I'm already working on a new terminal emulator but due to time limitations it's not functional to the point of switching yet.

P.S. I didn't actually intend to support floppy drives at all, but after reconsidering it, floppy drive support might be a nice feature, so I wrote a simple fdc driver (for Intel 82077AA) and it partially works (at least in read only mode), which would allow us to mount the root filesystem from the floppy disk and use it directly. While the mount does work (on /mnt for example), we don't have chroot support yet, and mounting on / directly is not yet fully supported.

I'll keep you posted with updates (and hopefully commit and push everything, it's been a while) if I manage to get these problems fixed and hopefully introduce some new features.

Cheers!

[informer2016]

@mohamed-anwar although to be honest I don't know if supporting the real physical floppy drives could be useful (compared to the "virtual floppy drives" of coreboot/SeaBIOS), of course that would be nice :smiley:

[informer2016]

@mohamed-anwar please let me know if there are any newer floppy releases, I love to help and will be happy to test them

[thethumbler]

Hello @informer2016 sorry for the late reply, I have been busy the past few days. There supposed to be an automated release system, but Travis is broken right now (mainly because building a toolchain takes too long). I have tried booting Aquila from an actual floppy disk and it worked fine, but the floppy drive driver didn't work, I might work on it further if I get time, but right now I'm uncertain. I will let you know if I either fix the automated build system or I release any new floppy images.