External disk support

[nuta]

Currently, Kerla uses initramfs embedded into the kernel as its root file system.

The major disadvantage is that the all file contents will be loaded into the memory. That is, if you want to run a 100MiB Docker image, 100MiB needs to be reserved for initramfs!

This issue is aims to allow running large Docker images by:

• Implement virtio-blk (or virtio-scsi) block device.
• Implement a file system (e.g. ext4).

[dust1]

I am more animalistic about implementing a file system. But I can only start with some simple tasks, Can we first determine the specific plan and then list the task list?

[nuta]

Great! If I work on this topic I will:

1. Survey how existing OS kernels implement file systems, block I/O subsystem, and how they cache file/disk contents.
2. Write virtio-blk device driver.
3. Write block IO and caching subsystem.
4. Look for file system which is easy to implement: ext2 is a promising candidate. Newer feature-rich filesystems like btrfs and zfs would be too complicated for the first step.
5. Add ext2 (or something) output support in docker2initramfs.py. You might want to use virt-make-fs to do this task.
6. Implement filesystem reader/writer no_std library: it will depends on a simple Disk trait like:

trait Disk {
    fn read_block(&mut self, lba: usize, buf: &mut [u8]) -> Result<usize /* # of bytes actually read */, DiskError>;
    fn write_block(&mut self, lba: usize, buf: &[u8]) -> Result<usize /* # of bytes actually written */, DiskError>;
}

8. Integrate the library into Kerla.
9. Attach the disk to QEMU (-device virtio-blk-pci,...).

First, I recommend you to learn how file I/O works in major operating systems like Linux and FreeBSD. I think this book is a good start point to learn the internals.

[dust1]

A great plan, thanks, I will first understand the implementation of the linux file system by reading this book. Are we sure to implement the ex2 file system?

[nuta]

Are we sure to implement the ex2 file system?

Yes. I agree ext2 is the best for the first file system implementation because ext2 and its descendants ext3/ext4 are popular and well-known: there're good articles, sample implementations, and tools.

[nuta]

The Block I/O subsystem is a bit off-topic so the first goal would be writing ext2 reader/writer library.

[michalfita]

Can I suggest starting from simpler filesystem, potentially some FAT derivative might be easier for start.

[nuta]

Of course! Other file system implementations are also welcome. That said, I think FAT is a good for us because:

• It's not designed for UNIX-like operating systems: it lacks attributes such as permissions.
• Part of the spec is protected by Microsoft's patent.

Perhaps minixfs could be interesting.

[dust1]

That might be interesting, maybe we should have a VFS?

[nuta]

Kerla already has a sort of file system abstraction 😃 You can mount an arbitrary file system here: https://github.com/nuta/kerla/blob/00f9c969e8a45ff2cac476fd390ec340863391e5/kernel/fs/mount.rs#L42

[dust1]

cool,It seems that I did not understand kerla carefully,That's not great

[nuta]

You don't need to say that! Please feel free to ask me anything even if the question is trivial. It also helps me to write docs 👍

[michalfita]

That said, I think FAT is a good for us because:

• It's not designed for UNIX-like operating systems: it lacks attributes such as permissions.
• Part of the spec is protected by Microsoft's patent.

While it's not designed for UNIX, it's very simple to implement - and I suggest that as starting point and test thing. Keep in mind to use SD Cards you pretty often need FAT and/or exFAT support as this filesystem in part of the SD Card specification. Moreover, (FAT is already implemented in Rust](https://github.com/rafalh/rust-fatfs). Metadata can be mapped via hidden files.

Most FAT patents expired already, potentially there are some for exFAT.

[nuta]

Makes sense. We need to support FAT anyway in the future to support some boards like Raspberry Pi.

[pothos]

I see https://github.com/pi-pi3/ext2-rs which could be a starting point but not sure how usable it is for this purpose

[Lurk]

I see https://github.com/pi-pi3/ext2-rs

There is an updated fork https://github.com/rcore-os/ext2-rs which is in better shape than original

[michalfita]

@pi-pi3 the original author of that crate is my countryman, but it seems he abandoned GitHub after Microsoft bought it. His profile on GitLab is private, so we cannot count on his contribution.

I wonder if @jiegec would like to contribute.