Add support for virtual filesystem in monolithic file

AmigaPorts / ACE

Amiga C Engine

Mozilla Public License 2.0

163 stars 26 forks source link

Add support for virtual filesystem in monolithic file #50

Open tehKaiN opened 6 years ago

tehKaiN commented 6 years ago

Moved from #25 since it's currently not that critical but will be for moving to ROM-based platforms or squeezing into floppy. It would be cool to have an ability to work on plain file structure of host OS or on virtual FS based on monolithic file.

[x] Determine lightweight, portable, virtual FS for such mode
[x] Implement virtual storage access functions
[x] Migrate ACE into using either disk-based files or virtual storage
[x] Ability to disable virtual FS overhead with single define during compilation
[x] Implement a tool to transform required file structure to monolithic bigfile

Before we introduce another custom file format, let's look at embedded systems for similar mature solutions. Perhaps SquashFS would do the job? Or is it too bloated?

tehKaiN commented 6 years ago

I've found romfs specs. Kinda overkill though.

tehKaiN commented 6 years ago

Nintendo uses ExeFS on their handhelds. However, it's quite barebone and it doesn't support long file names and directories.

tehKaiN commented 6 years ago

QFSGen generates files as byte arrays in C code in similar fashion to incbin and its shell/bat polyglot variant. We could have such tools a bit customized and build read-only filesystem on that.

tehKaiN commented 5 years ago

https://hackaday.com/2019/01/24/cool-tools-a-little-filesystem-that-keeps-your-bits-on-lock/

here's performance comparison with SPIFFS https://github.com/RIOT-OS/RIOT/pull/8316

tehKaiN commented 5 years ago

Also, I thought today of using TAR files since it's de facto standard for archiving lots of small files into one without compression. There are cons however:

Headers use ASCII coding, sizes are additionally coded in octal - wtf?
files are saved as multiples of 512 bytes

Also I've found today https://github.com/MarcoPon/SeqBox. Apart from being error resistant I know nothing about it, except it apparently won't work in intended manner on OFS floppy drives.

SamuraiCrow commented 5 years ago

TAR stands for Tape ARchiver. It's mainly used to get better compression under single file archivers like GZip and BZ2. If you can figure a way to index into it, it might work as intended.On Sep 26, 2019 1:53 PM, KaiN notifications@github.com wrote:Also, I thought today of using TAR files since it's de facto standard for archiving lots of small files into one without compression. There are cons however: Headers use ASCII coding, sizes are additionally coded in octal - wtf?files are saved as multiples of 512 bytes Also I've found today https://github.com/MarcoPon/SeqBox. Apart from being error resistant I know nothing about it, except it apparently won't work in intended manner on OFS floppy drives.

—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or mute the thread.

tehKaiN commented 5 years ago

yeah, and that's what I'm aiming at in this issue. To have single file holding whole file structure with as little overhead as possible, hopefully to support writing too.

Quake games were using zip files renamed to .pak. I have yet to see if this won't be a better choice, but I think this may be a bit of an overkill.