search

trillek-team / tec

The Trillek Engine
http://trillek.space
GNU Lesser General Public License v3.0
138 stars 21 forks source link

Virtual file system #148

Open milesrout opened 4 years ago

milesrout commented 4 years ago

Trillek needs a virtual file system (VFS). This is an abstraction layer over the real disk. By adopting a layered bind mount approach it would be very flexible allowing us to load assets from zip/tgz archives transparently, sandbox plugins' disk access to a directory and its children, load resources over the network transparently, load new versions of resources from different places hiding the previous version at runtime, etc.

With a file extension registration system, different systems will be able to register file extension handlers to load resources into the VFS in their own way. For example, client/resources/md5mesh.cpp can register a handler for the ".md5mesh" extension, and then whenever a file with extension ".md5mesh" is loaded, the handler will read the file, parse it, and load an MD5Mesh object into the VFS at that path. Or for example, a handler for the ".zip" extension will be able to load the zip archive and present it transparently to the rest of the system as a directory.

When directories are mounted, they would create virtual directories, the contents of which are lazily mounted to the filesystem as they are loaded. So if I mount (real filesystem path) ./assets.zip to (virtual filesystem path) /, then load /models/bob/bob.md5mesh, /models/bob/bob.md5mesh will be loaded from the archive and passed as a file stream (backed by a custom ZIP archive streambuf) to the ".md5mesh" file extension handler, which basically shouldn't need to know whether it's dealing with a real physical file or not. It just uses an iostream.

// archive.cpp
vfs.RegisterExtension(".zip", [] (VFS::Path mountpoint, std::iostream fileStream, VFS::Handle h) {
    h.MountDirectory(...);
});
// md5mesh.cpp
vfs.RegisterExtension(".md5mesh", [] (VFS::Path mountpoint, std::iostream fileStream, VFS::Handle h) {
   auto mesh = MD5Mesh::Create(...);
   h.MountFile(mesh);
});
// main.cpp
vfs.Mount("/", FilePath{"./assets.zip"});
vfs.Mount("/logs", FilePath::GetLogDirectory());
std::shared_ptr<MD5Mesh> mesh = vfs.Load("/models/bob/bob.md5mesh");

@adam4813 I would like to use std::filesystem, a C++17 feature, to replace the current FilePath which is pretty limited. Are you happy with that? Are we okay with requiring C++17?

adam4813 commented 4 years ago

I think i already bumped it to 17 iirc in the cmake file.

Also said file loaders are already registered for some extensions, so half the work is done.

On Thu, Nov 21, 2019, 7:57 PM Miles Rout notifications@github.com wrote:

Trillek needs a virtual file system (VFS). This is an abstraction layer over the real disk. By adopting a layered bind mount approach it would be very flexible allowing us to load assets from zip/tgz archives transparently, sandbox plugins' disk access to a directory and its children, load resources over the network transparently, load new versions of resources from different places hiding the previous version at runtime, etc.

With a file extension registration system, different systems will be able to register file extension handlers to load resources into the VFS in their own way. For example, client/resources/md5mesh.cpp can register a handler for the ".md5mesh" extension, and then whenever a file with extension ".md5mesh" is loaded, the handler will read the file, parse it, and load an MD5Mesh object into the VFS at that path. Or for example, a handler for the ".zip" extension will be able to load the zip archive and present it transparently to the rest of the system as a directory.

When directories are mounted, they would create virtual directories, the contents of which are lazily mounted to the filesystem as they are loaded. So if I mount (real filesystem path) ./assets.zip to (virtual filesystem path) /, then load /models/bob/bob.md5mesh, /models/bob/bob.md5mesh will be loaded from the archive and passed as a file stream (backed by a custom ZIP archive streambuf) to the ".md5mesh" file extension handler, which basically shouldn't need to know whether it's dealing with a real physical file or not. It just uses an iostream.

// archive.cpp vfs.RegisterExtension(".zip", [] (VFS::Path mountpoint, FilePath realpath, VFS::Handle h) { h.MountDirectory(...); }); // md5mesh.cpp vfs.RegisterExtension(".md5mesh", [] (VFS::Path mountpoint, / type to be determined / fileStream, VFS::Handle h) { auto mesh = MD5Mesh::Create(...); h.MountFile(mesh); }); // main.cpp vfs.Mount("/", FilePath{"./assets.zip"}); vfs.Mount("/logs", FilePath::GetLogDirectory()); std::shared_ptr mesh = vfs.Load("/models/bob/bob.md5mesh");

@adam4813 https://github.com/adam4813 I would like to use std::filesystem, a C++17 feature, to replace the current FilePath which is pretty limited. Are you happy with that? Are we okay with requiring C++17?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/trillek-team/tec/issues/148?email_source=notifications&email_token=AAW7KTI4Q2L5544TNZ6W4NLQU437TA5CNFSM4JQKSWJKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H3IPQSA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAW7KTJ3XDCIGPDIZAZ3XLLQU437TANCNFSM4JQKSWJA .

milesrout commented 4 years ago

Yeah, if you have a look at my filesystem branch, I've started work on this.

adam4813 commented 4 years ago

I will when i get a chance.

On Thu, Nov 21, 2019, 8:21 PM Miles Rout notifications@github.com wrote:

Yeah, if you have a look at my filesystem branch, I've started work on this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/trillek-team/tec/issues/148?email_source=notifications&email_token=AAW7KTPJFIHEBSGWSR5IMGLQU462PA5CNFSM4JQKSWJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE4JX3I#issuecomment-557358061, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAW7KTPLFLOPXGUJPZO6GO3QU462PANCNFSM4JQKSWJA .

milesrout commented 4 years ago

I don't think I pushed my latest changes to that branch, but essentially it makes it work as described above. Currently it's not quite architected right: it gives extension handlers the raw file path, assuming every resource they can access is a raw file on disk, which according to the above design wouldn't be true. Still, it basically works as of my latest unpushed changes. Just wanted some feedback before I put a whole lot more work into it.

adam4813 commented 4 years ago

Though it will be nice to have, i am not sure how important it is to the 0.11 release, so maybe dont invest a lot of time right now?

On Thu, Nov 21, 2019, 8:34 PM Miles Rout notifications@github.com wrote:

I don't think I pushed my latest changes to that branch, but essentially it makes it work like that. Currently it's not quite architected right: it gives extension handlers the raw file path, assuming every resource they can access is a raw file on disk, which according to the above design wouldn't be true. Still, it basically works as of my latest unpushed changes. Just wanted some feedback before I put a whole lot more work into it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/trillek-team/tec/issues/148?email_source=notifications&email_token=AAW7KTOIDCVZZ23YMEX4UR3QU5ALDA5CNFSM4JQKSWJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE4KOOI#issuecomment-557360953, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAW7KTN46PQLENPSMF5WRYTQU5ALDANCNFSM4JQKSWJA .