Enhancement: `vfs` is needed to uniformly manage KCL's current interaction with the file system during compilation.

[zong-zhe]

Enhancement

The KCL compiler has some technical implementation problems in the process of interacting with the file system.

• The Parse stage has a messy process of reading entry for compilation, either from the file system or from incoming code snippets, because it does not have a vfs, causing some work that requires vfs to be written directly to the parser. The resolver also relies directly on the local file system.

• Some apis that directly compile code snippets and the code snippets import each other, independent of local file system, and require a vfs to organize the associations between virtual code snippets.

[zong-zhe]

[WIP] KCL Virtual File System

Feature

1. Compatibility with Windows/Unix Path Differences: Supports using different styles of file paths.
2. File Existence on Local Paths: Files may not necessarily exist on local paths.
3. Incremental Compilation Compatibility: Supports flexible insertion of files, considering incremental compilation.
4. Ease of Extending Different File Systems: Designed for easy extension to various file systems.
5. Integration with Existing KCL Code: Minimizes code changes for existing projects when integrating.
6. Accurate Path Resolution for Virtual Files in Local Paths: Ensures precise localization if virtual files exist within local paths.
7. For the current KCL situation,the new vfs need to unify the rustc-vfs used in KCL and the ra-vfs used in IDE.
8. The performance requirements of the IDE need to be met.

Workflow

file/code ---> VFS + SourceFiles ---> Parser/Resolver/Codegen

• Pre-Compilation: Introduce the VFS (Virtual File System) to convert file paths into SourceFiles before compilation.
• During-Compilation: Use pkgpath as the key for retrieving files. Each pkgpath corresponds to Vec<SourceFile>.
• Path Consistency: Within the compiler, consistently use pkgpath style paths (regardless of Windows/Unix conventions). The conversion from different path styles to pkgpath is handled within the VFS.

Parts

• PkgPath: the unified path style for VFS.
• VFS: the virtual file system.
• SourceFile: the file in VFS.

PkgPath

PkgPath is the unified path style for VFS.

struct PkgPath {
    pkgpath: String, // the path in VFS, like "a.b.c.d"
    extension: Option<String>, // the file extension, default is "k"
    root: Option<String>, // the root path, default is "${KCL_MOD}"
} 

Implement the Into/From trait for PkgPath, convert PkgPath to PathBuf/Path, and adapt to the corresponding path style according to different platforms in the into() method.

VFS

VFS is the virtual file system.

pub trait VFS{
    // Load the local file path from `path` into VFS.
    fn load(path: String) -> Self;
    // If the `pkgpath` is loaded into VFS.
    fn exists(pkgpath: PkgPath) -> bool;
    // If return the `SourceFile` by `PkgPath`
    fn get_source_files_by_pkgpath(pkgpath: PkgPath) -> Vec<SourceFile>;
    // Add a specified file into VFS
    fn insert_source_file_by_pkgpath(pkgpath: PkgPath);
}

1. Load the local file path from path into VFS.

let vfs = VFS::load("/usr/local/main.k");

2. If the pkgpath is loaded into VFS.

pkgpath maybe a file or dir.

If it is a dir, a.b.c.d is ${KCL_MOD}/a/b/c/d. If it is a file, a.b.c.d is ${KCL_MOD}/a/b/c/d.k by default. the extension can be specified by argument.

vfs.exists(PkgPath::new("usr.local"));
vfs.exists(PkgPath::new("usr.local.main"));
vfs.exists(PkgPath::new_with_extension("usr.local.main", Some("k")));

3. Get SourceFile by pkgpath.

let pkgpath = PkgPath::new("usr.local");
let sfs: Vec<SourceFile> = vfs.get_source_files_by_pkgpath(pkgpath);

let pkgpath = PkgPath::new_with_extension("usr.local.main", Some("k"));
let sfs: Vec<SourceFile> = vfs.get_source_files_by_pkgpath(pkgpath);

SourceFile

At present, the SourceFile in rustc has been used in KCL as the file abstraction. Here, the SourceFile in rustc is encapsulated twice, and new capabilities can be extended on the basis of reusing the existing SourceMap capabilities.

struct SourceFile {
    // Still use the `SourceFile` in rustc.
    sf: rustc::SourceFile
    // `PkgPath` is used to associate `SourceFile` with `a.b.c.d` style path.
    pkgPath: PkgPath
}