Windows fixes -- vfs.c and preprocess.py

[xobs]

This patchset fixes two issues encountered when porting to use frogfs:

1. preprocess.py doesn't work correctly on Windows due to how Node.js implements its commands. The workaround is to prefix these commands with cmd /c.
2. vfs.c appears to be unimplemented, and contains a few typos. I have fixed these typos in order to get it to build, however I am unsure of what the len variable is supposed to be in:

diff --git a/src/vfs.c b/src/vfs.c
index ceb7ef9..dd0a54c 100644
@@ -427,8 +427,8 @@ static int vfs_frogfs_readdir_r(void *ctx, DIR *pdir, struct dirent *entry,
     if (vfs_frogfs->flags & VFS_FROGFS_USE_OVERLAY) {
         if (!dir->overlay_dir) {
             char overlay_path[256];
-            frogfs_get_overlay(vfs_frogfs, overlay_path + len, dir->path,
-                               sizeof(overlay_path) - len);
+            frogfs_get_overlay(vfs_frogfs, overlay_path, dir->path,
+                               sizeof(overlay_path));
             dir->overlay_dir = opendir(overlay_path);
             if (dir->overlay_dir == NULL) {
                 *out_dirent = NULL;

I do not have a solution to the lack of venv on ESP on Windows, aside from removing venv from cmake/functions.cmake, which works on my system.

[jkent]

Thank you for looking into this. I will look and test this soon!

[xobs]

I've added two more commits that improve Windows compatibility.

One checks for the build root, and fixes an issue wherein npm install was called as part of every build. This increased build time, and appeared to be caused by the frogfs cmake script assuming that node_modules is located inside the frogfs directory. In my project, node_modules got located in the project root.

The other issue avoids using venv on Windows, where Espressif doesn't ship a venv module.

[xobs]

Thanks for the merge!

In the interim, the yaml libraries appear to have rotted. As a workaround, and to keep things working longer, I replaced yaml with json in my most recent build. This keeps CI happy, since it rebuilds things each night: https://github.com/farpatch/frogfs/commit/6f19e25280a2c8ed2ee0436ee98c1600517d8f9c

[jkent]

Sorry it has taken so long to get back to this. I have plans moving forward to clean up the python and npm mess even further with a directory of filter scripts:

babel-convert-processor.js  heatshrink-compressor.py    uglifycss-processor.js
babel-minify-processor.js   html-minifier-processor.js  uglify-js-processor.js
gzip-compressor.py          terminate-processor.py

This will allow a project tools/filters directory to add and override default filters.

instead of configuring each one. Additionally they will manage their own dependencies:

try {
        require('@babel/core/package');
        require('babel-preset-minify/package');
} catch {
        const { spawnSync } = require('node:child_process');
        spawnSync('npm', ['install', '@babel/core', 'babel-preset-minify'], {'stdio': ['ignore', 'ignore', 'inherit']});
        spawnSync('node', process.argv.slice(1), {'stdio': 'inherit'});
        process.exit(0);
}

I'll probably further refine this to pin a version when running npm. nodejs should not be a hard dependency, which this resolves. I want to keep YAML configuration. The way I implemented it is ugly, however. I abused YAML very badly.

Expect a bump in major version binary compatibility as I refine the way compressed files are handled. I think its better that compressed files have a user defined header of given length prepended to them... and that header should identify the compression type along with the uncompressed length. That way compression could easily be an optional feature to reduce binary size. Sorry, I'm just thinking this out here.

[jkent]

@xobs I've put a lot of work into cleaning up the Python and made C decompressors modular -- including zlib deflate. Curious if you can take a look. My next step is to work on integration with ESP-IDF, so far I've only been working with the standalone Linux binary. I took care of the dependency nightmare -- things are used only when needed.

[jkent]

@xobs Woops, I got ahead of myself, I left off the url: https://github.com/jkent/frogfs/tree/next