mkfrogfs.py directory filters are broken

[jkent]

Currently it is not possible to match just a directory and apply filters on it. You have to glob it like the following to apply to all descendants:

filters:
  'path/to/folder_name*':
    - discard

You should be able to just do the following:

filters:
  'path/to/folder_name':
    - discard

...but this is currently broken.

[valkuc]

Web Root: webroot.zip And the config file:

filters:
  '/.test-data*':
    - discard
  '*.json':
    - compress deflate
  '*.js':
    - compress deflate
  '*.css':
    - compress deflate
  '*.woff*':
    - no compress
  '*':
    - compress heatshrink:
        window: 11
        lookahead: 4

[jkent]

Ah, try without the leading slash. It might seem ambiguous, but without a glob at the beginning, the filter starts with the root.

[valkuc]

It still won't work :)

       - Stage 1
         - index.html
           - compress heatshrink... done (41.0%)
       - Stage 2
Traceback (most recent call last):
  File "E:\Espressif\workspace\test-proj\components\frogfs\tools\mkfrogfs.py", line 528, in <module>
    generate_file_header(entry)
  File "E:\Espressif\workspace\test-proj\components\frogfs\tools\mkfrogfs.py", line 386, in generate_file_header
    data_sz = os.path.getsize(os.path.join(g_cache_dir, ent['path']))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 50, in getsize
FileNotFoundError: [WinError 3] Системе не удается найти указанный путь: 'E:/Espressif/workspace/test-proj/build/CMakeFiles\\frogfs-cache\\.test-data/state.json'

[jkent]

Delete your frogfs cache json file in the CMakeFiles directory. Yep. Another bug. Hahaha.

[jkent]

The whole caching thing, and keeping track of the state was (and still is) very difficult to get right.

[valkuc]

I'm actually did full clean (delete build dir) after changing yaml and before the build.

[jkent]

Oh interesting. Okay. I will work on this later tonight. Sorry I didn't do any local testing here. I've just experienced these things first hand when testing on my own during development. Thanks for your help in bringing these issues to my attention!

[jkent]

The discard verb should now work in the next branch. I've made a number of improvements to mkfrogfs.py. This issue is still unresolved as the cache verb does not work recursively on directories yet. The yaml format changed a bit, the filter glob patterns are a list now instead of a dictionary. You can now have duplicate filters, and they are evaluated first-come first serve. I need to update the documentation still, and I will do this once I fix the cache verb. Please refer to default_config.yaml

[valkuc]

Great! I'll try to find a time to check the changes next week. About cache keyword. Mmmm. I have mixed feelings about this. From one hand it's interesting to have FS which will provide data about every file and how to handle that file. But, from another hand, maybe it's better to stick with simplicity? It looks like FrogFs is based on (or takes it roots from) EspFs invented by SpriteTm. So, maybe it's better to go this way and just provide a stable and working read-only FS for ESP-IDF and without any "bells and whistles"?

[jkent]

Oh, the cache is a build time thing. It means that it doesn't watch the mtime of the file. It might need a better verb. It has nothing to do with runtime. Caching is enabled by default. "no cache" means that every time mkfrogfs.py runs, it will preprocess the file and run stage 2 every time. It is useful in debugging and developing transforms.

[valkuc]

Oh, the cache is a build time thing. It means that it doesn't watch the mtime of the file. It might need a better verb. It has nothing to do with runtime. Caching is enabled by default. "no cache" means that every time mkfrogfs.py runs, it will preprocess the file and run stage 2 every time. It is useful in debugging and developing transforms.

Oh, I had wrong understanding of "cache". Looks like it's very useful feature.

[valkuc]

The discard verb should now work in the next branch. I've made a number of improvements to mkfrogfs.py. This issue is still unresolved as the cache verb does not work recursively on directories yet. The yaml format changed a bit, the filter glob patterns are a list now instead of a dictionary. You can now have duplicate filters, and they are evaluated first-come first serve. I need to update the documentation still, and I will do this once I fix the cache verb. Please refer to default_config.yaml

I checked new changes and looks like all is fine. Noticed that now there is __pycache__ dir created in frogfs/tools directory. Maybe I'm wrong, but I think earlier this directory was created somewhere inside project build directory (but I'm not sure :D )

[jkent]

This is normal behavior for Python 3. It generates optimized bytecode along side source files and stashes it in there. I confirmed that __pycache__/ is in the .gitignore, so it shouldn't get pushed.

[jkent]

With the latest iteration of mkfrogfs, this is irrelevant. You can now build a filesystem from multiple sources, piecing it together. I developed it for a project using Lua, and it works really well.