Preset Parser to Support Syntax Specifications

[mertovun]

Current preset parser does not fully support the syntax specifications, e.g. disregarding multi-line comments. A parsing and language processing library (e.g. Boost.Spirit) can be used as an alternative to current line-by-line parsing logic.

[kblaschke]

Could you please provide a test case where this happens? In my tests, multi-line comments work just fine.

Multi-line comments should already be discarded in the scanner, using this Lex grammar here:

https://github.com/projectM-visualizer/projectm-eval/blob/ee180a2473c12856fd25dc754bafdab7e843cc7a/projectm-eval/Scanner.l#L49

If that doesn't work for you, please provide more details on how to reproduce the issue.

[mertovun]

As I understand it, preset files are parsed into a preset state in a line-by-line fashion, governed by the PresetFileParser and PresetState classes. Each line is evaluated as a potential key-value pair, which is then either assigned to a parameter or accumulated into their respective code buckets according to label and index (frame, pixel, shader, etc.). These buckets may then be evaluated and parsed again individually at a deeper level, which has been outside the scope of my review so far.

To reproduce the issue:

[kblaschke]

Comments are only supported within expression code (and only within the same block, e.g. perframe#) and shader code (warp and comp blocks), nowhere else. In your example, you try to use them in parameter values, which are handled completely different.

There are a few different "levels" of parsing happening there, which is the actual behavior in Milkdrop as well as projectM:

First, the milk file is read in, which is a simple Windows INI file, having key/value pairs and (actually ignored) sections within [] brackets. The file parser simply reads in all keys and stores them in a map with the first value encountered, so further duplicate keys are ignored.

Then, in the preset loading process, some keys are simply converted to int/float values (like the ones you tried the multi-line comments on).

Then there are the code blocks, which have a common prefix like "perframe", followed by a number starting from 1. Keys are read in increments of one until a key is missing. All values read are concatenated with line breaks and stored as a code block. Shader code has a back tick after the equal sign, which is truncated - not exactly sure why that was done, but it's what Milkdrop does. These code blocks are then either parsed as expressions using projectm_eval or the HLSl shader code is translated to GLSL via hlslparser. Multi-Line comments are only valid in those two parsers, and as said before, cannot span over multiple code blocks in any case.

That's basically all. Lines in the milk file which aren't used are simply ignored. You can write anything in there, both Milkdrop and projectM just won't care.

All that said, I'm still quite sure that the parser works as intended and in a very similar way as Milkdrop's file reader. The only real difference here is that Milkdrop scans the milk file line by line until "key=" or "key =" is found and the function returns the value, while projectM just reads the whole file into a map, so the file is only read once. The resulting behavior is identical though.

If you still think that the implementation is wrong and some case that works in Milkdrop doesn't work in projectM, please provide an example and ideally point to the original Milkdrop code that does the parsing you're referring to.

[mertovun]

Thank you for the clarification and the insightful explanation.

I’ve been considering the idea of a browser-based Milkdrop preset editor with UI panels, although I haven't fully thought through its feasibility or ergonomics yet. This project would require a TypeScript implementation that can parse the file format into a state object, which could then be programmatically adjusted and converted back into a string which could be fed to projectM through the wasm interface for visual feedback (as far as I understand from previous discussions, compilation into wasm is already possible). That’s what led me to review the original parsing logic here. Do you have thoughts on this?

[kblaschke]

Yes, projectM compiles to wasm. It also has a function to load a preset from a string, which would just be the contents of milk file. You'd have to implement reading and writing this INI file format, which is actually very easy - Milkdrop for example has very straightforward code that does exactly this.

The biggest challenge is reading all the many values and set them to the proper defaults if keys are missing in the file. For that, you'll have to go through either Milkdrop's or projectM's loading code and retrieve the defaults, as they're not fully documented elsewhere.

In addition to that make sure to correctly implement the reading code, in regards to reading the correct lines and aborting to read code blocks if an index is missing. Writing the file is just a static blob of code, writing all the lines with their respective values, ideally in the same order as Milkdrop - yet that's not strictly required. It would make changes easier to diff though.

See Milkdrop's state.cpp file for all the .milk file parsing and writing code. It's very basic stuff, not much logic in there.

If you want to parse and syntax-check the expression code, you can use projectm-eval for that as well. Compile the lib as wasm and create JS wrappers for the small C API. You can then compile and even run expressions. There's also a function that will return compilation errors including line and column of the error, so you could display that to the user as well.

The fragment shader code blocks would be more involved, but I'm sure there are HLSL validation libraries for JS out there. Note that the preset shader blocks aren't the full picture, there are lots of defines and other things added to the shader code which you'll have to do as well before passing the shader to a validator. See projectM's MilkdropShader class for an example.

[mertovun]

Great, I will look into it. Thanks!