I want to make a preprocessor.

[IngwiePhoenix]

I have...a problem. x) My "web library" is a total mess and I can do what I want, but WebPack won't play nice with OJ the way I /really/ want it. Therefore I would like to extend OJ by something that Objective-C++ has: a preprocessor.

Example:

@include <MyFoundation/dialog>
@include "some-local-file.oj"

@implementation WarningDialog : Dialog
+(id)run:(string)message {

    return [super makeDialogWithMessage:helper(message) cssClass:"warn"];
}
@end

$ ojc -I lib/ ./main.oj -o app/app.js
- Preprocess
- TypeChecker / Sanitize
- Compile

$ ojc -I lib/ ./main.oj -o main --bundle
...
- Compile
- Bundle runtime

The --bundle flag is something I randomly came up with. For single-page apps, this might actually be useful.

Anyway. Which files would be important for me when I wanted to place the preprocessor before the "actual" process?

My implementation, as I would imagine it at this point, would be:

• Option to add include path for <...> includes. "..." are resolved from file itself.
• Extending ojc to support -I (and maybe --bundle while i am at it...)
• Getting the source before the actual procession to "manipulate" it, possibly actually generating a source map from the various files used.

The preprocessor would support:

• @include <...>
• @include "..."
• @ifdef
• @ifndef
• @define
• @undefine

I am going with the @ symbols since they'd feel more natural due to @squeeze, and @property and the like.

[iccir]

Would using the existing preprocessor (gcc -E or clang -E) give you want you want? If you decide to roll your own, I'd recommend against using @ symbols and instead use #. @ should be reserved for language features and I can't guarantee that I won't use @define in the future ;) # tends to be instructions for the preprocessor stage.

[IngwiePhoenix]

Not everyone has GCC or Clang installed - especially not Windows users. So I'd rather roll my own. :)

I can go with # too - I just thought it'd fit more to the language at hand.

So which files/parts would be important for me to look at? I am not really good on parsers and lexers, so knowing where to hook into would be really good :) (I'll learn how to understand parsers and lexers one day... o.o)

[iccir]

So, generally: a preprocessor works as a separate module that takes a string input and also outputs a string. That string is then handed to the compiler (ojc in this case), which runs the tokenizer/parser/AST transformations/etc on it. The C preprocessor doesn't have any knowledge of the language that it's preprocessing, as the preprocessing stage happens before any parsing occurs (hence, there aren't any files that you necessarily should look at. What you will probably do is have a command that wraps ojc, takes additional arguments, preprocesses the input strings, and then passes those into ojc.)

I'd start with gcc/clang just to make sure that preprocessing solves your problem. Or I'd look at one of the existing JS preprocessors (https://www.npmjs.com/package/preprocessor maybe?) and possibly extend it.

If you decide to write your own, I'd start with trying to make simple #defines work.

Basically: 1) Take the input string and split it into lines 2) If a line starts with #define, parse the line and add the definition to a map 3) Else, split the line into very basic tokens (separate by whitespace and symbols). 4) For each token, if that token is in the definition map, replace it.

[IngwiePhoenix]

Oh don't worry - I know what a preprocessor is. I have written a tiny one that is used in connect-oj that allows arbitary JavaScript to be used for the pre-processing. A tiny and hacky aproach but it worked for that purpose.

I also have looked at existing preprocessors, but none really did exactly what I wanted, which is why I want to roll my own in the long run that supports #include "" and include <> in the way an Ansi-C style preprocessor would.

I'm going to work ont hat and then see how I can "add" it to OJ. But I really want to make one. :)

[IngwiePhoenix]

I have the preprocessor almost complete. The only things that it doesn't do just yet is #ifdef, #ifndef, #else and #endif.

To test the preprocessor more in action, I would liek to insert it at a point before the OJ source is given to the compiler or builder. I have made a fork with my own branch being currently local-only. But I am going to see how the preprocessor works inside oj itself.

Another idea that I just had was to not actually add the preprocessor into OJ itself, but rather create a project that is based off OJ but allows plugins.

ojc.use(function(ctx, next){
    // ctx.options
    // ctx.source
    // ctx.filename
    ctx.source = Preprocessor(ctx.source, ctx.filename,{
        include_path: [...],
        defines: {...}
    });
});

ojc.compile(...);

Which version would you prefer; the embedded preprocessor or extern using a plugin system?

[iccir]

Extern - I'd rather keep all components separate and modular. I'm not sure if I see the purpose of ojc.use here: you are presumably calling ojc.compile() with the files argument set to { path: ..., contents: ... } objects. To preprocess:

let files = ...;

for (let file of files) {
    file.contents = Preprocessor(file.contents, file.path, ...);
}

ojc.compile({ files }, ...);

[IngwiePhoenix]

Alright, this is what I have made thus far. https://github.com/DragonsInn/OhSoJuicy

The reason for this is, that neither Browserify nor WebPack can handle multiple files at once to a transformer or loader - only one at a time. Therefore, one can use a preprocessor to include files and sections that are required for the code to run properly.

All I am stuck at, is the if logic. But I will get that one at some point. I...just gotta find a start.

[IngwiePhoenix]

I got aorund it, and the preprocessor works for themost part - good enough that I am loading it into my production app. :)

[iccir]

Note: ojc 2.x added the before-compile hook and a simple preprocessor is included in the documentation. I don't think there is any work to be done here, so I'm going to go ahead and close this issue.