promote self-hosted C tokenizer and parser to a general-purpose C tokenizer and parser in the standard library

[andrewrk]

Currently Zig has the file src-self-hosted/c_tokenizer.zig which is used to tokenize macros for translate-c purposes. It also has some functions in src-self-hosted/translate_c.zig which do AST parsing of C tokens:

• parseCPrimaryExpr
• parseCExpr
• parseCSuffixOpExpr
• etc

Zig has set a (in my opinion successful) precedent in exposing its own tokenizer, parser, and AST in the standard library, in these files:

• std/lib/zig/ast.zig
• std/lib/zig/parse.zig
• std/lib/zig/render.zig
• std/lib/zig/tokenizer.zig

Now it is time to do the same for C:

• Decouple C tokenization and AST building from translate-c
• Expose C tokenization and AST building (and rendering?) in std/lib/c/* in the same way as zig.
• Complete the implementation of C tokenization and AST building. Make it compatible with the specification as well as GCC extensions, and make it robust. Someone should be able to write a C compiler in zig comfortably by using the standard library C tokenizer and parser.
• translate-c should use this generic and robust C tokenizer and parser API, rather than its current, only-what-we-need-to-make-it-work implementation.

cc @Vexu

[Vexu]

I'm assuming we also want a preprocessor? Are we trying to eventually replace clang in translate-c completely?

[andrewrk]

That could be a direction to go with this, although I wouldn't call it "planned" yet. The main idea here is to make the API abstraction layer of the C tokenization and parsing clear and understandable. That will have the benefit of making it easier to test - there are plenty of pre-existing test cases for parsing C code, for example. So this will result in a more robust translate-c feature.

[andrewrk]

Vexu landed #4317 in 0cc2489d22a27b2dc82ee9ef72e945b9fa97c8fe. This started the efforts toward implementing this issue. What remains until this can be closed:

• [ ] set up testing for the parser to make it more contributor friendly
• [ ] integration with translate-c

[moosichu]

This issue interests me because I'm using zig within the context of a C/C++ project (for now) - and being able to do simple static reflection on my codebase from build.zig using this library would be great. So I've subscribed to the issue for now, but if there is anyway I could help/contribute I might be able to look at it sometime down the line if no-one has done so before-then :)

[Vexu]

I've started work on a C parser/compiler that could eventually be upstreamed. I haven't touched it for some time but I'm interested in continuing it especially if someone would want to contribute.

[moosichu]

Let me know if know there's anything I can do to help.

[g-berthiaume]

I'm commenting to follow this thread. I'm considering making a metaprogramming tool for C and the ability to have a C parser/tokenizer out-of-the-box in zig would facilitate this project greatly.

As always, thanks for your work!

[Vexu]

This issue will likely be closed in favor of using https://github.com/Vexu/arocc as a package.