Would like an option to parse all includes but only output current file

[virtuald]

For robotpy-build, I use pcpp to preprocess the headers that we're wrapping to interpret any macros before generating the wrappers. Perhaps this is unique, but I don't actually care about the content of any of the included files, I only need them for their macro content -- I just want the current file contents (as modified by whatever macros were in the included files).

Currently my code writes the preprocessor output to a StringIO buffer, then checks each line for #line directives and only keeps the content that corresponds to the current file. https://github.com/robotpy/cxxheaderparser/blob/0280e8d7e190eef0180152dc693a0689d3a7338e/cxxheaderparser/preprocessor.py#L35

This seems rather inefficient if you have really large sets of header-only C++ libraries (like Eigen!), I'd rather have pcpp not write to the file descriptor at all if I don't need the content. If you're willing to accept such a change I'm happy to make a PR.

[ned14]

I would be surprised if the serialisation to a file in a tmpfs directory would be much overhead. Would you have some real world numbers showing the benefit of skipping the serialisation step?

[virtuald]

Well, here are some initial numbers of my robotpy-build branch that uses my new C++ parser cxxheaderparser. This was running 'create-gen' which only parses the files but doesn't write any wrappers to file. cxxheaderparser modifications to get these timing results are in https://github.com/robotpy/cxxheaderparser/commit/aef8128e13279f550ed15709957df2db23f52008

Here's the output from a single run on robotpy-wpimath, which has a lot of template-heavy math libraries (the individual reports are from different sections of the library that are being parsed in separate groups from each other). All times are in seconds using measurements from time.monotonic().

robotpy-wpimath]$ robotpy-build create-gen
Nothing to do!
-- report --
 files=2 total_time=3.91967
 preprocessor: parse=0.00005, write=3.86695, filter=0.02663
Nothing to do!
-- report --
 files=41 total_time=106.35850
 preprocessor: parse=0.00026, write=105.30888, filter=0.61420
Nothing to do!
-- report --
 files=4 total_time=4.05496
 preprocessor: parse=0.00002, write=3.99326, filter=0.02223
Nothing to do!
-- report --
 files=13 total_time=26.63185
 preprocessor: parse=0.00008, write=26.35617, filter=0.15224
Nothing to do!
-- report --
 files=1 total_time=0.81422
 preprocessor: parse=0.00001, write=0.79652, filter=0.00569
Nothing to do!
-- report --
 files=12 total_time=18.02341
 preprocessor: parse=0.00007, write=17.84688, filter=0.09790
Nothing to do!
-- report --
 files=5 total_time=15.14729
 preprocessor: parse=0.00003, write=15.02130, filter=0.08693
Nothing to do!

(I realized after running this that the parse() call merely does the parser setup and that write() does most of the work).

... so, you're right, but the preprocessor definitely is dominating my processing. With the long runtimes I'm seeing, I suspect there's a lot of low-hanging fruit that could be optimized in pcpp. I'll do some profiling and see what I find.

[ned14]

Yup, the parser is lazy, only does anything when pulled upon. The actual writing of bytes to a file I would guess will not be expensive, it'll be everything else leading up to that point. As a pure guess, you might get at best single digit percent improvements by not writing to a file.

BTW have you tried pcpp on pypy? Those 106 seconds would likely be greatly reduced on pypy.