Andrew's header removal scripts are great, but they're unwieldy to use with the current codebase. Something that I've been poking into for a while is using the Clang-parser based include-what-you-use tool (https://include-what-you-use.org/). This does an AST-based scan of files, assigning the used symbols to their respective headers, and marking up inclusions accordingly.
The benefits of IWYU is that it's relatively fast (a full run takes ~1.5 hours when spread across 12 CPUs). It also is more aggressive in forwardizing things. (As I understand it, Andrew's tool won't suggest converting a regular header to a forward header if the fwd.hh doesn't exist -- this version does.) On the flip side, my gut sense is that IWYU may not be as aggressive in removing headers, but it should get us most of the way there.
Unfortunately, we can't use the tool as-is, due to differences in how we lay things out and how it works. (Most notably how it handles forwards, and how it interacts with using directives.) But I've put together scripts to run IWYU, parse the output, and then apply the fixes. A big portion of the script is the "cleanup" phase, which attempts to massage the raw IWYU output into something which better matches Rosetta conventions. Part of that is a few configuration files which allow us to change (in an out-of-code fashion) how Rosetta.
An example of the current state of the automatic fixes is at https://github.com/RosettaCommons/main/commit/daea6e97b06586b42086dfa0f1c104de9f0ef3c8 or git show origin/roccomoretti/IWYU_example locally, (the auto fixes are just the most recent commit - prior commits on that branch are manual fixes to get the script to run semi-cleanly) -- Note that I haven't tried making sure everything compiles, so there are probably a large number of manual fixups which need to be done. I'm mainly putting it out there so people can spot-check it and help catch any bone-headed changes - particularly additions - which could be fixed by improving the tooling. (As mentioned, there's a decent amount of customizability in cleanup/handling.)
Finally, it has not escaped my attention that there's the possibility of potentially turning this into a (semi-)automated test which can run on a regular basis, but I leave that for future expansion.
P.S. This script is intended to run with the most recent IWYU (0.14) with Clang 10.
Andrew's header removal scripts are great, but they're unwieldy to use with the current codebase. Something that I've been poking into for a while is using the Clang-parser based include-what-you-use tool (https://include-what-you-use.org/). This does an AST-based scan of files, assigning the used symbols to their respective headers, and marking up inclusions accordingly.
The benefits of IWYU is that it's relatively fast (a full run takes ~1.5 hours when spread across 12 CPUs). It also is more aggressive in forwardizing things. (As I understand it, Andrew's tool won't suggest converting a regular header to a forward header if the fwd.hh doesn't exist -- this version does.) On the flip side, my gut sense is that IWYU may not be as aggressive in removing headers, but it should get us most of the way there.
Unfortunately, we can't use the tool as-is, due to differences in how we lay things out and how it works. (Most notably how it handles forwards, and how it interacts with using directives.) But I've put together scripts to run IWYU, parse the output, and then apply the fixes. A big portion of the script is the "cleanup" phase, which attempts to massage the raw IWYU output into something which better matches Rosetta conventions. Part of that is a few configuration files which allow us to change (in an out-of-code fashion) how Rosetta.
An example of the current state of the automatic fixes is at https://github.com/RosettaCommons/main/commit/daea6e97b06586b42086dfa0f1c104de9f0ef3c8 or
git show origin/roccomoretti/IWYU_examplelocally, (the auto fixes are just the most recent commit - prior commits on that branch are manual fixes to get the script to run semi-cleanly) -- Note that I haven't tried making sure everything compiles, so there are probably a large number of manual fixups which need to be done. I'm mainly putting it out there so people can spot-check it and help catch any bone-headed changes - particularly additions - which could be fixed by improving the tooling. (As mentioned, there's a decent amount of customizability in cleanup/handling.)Finally, it has not escaped my attention that there's the possibility of potentially turning this into a (semi-)automated test which can run on a regular basis, but I leave that for future expansion.
P.S. This script is intended to run with the most recent IWYU (0.14) with Clang 10.