boris-kolpackov
commented
5 years ago Thanks for starting the issue and capturing your thoughts. I think as a first step we need to get a good understanding of what are the common static analysis tools (which we may want to support in build2) and how they are typically (or atypically -- we may want to do things differently in build2, for example, to support incremental analysis) used.
We've done a bit of research on clang-tidy and it appears the way it is commonly used is by first generating the compilation database and then invoking a (Python) scripts that (presumably) executes the actual analyzer on each translation unit from said database. There is also a somewhat related discussion.
So I guess the question that I would like to answer is what other common static analyzers are there and how are they used? Also, I wonder if this "analyzing one translation unit in isolation" is the norm. Any tools analyze headers?
Klaim
matta
johan-boule
From Slack channel:
About static analysis
In the project(s) I was thinking about the answer would be both:
This is to enable both developer's work station checks and ease adding static analysis to CI (though this last case is far easier to do with a CI script).
The main reason I can think about why static analysis would not be automatic is if it takes too long and that depends a lot on tooling and the code itself. Therefore I think it's important to give the choice. For example we have a situation where some components/projects are critical dependencies for all other projects in the company. Having them checked all the time (in diverse way) prevents a lot of issues that will only be visible once users try to use them, and often are silent issues. For these critical projects, having a maximum of checks going at all developments stages is useful. For the users projects, it's less important and can slow down development speed of things that are perceptible to the end-users (while critical projects are not perceptible, except when they fail).
It looks to me that if operation 1) is possible, then a choice of deciding if having reports from analysis should prevent further build operations would be useful (and relative to the project or use case).
How to enable this kind of feature: custom (meta-)operations?
Trying to look at the general picture, maybe having a way to add operations to the build system would enable static analysis operations. If the project using build2 can describe these operations in it's own source code, that would enable a lot of custom project or organization-specific operations without having to formalize an interface for every category of operation. One other advantage is that it would allow custom behaviors depending on some custom parameters, like stopping the build if static analysis report any important issues, or not, or generating some output locally or on a specific location online.
It would also go in the same direction that is being taken (work in progress at the moment if I understand correctly) to allow custom target definitions. I'm not sure how harder allowing custom operations would be compared to targets, but it looks like they would be less contrived in possible behavior than targets, which might be a design problem (in the same way enabling procedural scripts at build time enables a lot of problems).
One other possible usage of such custom operation would be to add a benchmark operation. This was slightly discussed before as already possible by running tests that are in a specific directory which contains only benchmarks. This is enough for most cases I believe. Now if instead of that we add a custom benchmark command, we can also add collection and storage in remote location for analysis to the whole process, after running the targets that are marked as being benchmarks. While this would be useful, it is to be determined if it's a big enough advantage to have that complex process being done through a custom build2 command, compared to just adding a separate script somewhere to do that, launched manually or by the benchmark tests programs themselves.
How to enable this kind of feature: custom target
If we were using GNU make, my understanding is that launching static analysis could simply be a target. In build2 it would be a target type, to generalize.\ In this case, automating analysis could be achieved by implementing a custom target and adding that target as requirement for projects that need to be analyzed throughly.
However, how then would we run analysis "on demand" and/or for a specific target? I didn't think much further yet, so maybe some target setup can allow this kind of things.
How to enable this kind of feature: specific analysis meta-operation
Static analysis is a well known topic for all languages so it's not something that is uncommon. Maybe just adding an operation (like for test) would be simpler and more useful. In this case, the main work would be to decide what are the general behaviours of static analysis and add a way to specify static analysis tools and usage patterns so that build2 can use them (in the same way it works with compilers).