eli-schwartz
commented
2 years ago Your fuzzing tool which just generated 17 autogenerated bug reports, has submitted reproduction cases that consist of meson.build samples which use fully entropic variable names, which may be hard to distill into clear (human-readable) reproducers.
In at least one case (#9648) the ticket has 15 comments! And the reproducer is 500 lines long, and it looks like the comments may be too. I simply wouldn't bother to even look at that one.
Avoiding crashes sounds like a nice idea. It would be good to target it in some coordinated way rather than having a raft of issues for each possible route. Perhaps with a tracking meta-bug that links to a collection of test cases in another git repo, which could be run, reproduced, conveniently distilled, etc.... and declare success and close the meta bug once running the collection as s series of unittests, no longer produces any python tracebacks.
I wonder how many of these issues could be solved just by enforcing more consistent typed_kwargs and typed_pos_args, work that has been ongoing for a while now? It's a known problem that originally, all functions just accepted "literally anything" as values, and many still do.
gerion0
mensinda
bonzini
nnamua
Volker-Weissmann
Description
Starting in spring, I wrote my bachelor thesis. My task was to develop a classification of Meson objects and implement a checking tool that fuzzes Meson automatically to check against this classification.
Motivation for this were bugs like #3469 and #5618 where the user expects something which Meson doesn't provide. Furthermore it should clarify inconsistencies or inaccuracies in the documention. For example the allowed input types of the
message()function which are implemented but not documented. My supervisor (@gerion0) has already created a meta issue (#6068) for this.The result of this work can be found here. This tool was/is able to find the following already reported bugs/feature requests: #1550, #2657, #3175, #3206, #3469, #3667, #4280, #4892, #4893, #4891, #5401, #5706, #5935, #6073, #6476, #6567, #6675, #6702, #6960, #7585, #8039, #8295, #8353, #8373, #8433, #9108, #9175
It also reports a bunch of bugs (Meson crashes with a Python trace in this case). I have reported these bugs in the following issues (grouped by their last line in the Python trace): #9647, #9648 ,#9649, #9650, #9651, #9652, #9653, #9654, #9655, #9656, #9657, #9658, #9659, #9660, #9661, #9662, #9663
Additionally, my tool reports a lot of inconsistencies which I want to report as "feature request". For example one result is that an entity of
files()does not have afull_path()function. The following fails:All those "inconsistencies" together are quite a lot. Therefore I have not created issues for that but like to ask how I should report them.
My plan is not only to report those bugs, but to bring the work upstream.
Recently, I discovered this pull request #8960 which seems to solve a very similar problem. The pull request is merged. However, both approaches are somewhat different. Therefore, I want to provide my solution as well. Maybe it can be merged with the other approach to make Meson even better.
The differences (as I experience them) are:
So to summarize. Both approaches are similar. My approach should automatically retrieve bugs and Python does the parsing, the upstream approach should generate an advanced documentation and the parsing has to be done manually.
For the automatic checking, I need to interact with Meson. I have not found a way for Meson to report object types directly. So my approach for this is:
Ideally, both approaches can be combined, so Meson uses the yaml files directly (Or is this already happening?).
Do you think my approach provides an additional benefit? Is it worth to combine both approaches (a bug finding tool that use the yaml files instead of the Python files should be doable)?