dashboard/app: support folded fixes, bugs in syzkaller, etc

google / syzkaller

syzkaller is an unsupervised coverage-guided kernel fuzzer

Apache License 2.0

5.39k stars 1.23k forks source link

dashboard/app: support folded fixes, bugs in syzkaller, etc #1878

Open dvyukov opened 4 years ago

dvyukov commented 4 years ago

Currently we have to mark classes of real bugs as #syz invalid b/c we can't express how to handle these bugs with existing commands. These include:

folded/amended fixed in linux-next
bugs in syzkaller itself
bugs in tools under development (in particular KMSAN, where a fix won't reach all tested trees, at least soon)
bugs that are real but stopped happening and we don't know what exactly fixed them

For tracking purposes it may be useful to allow distinguishing these classes from actually "invalid". We could either extend commands and/or use tags (#608). But ideally it should be something very simple because spreading knowledge about each new command is very hard.

dvyukov commented 4 years ago

@ebiggers do you have any suggestions on what would be a good way to mark these? or any other classes of bugs that would be useful to handle differently.

ebiggers commented 4 years ago

folded/amended fixed in linux-next

#syz folded-fix: $commit_title

bugs in syzkaller itself

Just keep #syz invalid

bugs in tools under development (in particular KMSAN, where a fix won't reach all tested trees, at least soon)

Do you mean cases like https://groups.google.com/g/syzkaller-bugs/c/XEO4s9Kj37E/m/hqSlXzzSAQAJ? That seems like a bug in syzbot -- it was waiting for the bug to be fixed in branches that don't have the tool.

Either that or just keep #syz invalid

bugs that are real but stopped happening and we don't know what exactly fixed them

#syz fix: unknown?

dvyukov commented 4 years ago

#syz fix: unknown is an interesting syntax. Indeed we could some reserved commit titles that would play special role. Also e.g. #syz fix: syzkaller for bugs in syzkaller.

For #syz folded-fix: $commit_title we can't understand when it's safe to close the bug... maybe we could wait some fixed period (e.g. 1 week) and then close as #syz invalid would do...

That seems like a bug in syzbot -- it was waiting for the bug to be fixed in branches that don't have the tool.

But is it possible to understand automatically what commit will end up in what tree? I don't see a reasonable way to do it. Eventually that commit may appear in other trees as well... Or what do you mean?

ebiggers commented 4 years ago

But is it possible to understand automatically what commit will end up in what tree? I don't see a reasonable way to do it. Eventually that commit may appear in other trees as well... Or what do you mean?

Since KMSAN isn't upstream, we should have a list of branches it's on, right? I expected that bugs reported from a KMSAN branch would only wait for the bug to be fixed on KMSAN branches, not all branches.

dvyukov commented 4 years ago

But is it possible to understand automatically what commit will end up in what tree? I don't see a reasonable way to do it. Eventually that commit may appear in other trees as well... Or what do you mean?

Since KMSAN isn't upstream, we should have a list of branches it's on, right? I expected that bugs reported from a KMSAN branch would only wait for the bug to be fixed on KMSAN branches, not all branches.

Right, we could special-case kmsan tree. It should work because we don't report any non-KMSAN bugs on that tree. We still few open bugs that happen on both kmsan and non-kmsan trees (like "lost connection", "no output"): https://syzkaller.appspot.com/upstream?manager=ci-upstream-kmsan-gce But fortunately, they all already happened on all trees, so we won't have a situation when a bug looks like kmsan-only, but later becomes non-kmsan-only (which would cause problems for this special casing).

dvyukov commented 4 years ago

For #syz folded-fix: $commit_title we can't understand when it's safe to close the bug... maybe we could wait some fixed period (e.g. 1 week) and then close as #syz invalid would do...

I think we should also suggest using commit tags for folded fixes. I think the main complain was that using "Reported-by" suggests this is a bug fix, but the resulting commit is not necessary a bug fix. But we could suggest using only "Tested-by" for these cases, which seems reasonable and won't make it look like a fix.