search

php / php-src

The PHP Interpreter
https://www.php.net
Other
38.22k stars 7.75k forks source link

A new fuzz testing tool for PHP #16815

Open YuanchengJiang opened 3 hours ago

YuanchengJiang commented 3 hours ago

Description

Hi all,

I have submitted hundreds of bugs during the past months and I first thank all the developers who take time to fix these issues to make PHP better.

I am thrilled to introduce one fully automated fuzz testing tool, FlowFusion, for discovering various bugs of the PHP interpreter.

The core idea behind FlowFusion is to leverage dataflow as an effective representation of test cases (.phpt files) maintained by PHP developers, merging two (or more) test cases to produce fused test cases with more complex code semantics. We connect two (or more) test cases via interleaving their dataflows, i.e., bringing the code context from one test case to another. This enables interactions among existing test cases, which are mostly the unit tests verifying one single functionality, making fused test cases interesting with merging code semantics.

FlowFusion additionally fuzzes all defined functions and class methods using the code contexts of fused test cases. Available functions, classes, and methods are pre-collected and stored in sqlite3 with necessary information like the number of parameters. FlowFusion will be automatically upgrading if phpt files keep updating. Any new single test can bring thousands of new fused tests.

The search space of FlowFusion is huge, which means it can cover various corner cases. Reasons for the huge search space are three-fold: (i) two random combinations of around 20,000 test cases can generate 400,000,000 test cases, and we can combine even more; (ii) the interleaving has randomness, given two test cases, there could be multiple ways to connect them; and (iii) FlowFusion also mutates the test case, fuzzes the runtime environment/configuration like JIT.

I can open-source the tool under my personal repository. I wonder by any chance if I can contribute it as the official PHP tool under https://github.com/php, and I would be happy to maintain it for a long time.

devnexen commented 3 hours ago

And thanks for all your reports. I invite you to discuss this matter in the internal mail list, in addition you will garner more attention.

TimWolla commented 27 minutes ago

For reference: https://externals.io/message/125963