techraed
commented
1 month ago Thank you, @playX18! Great research been done! :+1:
Some thoughts to share.
Increase amount of gas by calculating theorethically needed amount of gas for each program we generate instead of using constant gas which also should help us run more programs without failure after short period of time. I think it's worth to also calculate allowed gas for each program that we generate based on estimation through config and total instructions in the program so we can run the generated programs until completion more often rather than failing because we exceeded our gas limit.
I guess we are absolutely ok with amount of gas we send right now. Having gas counter exceeded errors is ok, especially when they aren't that much, because the aim is not to have absolutely infallible executions.
max_funcs: almost never results in increasing coverage min_funcs: very similar to max_funcs in terms of what happens to coverage and program amounts.
Absolutely agree, no need to touch that. Maybe make it lower (max/min: 1) in some situations, so we have only syscalls executed. But that requires making lower the range of syscalls inserting into the program.
Control instructions do have a high impact on runtime of a programs and increase coverage significantly
Btw, in theory it looks wrong. You see, without Control instructions the ends up pretty fast as there are no loops and branches, just plain N instructions with syscalls. Actually, a variant with 1) no Control instructions, 2) less than 100 instructions (even less, but you still need some of them so memory related instructions are generated and program memory updates are performed in runtime), 3) 1-2 functions maximum and 4) a proper control of the amount of syscalls to be injected can be the most hopeful for the coverage. Can you check that please?
Random range of SyscallInjectionTypes in StandardGearWasmConfigsBundle
Here are 2 things I'd point.
First, I agree with the idea of changing the ranges for syscalls. That's great, but there can't be general approach, when you create semi-random syscalls injection ranges. Some of syscalls can be injected more than others. You should pay attention here to the possible gas consumption. See, in current config we insert 10-15 gr_send syscalls, which is absolutely okay when having 500 instructions and Control instructions enabled, as you have really low chance to execute even 5 of them (can't remember being so lucky, maybe when having loops, but that's a different story). If you lower amount of instructions and functions and leave 10-15 injections of gr_send, or even remove Control instructions, that can cause lots of problems with gas, as you won't have enough gas to invoke all 10-15 injected gr_sends. What I want to point is that choosing the injections range must be done in some context of the amount of max_instructions, Control and min/max_functions for each (bunch) of syscalls.
Second, I think would be nice here to change not only the injection rates randomly, but the bunch of syscalls to be injected. I mean, sometimes having some syscalls being injected 0 times (i.e., not injected). Some of them aren't injected more frequently, then the others. Say, you need gr_send in 90% of executions.
Problem to Solve
At the moment fuzzer config and what we pass to
wasm-smithfor WASM generation is mostly static, this does not allow us to get fully randomized programs that cover wide range of possible usages of Gear API and potentially lowers the coverage percentage we might actually get.Possible Solution
StandardGearWasmConfigsBundleandSelectableParamsinwasm-gen. Below is the observations based on running fuzzer multiple days with tuning each of configs possible.Notes
What to not randomize
max_funcs: almost never results in increasing coverage or in rate of successful terminations. It also increases the total runtime for a single generated program. In 5 minutes on my i5 13400f Fuzzer is able to run only about ~50 programs total.min_funcs: very similar tomax_funcsin terms of what happens to coverage and program amounts. Not really worth it. Lowering the value helps to get more programs to run but overall not a big increase: with all configs set to default (min_funcs: 3, max_funcs: 5) I am getting ~700 programs ran in 5 minutes and 83% of them are successful while other 17% are terminated for various reasons: 80% of terminations come from gas insufficient error. In case of increasing themax_funcswe ran out of gas more often and amount of successfully terminated programs shifts to 30% and total amount of programs executed also drops from 700 to 50 in case ofmax_funcs: 20What's worth randomizing
max_instructions: This value highly impacts on how many programs we can run under limited time-frame: the larger it is the less programs we can run as more code == more gas consumption. Plus if we enableControlthis has even larger impact on gas. My proposal here is to randomize this value from in between 100-800 instructions:>600 and <900range is good enough to not significantly lower the amount of programs ran and also does not make programs ran out of gas noticeable faster.>100 and <500range is good for running small quick programs withControlinstructions enabled as it does not use gas so quicklyAdding or removing
Controlinstructions to WASM programs: Control instructions do have a high impact on runtime of a programs and increase coverage significantly: based on pallet/gear I can see decrease of coverage withot control to about ~50-60% while with control it's up to 90% for gear pallet. My suggestion is that it's worth to enable control instructions in case of<= 500max_instructionswhich would not result in program runtime increase and also does not change rate of successes/traps in generated programs.Random range of
SyscallInjectionTypesinStandardGearWasmConfigsBundle: by default we generate one of each syscalls, with increase of this range to 1..=10 more coverage was seen in pallets plus more branches and more state transitions should happen which should test our runtime more thoroughly.What to increase
I think it's worth to also calculate allowed gas for each program that we generate based on estimation through config and total instructions in the program so we can run the generated programs until completion more often rather than failing because we exceeded our gas limit.