cloudhead / rx

👾 Modern and minimalist pixel editor
https://discord.gg/xHggPjfsS9
GNU General Public License v3.0
3.09k stars 109 forks source link

Tests #59

Open cloudhead opened 4 years ago

cloudhead commented 4 years ago
cloudhead commented 4 years ago

@pop if anything above looks interesting, let me know and I can provide more detail. Would be great to have some help in making rx more robust.

pop commented 4 years ago

If I had to pick one, fuzz-testing the commands that take user input seems interesting. i.e.,

session.rs: all commands that take user input should be tested with unexpected inputs to ensure there is no division by zero for example

That said I have not dug too far into any of the topics so if there is something else you'd like to pass off to a contributor I'm open to one of the other topics.

I'm not entirely sure how to do headless testing but I'm sure once I get started writing the headless fuzz-tests would be pretty straight forward.

cloudhead commented 4 years ago

Great, I also think the fuzz testing will be the biggest ROI at this point, but it's also possibly more work than the others.

A good place to start looking is the execution module and the current tests in tests/, as there's already the ability to run rx in headless mode and just feed it inputs from a script. There's two levels at which the fuzz testing could be useful:

  1. Create random Command objects with random parameters and feed them into the Session and try to make it crash.
  2. Send random keyboard/mouse inputs directly to the session.

Both might be useful, but 2. is less likely to trigger complex behavior, since most inputs won't be valid commands.

Another thing that might be helpful alongside this is to add another input which is the current session mode, so run the fuzz tests in Visual, Normal and Command mode; I think this will increase coverage quite a bit.

Finally there's the question of whether to turn the renderer and platform layer off or not. I think to start out it would make sense, ie. to use a dummy renderer (trait Renderer) and context manager (platform). This way the fuzz tests can run 100x faster and test many more inputs.

These are my initial thoughts.