Automated "blackbox" testing of the GUI

darosior commented 10 months ago

@edouardparis Could it be possible to start the Iced application inside a unit test, and send signals to it to simulate a user clicking on a button? This would save us a great amount of time.

Related to #712.

edouardparis commented 10 months ago

There is potential work to be done on this by passing list of messages in the update flow, it will help to test state logic but not view logic as buttons may be displayed according to context. In order everything with a full automated test we could use something like: https://stackoverflow.com/questions/73174957/rust-how-to-interact-with-a-window

pythcoiner commented 10 months ago

@edouardparis Could it be possible to start the Iced application inside a unit test, and send signals to it to simulate a user clicking on a button? This would save us a great amount of time.

Related to #712.

I heard about PyAutoGUI for that kind purpose, maybe it worth have a try... edit: using in combination with opencv + tesseract it appears to be not so hard to build a feature for detecting button positions given the button name

darosior commented 10 months ago

That seems potentially useful, could be interesting to explore further. One thing i'm skeptical of is the maintainability. How about waiting times? Can it detect that a button isn't available just yet before pressing it would it need to rely on fixed timeouts?

pythcoiner commented 10 months ago

i'll try to play with that when i got some time, i've some way in mind how to use it.

pythcoiner commented 5 months ago

maybe we can also have a look to a combination of these 3 crates:

pythcoiner commented 5 months ago

I had a check today about this, detect buttons/labels positions by their text is doable using:

xwininfo to detect the window position by its name
rusty-tesseract for OCR
autopilot for screenshot + interaction (key/mouse press/ mouse scroll / keyboard input/ moving cursor)

here some result of window detection + screenshot + OCR detection:

home

output

send

pythcoiner commented 5 months ago

i think if we want do something like this we should add some features to GUI:

Logging or output (to a named pipe?) updates messages, at least for some events/updates
Add a theme that we can set by an ENV var on gui lauch , in order to allow us to detect more easily some widget (input fields, check boxes...)

pythcoiner commented 5 months ago

you can have a look there on the (RUST) test framework prototype i've made

pythcoiner commented 3 months ago

work a bit on this today, i was getting some bad result doing directly OCR w/ tesseract on the raw screenshot so i changed a bit the flow:

implement a custom Liana theme (use an ENV var on liana-gui call), so it became easy to differenciate widgets by their kind:

use OpenCV to do some image processing and select some POI area (I used here a cli tool i made using opencv c++ lib as i don't like APIs of currently availables rust bindings)

this binary output JSON data that will be quite easy to parse:

{ "status": "OK", "items": [ { "color": { "r": 254, "g": 167, "b": 0 }, "top": 696, "bottom": 734, "left": 871, "right": 891 }, { "color": { "r": 226, "g": 78, "b": 27 }, "top": 697, "bottom": 741, "left": 748, "right": 768 }, { "color": { "r": 0, "g": 255, "b": 0 }, "top": 546, "bottom": 670, "left": 350, "right": 371 }, { "color": { "r": 0, "g": 255, "b": 0 }, "top": 406, "bottom": 444, "left": 886, "right": 906 }, { "color": { "r": 0, "g": 255, "b": 0 }, "top": 406, "bottom": 513, "left": 404, "right": 424 }, { "color": { "r": 0, "g": 255, "b": 0 }, "top": 356, "bottom": 461, "left": 405, "right": 425 }, { "color": { "r": 0, "g": 255, "b": 0 }, "top": 306, "bottom": 370, "left": 404, "right": 424 }, { "color": { "r": 127, "g": 0, "b": 127 }, "top": 486, "bottom": 608, "left": 812, "right": 835 }, { "color": { "r": 255, "g": 105, "b": 180 }, "top": 772, "bottom": 836, "left": 43, "right": 65 }, { "color": { "r": 255, "g": 105, "b": 180 }, "top": 442, "bottom": 493, "left": 44, "right": 65 }, { "color": { "r": 255, "g": 105, "b": 180 }, "top": 392, "bottom": 489, "left": 43, "right": 63 }, { "color": { "r": 255, "g": 105, "b": 180 }, "top": 343, "bottom": 389, "left": 43, "right": 62 }, { "color": { "r": 255, "g": 105, "b": 180 }, "top": 292, "bottom": 352, "left": 44, "right": 65 }, { "color": { "r": 255, "g": 105, "b": 180 }, "top": 195, "bottom": 243, "left": 44, "right": 60 }, { "color": { "r": 228, "g": 171, "b": 183 }, "top": 242, "bottom": 285, "left": 43, "right": 63 } ] }

pythcoiner commented 2 months ago

Here notes from discord for record:

idea it's call liana w/ a ENV VAR to set the custom theme (custom theme use a different color for each different kind of widget (Primary buttons, Secondary buttons, menu buttons, selected menu buttons, text inputs, check boxes, etc...) then control liana by by a programatic way, (maybe also control a bitcoind regtest) something like:

let mut screen = Screeenshot::new(); 
assert_eq!(screen.selected_menu, "Send");
assert!(screen.text_inputs.contains("Address");
screen.text_inputs.select("Address");
screen.keyboard_input("bc1................");
screen.text_inputs.select("Payment label");
screen.keyboard_input("Maria paycheck");
screen.text_inputs.select("0.001 (in BTC)");
screen.keyboard_input("Maria paycheck");
screen.text_inputs.select("42 (in sats/vbyte)");
screen.keyboard_input("100");
screen.buttons.click("Next");

sleep(2.0);

let mut screen = Screenshot::new();
assert_eq!(screen.buttons.contains("Sign");
screen.buttons.click("Sign");

// etc....

What Screenshot::new() should do under the hood:

detect Liana window position & size
screenshot the liana window
do some image processing in order to detect Items/Wigets positions/bounding boxes by their label color, we then extract such cropped images knowing their positions & widget type:

do some OCR to extract the label of each item
sorts items by their type into the Screenshot struct into HashMaps

pythcoiner commented 2 months ago

having a lot issues getting a good OCR result using tesseract on cropped images, i give a try to ocrs, that give very good result (even spaces, special chars and case are very well detected), also looks faster then tesseract and it is built in rust.

pythcoiner commented 1 month ago

note to myself:

I think what's worth automating is the "trivial" tests. Usually when a release comes out we extensively test the new features but we gloss over the existing trivialities. If we could consistently test them in an automated manner it would greatly improve our Q&A.
Already just with the send panel you've got a lot of trivial tests to automate:
I can send to any type of addresses;
I can send to one or multiple outputs;
I can spend one or more coins;
If none are selected, coins get automatically selected;
I can't set an insane feerate;
I can send my whole balance;
I can send my remaining balance in one of the outputs;
If i don't a change output is created;
All of the above combined, lol

wizardsardine / liana

Automated "blackbox" testing of the GUI #713