OwlEye

A tool to automatically detect and localize UI display issues in the screenshots of the application under test.

We give dataset (./Dataset/), UI display issues detection and localization methods (./OwlEye/), data augmentation methods (./DataAug/).

Dataset (./Dataset/)

The experimental dataset comes from two sources. The first is the screenshots from crowdtesting, which contains 4,470 non-duplicate screenshots with UI display issues and equal number of bug-free non-duplicate screenshots.The second is the screenshots generated with the data augmentation method.

TrainData

./Dataset/traindata/ is the training dataset of the experiment, with 21,980 images.

• bug-crowd.rar : 3,170 screenshots with UI display issues from the crowdtesting dataset.
• bug-aug.rar : 7,820 screenshots with UI display issues generated by data augmentation method.
• normal-crowd.rar : 3,170 screenshots without UI display issues from the crowdtesting dataset.
• normal-aug.rar : 7,820 screenshots without UI display issues generated by data augmentation method.

TestData

./Dataset/test.rar is the testing dataset of the experiment, with 1,600 images.

• 1.jpg ~ 800.jpg : 800 screenshots with UI display issues from the crowdtesting dataset.
• 1001.jpg ~ 1800.jpg : 800 screenshots without UI display issues from the crowdtesting dataset.

ValData

./Dataset/val.rar is the Val dataset of the experiment, with 1,000 images.

• 1.jpg ~ 500.jpg : 500 screenshots with UI display issues from the crowdtesting dataset.
• 1001.jpg ~ 1500.jpg : 500 screenshots without UI display issues from the crowdtesting dataset.

Issues Detection and Localization Approach (./OwlEye/)

OwlEye is named considering it is like the owl's eye to effectively spot UI display issues, also considering this nocturnal animal serves as the supplement of traditional automated GUI testing. Automatically detect and localize UI display issues based on the screenshots of the mobile apps under test.

• getdata.py : Preprocessing dataset
• network.py : Neural network structure of the model
• train.py : Training model
• test.py : Testing model
• val.py : Validating the model
• localization.py : UI display issues localization methods
• ./data/ : Decompress the dataset and put it into their respective folders (./train/ & ./test/ & ./val/)
• ./model/ : Model file
• ./record/ : Training record
• ./input_pic/ : Screenshot to locate
• ./output_pic/ : Localization result

Heuristic-based Data Augmentation (./DataAug/)

It is a heuristics-based training data augmentation method which can automatically generate screenshots of UI display issues with normal screenshots.

• DataAug.py : The heuristics-based training data augmentation method
• ./picture/ : Normal screenshots
• ./json/ : Run-time view hierarchy of screenshots
• ./IMG/ : Pre-prepared image icon
• ./output/ : Generated screenshots with UI display issues

Confirmed or fixed issues (RQ3)

We evaluate the usefulness of our model by applying our OwlEye in detecting the UI display issues in the real-world apps from Google Play and F-Droid. Among 2,200 apps, we find that 56 of them are with UI display issues. We issued bug reports to the development team and 25 are confirmed and fixed by developers.