search

marco-c / autowebcompat

Automatically detect web compatibility issues
Mozilla Public License 2.0
35 stars 41 forks source link

Get full page screenshot #227

Closed sagarvijaygupta closed 6 years ago

sagarvijaygupta commented 6 years ago

We need to get a full page screenshot for the web page open. In some cases it will be required to scroll horizontally also. We should discuss the naming convention and also what should be written in the corresponding file.

Shashi456 commented 6 years ago

What about #184 , isn't consistency of more importance right now ?

What do you mean by a full page screenshot , do you mean a scrolling screenshot? As you pointed out there's multiple cases here you could either have vertical scrolling or horizontal

sagarvijaygupta commented 6 years ago

@Shashi456 #184 is definitely important but it will take some time to get to work on it. I don't have much experience with Android Emulator, but I think for that we will have to write the complete crawler script again.

What I mean by a full page screenshot is following:

  1. Take a screenshot. Name it something like BUGID_ssNumber_HS0_VS0_BROWSER.png
  2. Scroll by a fixed amount (say HShift) horizontally (if possible). Name the screenshot BUGID_ssNumber_HSHShift_VS0_BROWSER.png.
  3. Continue horizontally, else shift to vertically. Name it BUGID_ssNumber_HS0_VSVShift_BROWSER.png.
  4. Traverse the whole page as a matrix of screenshots.

Thats what I have in mind. We can think about other ideas also. This will help in verifying #226

marco-c commented 6 years ago

Sounds good to me. The additional screenshots can also be used for training. One thing to handle will be that each website will have a different number of screenshots. While we can handle each screenshot as a different element, using the additional information that some screenshots belong to the same site could improve accuracy (it's the same with the screenshots taken after interacting with the page).

sagarvijaygupta commented 6 years ago

@marco-c with this addition we will can discuss following and I will start implementing them one by one (whichever we agree upon)

  1. Since we have a complete page screenshots, we should focus on webpage compatibility rather than screenshot-screenshot compatibility. We can add that these screenshots belong to same page (as you suggested). What I mean is that consider a webpage is loaded with an advertisement of different size in both browsers, so scrolling might seem that they are not compatible but we know they are. So we should mark them different yet compatible.
  2. Since screenshots now load in a sequence, I think we can consider adding criteria that screenshots belong to same bug_id (along with same webpage).
  3. Adding the shift (horizontal and vertical) to training for every screenshot can also be considered, this could also improve accuracy.

Example for first (though sign in is an incompatibiliy)! screenshot from 2018-07-09 18-13-02

marco-c commented 6 years ago

For 1 we don't have to do much, just take it into account while labeling, right? 2 is being done now with your latest PR. 3 would be interesting, but I'm not sure yet how to handle that. Can you file a follow-up issue for it?

I think we can now close this issue.

sagarvijaygupta commented 6 years ago

For 1 we have to take it into account in labeling and in training also. I am not sure how to handle it but it will be important that models learn that there is a relation between two screenshots of same webpage and that is why we are marking them compatible even if they look different corresponding to each other, but when stitched together they look totally same beyond that advertisement point. Okay I will file a follow-up issue.

I will close it as it is solved by #228