search

bbc / wraith

Wraith — A responsive screenshot comparison tool
http://bbc-news.github.io/wraith/
Apache License 2.0
4.84k stars 359 forks source link

Wraith history mode issues after changing screen_widths in the config #409

Open katekligman opened 8 years ago

katekligman commented 8 years ago

I often suggest Wraith to folks who are new to visual regression testing. I've noticed that when new users experiment with history mode they often run into a couple of similar issues when editing the screen_widths: parameter in their configs. Here are two use cases and steps to reproduce the issues with Wraith 3.1.2 and the default configs.

  1. edit configs/history.yaml to remove the before_capture: from clickable_guide__after_click
  2. run wraith history configs/history.yaml
  3. run wraith latest configs/history.yaml
  4. (all is usually well at this point)
  5. edit configs/history.yaml to remove 320 from screen_widths
  6. run wraith history configs/history.yaml
  7. run wraith latest configs/history.yaml

Wraith will error out with:

GENERATING GALLERY /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/gallery.rb:114:in `each': comparison of NilClass with 0.0 failed (ArgumentError)

Here's slightly different error that happens when the default configs/capture.yaml is used:

  1. add history_dir to the default configs/capture.yaml and set a single domain to http://www.google.com
  2. run wraith history configs/capture.yaml
  3. run wraith latest configs/capture.yaml
  4. (all is usually well at this point)
  5. edit configs/capture.yaml to remove 320 from screen_widths
  6. run wraith history configs/capture.yaml
  7. run wraith latest configs/capture.yaml

Wraith will error out with:

CROPPING IMAGES /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/crop.rb:48:in `initialize': no implicit conversion of nil into String (TypeError)

It seems that the stale 320* files in the history_dir are the cause of the issues. I think that Wraith should be a little smarter if 'wraith history' leaves old baseline files hanging around. :)

ChrisBAshton commented 8 years ago

Hmm, thanks for this. We solved a similar issue with adding/removing paths: #293. I don't think we've ever changed the screen sizes after the initial choice so have not run into this before, but yes, Wraith ought to be able to cope with this!

renettarenula commented 8 years ago

I am also receiving this issue but I did not remove 320 from screen_widths at all. I got this error when I use PhantomJS and remove the clickable_guide and clickable_guide__after_click:

CROPPING IMAGES /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/crop.rb:48:in `initialize': no implicit conversion of nil into String (TypeError)

And I got this error when I am using the script (history.yml) as is:

GENERATING GALLERY /usr/local/lib/ruby/gems/2.1.0/gems/wraith-3.1.2/lib/wraith/gallery.rb:114:in `each': comparison of NilClass with 0.0 failed (ArgumentError)

Perhaps there is more to it than changing widths?

patricknelson commented 5 years ago

There's another situation where exactly the same error comes up (regarding crop.rb): If you're performing a latest against a history that happened to have a different list of paths. In my case, I had a sort of complicated setup (which was also CPU intensive) that made it tough to track down:

  1. Checkout master
  2. Run front-end & back-end builds
  3. Run wraith history config.yml
  4. Checkout feature-branch
  5. Run front-end & back-end builds
  6. Run wraith history config.yml <-- 💣💥

Irritating that it'd die on that last step with no explanation for why until I went through all 54 directories (yeah, I have that many VQA regression tests 😄) and found the one that was missing. I keep my config.yml file checked into git, so since that differed, wraith was exploding.