Baseline images and deleting previous runs

[jasonmacdonald]

I noticed that if you delete a previous run from the UI, and that run contains an image saved as baseline, it resets the baseline to an undetermined (not yet set) state. Would it be possible to not link the accepted baseline to a particular (dated) entry so if people delete a previous run to clean up the UI, the baseline is still saved?

[skwakman]

Ouch, good find. I've managed to reproduced this, however I saw some different behavior.

After I deleted a previous run which contained a baseline image, suddenly all other runs no longer show the images that were added to the baseline during that deleted run. This shouldn't happen.

However, the baseline itself was not actually gone. After I executed a new run, it was still being compared against the previously made baseline.

It seems that somewhere during deletion of a run, relations to baseline images that were created in the run are being removed as well. I'm looking into a fix.

[skwakman]

Small update in regards to this issue: I've got a potential fix ready on a local repository. Later this week I'll do some additional testing and, if all goes well, publish the fix.

[jasonmacdonald]

Thanks for the update. Any progress? Cheers!

[skwakman]

Sorry this took a little while longer. The fix required a database change for which we didn't have any proper tests yet. While I'm still finalizing these tests, you might already try out the fix. I've uploaded a fixed version here: https://drive.google.com/file/d/0B3A9v168KLhFckRrZS1Xd1U3QTA/view?usp=sharing .

Before trying out this release, make sure you read the wiki page on how to upgrade without losing your data (see here: https://github.com/xebia/VisualReview/wiki/Upgrading-VisualReview).

If this fix works out for you, I'll make it a new release.

[jasonmacdonald]

Thanks slwakman, it seems to remember the baseline now.

On a slightly related topic, if an entry is deleted form the UI, is there a reason the PNG's still stay in the sceenshots folder. This thing gets massive if you have 1000's of tests in a run, and run that daily. It seems there's no clean way to delete all unneeded files despite the baseline being saved, I assume, in the DB. If you delete the images you start getting java error's even if the UI is completely emptied of previous runs.

[jasonmacdonald]

As a thought, what would you think about separating baseline images from the screenshots directory for a run? This could allow you to check-in baseline images to source repo while being able to delete old runs that don't matter anymore (including all images). I find I'm constantly deleting stuff from the UI, but it doesn't clean-up the images. Also, if oyu delete a spec entirely to clean up the ui, you also delete the baselines for it. This means I have to manually delete each entry in each spec to avoid deleting the baseline.

I also noticed that when you have multiple browser tests, every run shows up in the UI as a line, but only the last contains all the images.

Sorry, I know I'm complaining a lot, but just wanna help make things easier to manage. Wish I knew clojure but I'm a JavaScript guy :)

[skwakman]

I've closed this bug for the original issue for which is now fixed in master, ready for the next release.

About your other comments:

On a slightly related topic, if an entry is deleted form the UI, is there a reason the PNG's still stay in the sceenshots folder. This thing gets massive if you have 1000's of tests in a run, and run that daily.

VisualReview's current version does not offer any image delete or cleanup functionality, something I agree is required at this point. I've already started work on this in another branch, I expect to finish it somewhere next week. To track this new feature, see #56

The reason why the images are not deleted with runs: we want to separate image-storage from run/screenshot metadata so we can re-use certain images for runs that generate the same image data. For example: consider a web page that's being screenshotted in every run and always returns the same image. Screenshots of this page in all runs might as well point to the same image file, saving up storage space. This optimization however has not yet been implemented and is slated for a future version. However, the storage system for it is already in place.

I also noticed that when you have multiple browser tests, every run shows up in the UI as a line, but only the last contains all the images.

This sounds like a separate bug. Could you post a new issue with some more details on what's going on? Perhaps with a screenshot showing the problem?

Sorry, I know I'm complaining a lot, but just wanna help make things easier to manage.

Hehe no problem at all. This tool is certainly not feature-complete yet so it is to be expected to run into some missing pieces. I'm actually grateful for your feedback, it gives us some focus on what to work on next.

[skwakman]

@jasonmacdonald In PR #58 there's a link to a build where image files are deleted when runs/suites/projects are deleted. You could test it if you'd like before it ends up in a new release.

[jasonmacdonald]

@skwakman Great. I'll give it a shot and let you know