TST: Use matplotlib's compare_images to check plots

[jtratner]

With #5378 - could be a good time to start comparing against images for these plotting tests... matplotlib has a test function and GeoPandas is working on it here - https://github.com/kjordahl/geopandas/pull/43, so we could work off of their example, maybe a decorator that handles ensure_clean and comparisons...

[jankatins]

ggplot has some similar code which can be used similar to matplotlibs @image_comparisondecorator or via a assert_same_ggplot function: https://github.com/yhat/ggplot/blob/master/ggplot/tests/__init__.py

Testing is very easy:

from . import get_assert_same_ggplot, cleanup
assert_same_ggplot = get_assert_same_ggplot(__file__)

@cleanup
def test_geom_density():
    df = _build_testing_df()
    gg = ggplot(aes(x="x", color="c"), data=df)
    gg = gg + geom_density() + xlab("x label") + ylab("y label")
    assert_same_ggplot(gg, "geom_density")

or via the decorator:

from . import image_comparison

@image_comparison(baseline_images=['theme_mpl_all_before', 'theme_mpl_all_after'])
def test_theme_matplotlib4():
    gg = ggplot(aes(x='date', y='beef'), data=meat) + \
        geom_point(color='lightblue') + \
        stat_smooth(span=.15, color='black', se=True) + \
        ggtitle("Beef: It's What's for Dinner") + \
        xlab("Date") + \
        ylab("Head of Cattle Slaughtered")
    print(gg + theme_matplotlib())
    print(gg+theme_matplotlib({"font.family": "serif"}, matplotlib_defaults=False))

It will compare the current images to once which are in <dir where the tests are>/baseline_images/<testfile without ".py">/. Running the tests will copy the baseline images to a "result_images" dir and adding the current images and comparing them. Generating baseline images can be done via running the tests and copying the images from result_images/test_xyz to .../baseline_images/test_xyz.

The fork of the decorator was needed for two reasons: using it outside of matplotlib can only be done after matplotlib 1.1.1 (which we test against) and we also wanted to change some antialising settings which were hardcoded in matplotlib. I also changed the code to only produce/compare png images (https://github.com/yhat/ggplot/commit/901f049863f02cc6bdc64074fe99abd6cac0eb3f).

ggplot also has a https://github.com/yhat/ggplot/blob/master/visual_tests.py which generates a browser page which can be used to visually compare the images.

[ryankarlos]

I would be interested in this - if this is still requiring a PR ? If so, do all the plotting tests need to be dealt with in this PR itself or split into different ones ?

Is there a preference for using @compare_images over @image_comparison from the matpoltlib testing suite ?

[jbrockmendel]

I tried using this for a personal project 5ish years ago and found that I got very slightly different images on different machines (mac vs ubuntu), python versions, dependency versions, ... In the end I threw in the towel on it. I'd be happy to find I was doing something wrong.