Improve handling of multiple images

[birdsarah]

I'm using pytest-html with pytest-selenium. My selenium tests are doing an image diff.

I want to include the results of a failed image diff test in my pytest-html report.

However, if I just use pytest_html.extras.append(...) then my output gets mixed up with pytest-selenium's output and makes for a very incoherent report e.g.

Full report at: https://s3.amazonaws.com/bokeh-travis/112986254/tests/pytest-report.html

I have hacked around this at the moment by overriding longrepr and extras completely so that I get something like this:

Full report at: https://s3.amazonaws.com/bokeh-travis/112997379/tests/pytest-report.html

What I'd like is a hook to be able to add another row of my own custom test output.

[The-Compiler]

This is slightly offtopic, but do you have a blogpost or something (or a project link, if it's some library) about how you generate those diffs? I plan something similar for qutebrowser's tests, so that'd be very helpful!

[birdsarah]

@The-Compiler - here's the PR: https://github.com/bokeh/bokeh/pull/3975/files

It's very early days, I'm expecting some work figuring out what make tolerant diff tests, and to have to tweak with the perceptualdiff parameters a fair amount.

[davehunt]

@birdsarah would it work for you if the report had improved support for multiple images? It's not something I've ever played with, but it feels like we should present these images as a gallery, possibly with the ability to provide captions and to indicate which image should be displayed on top. From your full report, it appears that both Diff and Base links are also showing the same image.

[birdsarah]

@birdsarah would it work for you if the report had improved support for multiple images? It's not something I've ever played with, but it feels like we should present these images as a gallery, possibly with the ability to provide captions and to indicate which image should be displayed on top.

@davehunt, that sounds great, I was just thinking of something simple and generic, and not too customized to my use case. I think the big challenge is running out of space on that row that already gets populated with an auto-generated screenshot, a video, and a log.

From your full report, it appears that both Diff and Base links are also showing the same image.

Yes, you'll notice in my hacked report, I just use " " as the link text to hide them.

[birdsarah]

From your full report, it appears that both Diff and Base links are also showing the same image.

Yes, you'll notice in my hacked report, I just use " " as the link text to hide them.

For what it's worth I see no value in the image link feature. Browsers already let you get access to image links if you really need them. I do like the idea of supporting externally linked images though. That's how our examples test report works: https://s3.amazonaws.com/bokeh-travis/113026629/examples.html - this would be far too big an html report to handle with b64-embedded images gracefully.

[davehunt]

I think the big challenge is running out of space on that row that already gets populated with an auto-generated screenshot, a video, and a log.

Yeah, I'd by default have the screenshot as the key image in the gallery if there's more than one, so we don't use up any additional space. Not sure how best to handle if JavaScript is disabled though - perhaps screen width would have to be a sacrifice there.

I do like the idea of supporting externally linked images though. That's how our examples test report works: https://s3.amazonaws.com/bokeh-travis/113026629/examples.html - this would be far too big an html report to handle with b64-embedded images gracefully.

This is planned as part of #32

[kdexd]

Hello @davehunt @birdsarah, I have read the conversation here, before this whole support is worked upon, we can have a quick support for multiple images like this:

I am using this plugin for TARDIS, we are uploading our reports like this one here.

We attach plots with each test, sometimes we require multiple plots embedded at times.

To display it this way, I passed a list of base64 encoded strings of images through pytest_runtest_makereport hookwrapper, instead of a single string. The images are displayed from top to bottom according to order of the list.

(I had mailed you about the implementation discussion, but I somehow managed to do it.) Shall I make a PR for it ? It would be great if this feature is added for now and version is bumped to 1.8.2 or something. I can even help in extending this feature of multiple images in the lines of above discussion. Cheers ! :grinning:

NOTE: The images in screenshot are dummy, and kept same intentionally. Different encoded base64 strings can be passed in the list.

[davehunt]

@karandesai-96 sorry for the late response - I think a patch that makes displaying of multiple images better would be great. I'm not sure I understand your approach, but I'd be happy to discuss it in a pull request.

[kdexd]

@davehunt That's great. I'd do that this weekend !

[davehunt]

@karandesai-96 are you still interested in working on this?

[kdexd]

@davehunt Yes I am, but I am through final weeks of GSoC project, hence might not be able to work on it till 15 August. I'll open a PR soon after my GSoC is done. Cheers !

[davehunt]

Awesome, thanks! Good luck with your project.

Dave Hunt

On 29 Jul 2016, at 21:07, Karan Desai notifications@github.com wrote:

@davehunt Yes I am, but I am through final weeks of GSoC project, hence might not be able to work on it till 15 August. I'll open a PR soon after my GSoC is done. Cheers !

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[kdexd]

Hi @davehunt: Finally my project has concluded. The source code here has moved pretty fast, it'll take time to analyze and re implement my logic in this new setting. I'm letting you know that I'll be starting to address this issue now. Cheers ! :smile:

[kdexd]

I managed to take out some time for this issue, a lot had changed in the code but I finally came to this:

[davehunt]

Hey @karandesai-96 that looks great! I think it would be great to present multiple images in a gallery style, but that can easily be split out into a separate issue.

[kdexd]

@davehunt Yes I agree with you, gallery style display will look great, and developing things incrementally also makes sense. Once we have a working code in place we can certainly extend it further. I'll try to stick around with it and enhance it as well !

[fenchu]

Hi Was this code ever published?, How do I get my images under each other so it does not eat up the console output?

[BeyondEvil]

@fenchu Nope.

There's an old PR you could look at it if you wanted to: https://github.com/pytest-dev/pytest-html/pull/112

[codeturtle00]

For @fenchu and whoever else that lands on this page like me. A possible workaround is using extras.html, which will place images above your log. This way you can also add captions to your images and I imagine with some effort you can hack up a slideshow/gallery.

A very simple example:

for img_str in base64_screenshots:
    extra.append(pytest_html.extras.html(f'<img src="data:image/png;base64,{img_str}" style="max-width:100%;">'))

Alternatively you can also use Pillow to concatenate images to create something like kdexd's report