search

garris / BackstopJS

Catch CSS curve balls.
http://backstopjs.org
MIT License
6.82k stars 607 forks source link

Sauce lab integration #223

Open nobso opened 8 years ago

nobso commented 8 years ago

@garris What do you think about integrating with Sauce labs for running the tests?

garris commented 8 years ago

I think that sounds great!

nobso commented 8 years ago

Cool, i will work on it soon!

nobso commented 8 years ago

Can you please change the issue label to enhancement ?

NimaSoroush commented 7 years ago

@nobso : What is the status on this issue?

adamorlowskipoland commented 7 years ago

@nobso I follow up the @NimaSoroush question. Can we count on it?

nobso commented 7 years ago

Hi @garris I'm sorry that i couldn't do that as i'm tied up with other things now. I'm ready to help if someone starts it.

garris commented 7 years ago

No problem @nobso -- thanks again for your help on other features! They really had a positive impact.

Sauce labs integration would be great if someone wants to start looking into it.

greaveselliott commented 6 years ago

Did work on this integration ever start?

garris commented 6 years ago

Not that I know of.

nsp27 commented 5 years ago

Is this being worked on? connecting backstopjs with saucelabs

garris commented 5 years ago

I am pretty sure this is not being worked on by anyone at the moment. At least nothing I am aware of. I am happy to support anyone who wants to work on it. Cheers.

Craignewkirk commented 5 years ago

Hopefully saving some headaches here.

We were using Backstop for visual regression testing and then moved to Gemini for testing in Sauce Labs.

It wasn’t great.

Due to being in the cloud, there was no guarantee that the hardware the images were rendered on initially would be the same hardware that the tests would be run on.

As a result, our tests would never pass. Turns out different graphics cards render fonts differently.

This rendering caused a ton of small shifts in the testing screenshot.

We tried lowering the threshhold, however to get it low enough that the tests passed made visual regression testing almost pointless for our purposes (developing a front end framework).

This was last year, so maybe Sauce now has developed a way (or already had a way I didn’t find) to guarantee your tests run on the same hardware config if not the same machine.

Just putting this out there if someone takes this up. We loved Backstop. The only reason we switched was for the Sauce Labs integration.

Bonus info: apparently advertisers have used the minute changes in font renderings to detect people’s machine specs.

garris commented 5 years ago

@Craignewkirk I am very curious -- how do you use Sauce for your FE framework? What is your workflow? And what are you testing for? Are you just visual diffing -- or are you also testing that your JS is running everywhere etc?