Huxley failing immediately after --record (on a test app) and on any running of tests in example app

[igreulich]

I am trying to get Huxley up and running so I can evaluate it, and see if it fits my needs. Unfortunately I can't.

I installed Huxley, pip install huxley.

I did have to install all the dependencies manually, pip install jsonpickle, pip install plac, pip install selenium, pip install PIL.

Example App

I navigate to /examples/webroot/ and fire up the example app, python -m SimpleHTTPServer.

When I run the tests, huxley, I get the following:

Fail 1 Gist

Test App

My Huxleyfile looks like this;

[index]
url=http://localhost:3000/

I navigate to my test app's huxley directory, and try and record, huxley --record.

First, I get 2 Firefox windows that open;

After I have clicked around, and made my screen shots, when huxley begins re-running my actions, the empty window closes;

Then the whole thing fails;

Fail 2 Gist

I really love the idea of Huxley, and want to use it, but this is keeping me from evaluating it. Any thoughts?

[chenglou]

Hey, to see the difference between screenshots, pass the --save-diff flag. The difference will be highlighted.

Most of the times this is due to the window focus. For example, if you had your input focused, the blue input aura will be captured. It isn't displayed the same way for an unfocused window.

[igreulich]

So I ran huxley toggle --save-diff on the included example app.

Huxley Toggle Fail

diff.png

last.png

I think maybe Huxley is recording with the button's hover state active?

Regardless, this time, for some reason, the test ran w/o errors. However, I still cannot get Huxley to --record successfully (or --save-diff). I get a similar error as before:

Huxley Record Fail

I tried running huxley just to see if it did record, and I get ValueError: No JSON object could be decoded. I presume that is because Huxley didn't successfully record.

[petehunt]

So hover states should be normalized away because we don't actually use the screenshots from your record session. We replay the log of actions you performed in an automated way and take screenshots without actual user interaction.

With that said it may be a browser window focus issue. I've seen this on a few platforms. Try making sure the window isn't focused when running Huxley after you record. What I do at Instagram is run the records on my local machine and all playback on a remote webdriver server.

What platform are you on? I wonder if there is a way we could fix this in a platform-independent way, or at least throw a warning in there.

[igreulich]

I am on OS X 10.8.5.

On my test app (one I created in rails) huxley doesn't finish a huxley --record. The Huxley Record Gist shows the output.

While running huxley --record I do the following;

Note: Selenuim and my test app are already running.

• huxley --record
• (Selenium opens 2 Firefox windows, a blank one, and one containing my test app)
• Click the Huxley terminal (might not be strictly necessary)
• Press return to record initial screenshot.
• Click the browser window with my test app.
• Click a button one time (that changes color, same as the example app included in Huxley)
• Click the Huxley terminal.
• Press return, to record second screenshot.
• Click the browser window with my test app.
• Click the aforementioned button once.
• Click the Huxley terminal.
• Press return, to record final screen shot.
• Press q, then return to quit.
• Wait for Huxley to rerun the recorded actions.
• Watch Huxley fail.

I can't even get to a point where I could run the playback on a remote webdriver.

[chenglou]

@petehunt I was thinking of opening a second small window during the playback so that it automatically unfocuses the first one, what do you think?

@igreulich do you touch anything after the before-last step? Like petehunt said, the screenshots you captured during the recording phase don't do anything actually. The playback is e first real recording and the verification is where it fails. Try, during these two steps, to manually focus the window both times. If that doesn't work, unfocus me both times.

[igreulich]

@chenglou No, I don't change focus from the Huxley terminal. Once I press q, then return, I am hands off.

However, I just tried huxley --record and it worked. Here is what I did this time;

Note: Selenuim and my test app are already running.

• huxley --record
• (Selenium opens 2 Firefox windows, a blank one, and one containing my test app)
• Click a button four times (that toggles the button color, same as the example app included in Huxley)
• Click the Huxley terminal.
• Press q, then return to quit.
• Wait for Huxley to rerun the recorded actions.
• Watch Huxley succeed.

I then ran huxley, and it succeeded.

I then ran the following, that also succeeded;

Note: Selenuim and my test app are already running.

• huxley --record
• (Selenium opens 2 Firefox windows, a blank one, and one containing my test app)
• Click a button one times (that toggles the button color, same as the example app included in Huxley)
• Click the Huxley terminal.
• Click the test app window.
• Click a button one times (that toggles the button color, same as the example app included in Huxley)
• Click the Huxley terminal.
• Click the test app window.
• Click a button one times (that toggles the button color, same as the example app included in Huxley)
• Click the Huxley terminal.
• Click the test app window.
• Click a button one times (that toggles the button color, same as the example app included in Huxley)
• Click the Huxley terminal.
• Press q, then return to quit.
• Wait for Huxley to rerun the recorded actions.
• Watch Huxley succeed.

huxley also succeeded here as well.

So, it seems like taking screenshots while recording might be a bit of a problem.

Thanks for the help guys!

[igreulich]

Is there a good place for general Huxley discussion?

I ask because I can't make Huxley fail now, and there doesn't seem to be anywhere else to talk about it...

In my test app the button I am clicking toggles color; blue -> grey. I am recording blue -> grey -> blue -> grey -> blue. Then to try and make Huxley fail, I switch the order of the toggle to grey -> blue. As I understand it, the test the should playback grey -> blue -> grey -> blue -> grey.

The test passes.

Per the tutorial, changing the button color should make Huxley, rerecord, and warn me that something has changed. Right?

[chenglou]

Woah, let's get some stuff cleared so that we dont havé any fondamental misunderstanding: the steps are

• manually record
• followed by auto playback
• then auto verification

If you want to verify again, you only run huxley --playback-only. You don't manually re-record anything. Right?

[igreulich]

So once I --record, after the auto verification passes, I change the ui of my test app. When I run huxley after the ui change, it still passes.

I ran the same test with huxley --playback-only, it also passes.

I am trying to see a ui test with Huxley fail.

[chenglou]

Running huxley actually updates the screenshots. Don't do that right after you change the UI in your case.

[igreulich]

Okay, so I reread the README on the repo. I do have a fundamental misunderstanding of how Huxley works.

What I want to see is the screenshots after I change my ui, (not a failed test). But I am not able to record screenshots per the instructions AND have Huxley complete the auto verification step during huxley --record.

I always get huxley.errors.TestError: Screenshot 0 was different.

Full output here.

[chenglou]

You don't re-record w screenshots manually. The 'update screenshot' option is huxley.

[igreulich]

I understand. I am talking about the very first time I run huxley --record, when I am setting up the initial tests w/screenshots for huxley to have something to compare current ui to.

I cannot, nor have I ever had a successful, run of huxley --record if I record screenshots, as the README instructs.

[chenglou]

Can you upload a small screen recording somewhere? It'll help a lot because I suspect the problem is in some detail. QuickTime can record screencasts.

[igreulich]

You were apparently correct. There was some detail getting in the way.. Maybe an initial failure in set up or recording.. I am not sure, but when I record the process, everything works fine.

Thank you for taking the time to help me through all this.

I do feel like I understand how Huxley works much better, having gone through this.

Thanks!

[chenglou]

Glad to help!

[petehunt]

@igreulich your misunderstanding is probably the result of a bug in the docs -- any feedback for how we could clear that up?

[chenglou]

@petehunt to add my own experience: I've had probably the same problem initially, so in my port the options are only --record and --update The default is playback, and an explicit playback flag doesn't exist. This is less for convenience reason and more for preventing newcomers accidentally overriding their old screenshots thinking that huxley is for verification. Like we discussed before, I also don't record any screenshot during record to prevent confusion. There's also no automatic verification after the initial recording (just a playback to actually record the screenshots).

Tl; dr: hux --update to update, hux --record to record followed by an automatic --update and hux to compare and automatically save diff (no explicit flag). Three modes with minimal overlap.

[petehunt]

I like that the default behavior is writing new screenshots because that's the whole point of the workflow :) I'm afraid if it's not the default people may feel that --update has more friction and should be used sparingly which is not the case.

What if instead I printed a message to the console saying exactly what was going on when the tool is started?

[chenglou]

Update can be written as hux -u so it's no extra hassle. But you're right, it's more mental than anything else. 

Better messages is nice. I use red/yellow/green to indicate error/warning/success. They really stand out.

About saving screen shots during recording: that might still be confusing?

On Thu, Sep 19, 2013 at 12:01 PM, Pete Hunt notifications@github.com wrote:

I like that the default behavior is writing new screenshots because that's the whole point of the workflow :) I'm afraid if it's not the default people may feel that --update has more friction and should be used sparingly which is not the case.

What if instead I printed a message to the console saying exactly what was going on when the tool is started?

Reply to this email directly or view it on GitHub: https://github.com/facebook/huxley/issues/27#issuecomment-24750668

[igreulich]

@petehunt, If we are talking about making changes to the doc's;

There might be some confusion in your README's Step 3 and 4, since they happen at the same time, in the same command.

• huxley --record - no problem here
• huxley - Initially I expected this to tell me that something had changed only, not to also update screenshots.
• huxley --playback-only - This is what I expected huxley to do, (per Step 3). Also calling this CI mode might be unintentionally misleading. (Thought now that I know how Huxley works, and what it is supposed to be doing, this does make some sense.)

I come from a server side unit test world, so that might have colored my expectations, but I think this makes a bit more sense;

• huxley --record - same as before
• huxley runs the tests, but does not update the screenshots, but saves a diff, so it would be equivalent to the current huxley --playback-only --save-diff, but I am not sure you can save a diff if you don't update the screenshot?
• huxley --update - updates the screenshots, so it would be equivalent to the current huxley. Maybe there is a --save-diff option for huxley --update, making it equivalent to the current huxley --save-diff.