Using inside Heroku dyno

[VesterDe]

This is less of an issue than it is tech-support I suppose, but I can't seem to get Backstop working inside Heroku.

For sanity's sake I'm trying to run 1 scenario, capturing the entire page, with these config options:

"engine": "chromy", "report": ["CI"], "cliExitOnFail": false, "asyncCaptureLimit": 1, "asyncCompareLimit": 1, "debug": true,

I can generate references and test as expected locally, but on Heroku, the following happens. Once the app is ready and I open the app url, a testing run is started. Chromy is launched and I see the entire page html echoes in the logs, then this:

ChromyJS returned an unexpected error while attempting to capture a selector. { WaitTimeoutError: evaluate() timeout at EvaluateTimeoutError.ExtendableBuiltin (/app/node_modules/chromy/dist/error.js:44:28)

I'm stuck here. The only notable detail is that an iframe (belonging to Stripe) is parsed not long before it hangs (the last html I see is </iframe><a id="scrollUp" href="#top"></a></body>).

The bitmaps_test folder never gets populated with images.

As far as Heroku is concerned, I'm using 2 buildpacks, https://github.com/heroku/heroku-buildpack-google-chrome.git and heroku/nodejs.

Any help would be greatly appreciated.

[garris]

Can you post a gist of your config verbatim? Also, that error usually means that the selector backstop is waiting for was not found — have you tried (just for troubleshooting) using a long delay with no selectors? For that matter — have you tried just a simple static html page with no JS or iFrames? This kind of thing would rule out if it’s actually heroku or maybe something related to your webapp.

[VesterDe]

Yeah sure, here it is:

https://gist.github.com/VesterDe/8fd7e8152e7330802b32c30f75c42945

I've looked at it more now and the problem is completely different than I thought. It's a credentials issue. Is seems that basic auth isn't being resolved properly from the data provided by Heroku Config vars. I'm still debugging it and trying to find out what's going on, but running it on a different site without basic auth works normally with casper (and probably chromy too, I guess).

[garris]

Yup. The config looks good too.

Just a suggestion — you might be able to grab some valid cookies, stash them in a file and use those instead. That method tends to be reliable as long as they are valid for a reasonable duration. There are some tips for this in the docs.

[VesterDe]

I am stumped by this error still, but implementing Backstop was more of a hobby thing for me I can't spend more time looking for the cause here.

The authentication error went away when I started using the (deprecated) user:pass@domain.com scheme for authentication, instead of the "proper" auth methods.

In case someone sees this issue in the future here is where I'm at:

1. The casper engine works on Heroku and locally, but gives different results (fonts are at fault, as far as I can tell), and it also gives different results than the other engines. Although it works most consistently with concurrency set to 1.
2. Chromy and Puppeteer work intermittently on Heroku for me, but fail for different reasons. Chromy doesn't work because I've not figured out Basic Auth for it outside of visiting the page with user:pass@domain.com, which I want to avoid. Pupeteer and Chromy both sometimes fail with an "Unhandled promise rejection handler error", which I've not been able to track down within my code but does appear less on sites without basic auth...
3. Puppeteer does not generate some images. It just makes empty files. It doesn't say the selector wasn't found, or wasn't visible, it just makes an empty file. The pattern of empty files is consistent though... It's always about the same 15% of all cases (of my 230 images across 8 pages and several selectors) and seems independent of concurrency.

I have not ruled out the following:

1. Heroku is running out of resources as puppeteer runs (although, it should complain in the console if memory use goes above 500MB).
2. The problems are limited in large part to pages behind basic auth.

I don't know if closing this issue is the best idea now?

[garris]

Thank you for reporting. I think you we can leave this open for a bit. Cheers.