Running Nightmare headlessly on Linux

[richard5mith]

I've taken the cnn.js example, and my only change is changing the require line from ../nightmare to nightmare, but when I run it I only get...

DEBUG=nightmare node --harmony cnn.js

nightmare queueing action "goto" +0ms    
nightmare queueing action "evaluate" +5ms

I don't ever seem to get the title of the page.

I've tried various other examples from Nightmare < 2 from around the web, and I can't get any of them to work either. Nothing ever errors, I just never seem to get any screenshot, pdf or elements pulled off the page in the output.

After 4 hours of bashing my head, I'm not sure what else to try.

[idrakimuhamad]

I tried this and got it returned properly.

nightmare queueing action "goto" +0ms
nightmare queueing action "evaluate" +1ms
Breaking News, U.S., World, Weather, Entertainment & Video News - CNN.com

But if I tried with a website that have a SLL problem, like below, it hang there for, forever.

    vo(run)(function(err, result) {
      if (err) throw err;
    });

    function *run() {
        var title = yield nightmare
        .goto('https://cas2.northport.com.my/myapp/wa/r/nmbctr')
        .evaluate(function() {
            return document.title;
        });
        console.log(title);
        yield nightmare.end();
    }

It just hang here. No error, and electron no exited.

    nightmare queueing action "goto" +0ms
    nightmare queueing action "evaluate" +1ms

[richard5mith]

What's your environment for running the cnn.js script?

Entirely headless server, or in a terminal in an existing X session, or OS X etc?

[blechatellier]

I do have the same issue on a headless server. Everything runs correctly on OSX but not on my build server.

[matthewmueller]

Yikes, major issue.

Seems like the electron linux binding is broken or something. Anyone have any ideas?

[richard5mith]

Odd that at least you get an error message when I don't. Mine just fails silently.

My guess is that it can't get the framebuffer, which is where xvfb would normally come in. But I've tried running electron with that too and just get gtk errors.

[matthewmueller]

Okay. I'm not exactly sure how to codify this right now (or what layer to support this in), but to get it working on the official node (docker) image, you need to do the following:

# Install dependencies
apt-get update &&\
    apt-get install -y libgtk2.0-0 libgconf-2-4 \
    libasound2 libxtst6 libxss1 libnss3 xvfb
npm install segmentio/nightmare

# Start Xvfb
Xvfb -ac -screen scrn 1280x2000x24 :9.0 &
export DISPLAY=:9.0

# Test it
apt-get install vim
vim index.js
# <paste in example>
node index.js

More info: https://github.com/atom/electron/issues/228 Dockerfile: https://github.com/aheuermann/docker-electron

---

Could use some help on getting this bug resolved.

[richard5mith]

That doesn't solve it for me on my environment. Still no errors, but still no title from the CNN example either.

I had previously got Gtk3::Webkit Perl module running, which also uses xvfb, and had everything installed except libxss1 already. So I know my xvfb works.

Going round in circles trying to find suggestions. I see the same as this person when running electron through xvfb-run, https://gist.github.com/AspireToCodeBetter/130877925f52c8fb2557.

And the Stack Overflow question points people here, http://stackoverflow.com/questions/32612868/running-electron-atom-shell-headlessly-on-linux-server-through-nightmare-js.

[matthewmueller]

Oh I don't think this is it, but can you try running master? npm install segmentio/nightmare ?

I upgraded it to fix: https://github.com/mafintosh/electron-prebuilt/issues/54#issuecomment-140625958

[richard5mith]

Hey! That did it. I can now start Xvfb on display 9 as you posted before, then I can now run the cnn example.

Looks like it was an incomplete Electron install before.

So can you modify your launch of electron process to use xvfb-run?

[matthewmueller]

@richard5mith i'm not very familiar with xvfb, by xvfb-run do you mean the instructions i provided? or is there a simpler way?

I think it's starting a server (Xvfb -ac -screen scrn 1280x2000x24 :9.0 &), which is something I'd like to avoid. or at least have a good way to clean up after we quit.

[richard5mith]

I was thinking something like this as a variation on your steps, which is how I had run Gtk Webkit .

xvfb-run --server-args="-screen 9 1280x2000x24" ./electron

That does seem to launch correctly and the renderer process starts. So I'm not sure if you can do something similar from your JS and then do the IPC with that process.

[richard5mith]

Actually, this works, without having Xvfb running in the background first as per your previous steps...

DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24" node cnn.js

[blechatellier]

@richard5mith thanks for that, runs for me now on my build server!

[matthewmueller]

Nice! Now just need to figure out the best way to get these dependencies on linux boxes (without using custom buildpacks or anything)

[blechatellier]

@matthewmueller that might help: https://github.com/shippableImages/minBase/blob/master/ubu1204/01base/containerSetup.sh https://github.com/shippableImages/minBase/blob/master/ubu1204/01base/xvfb.sh

Using the shippable container to run nightmare and mocha.

[blechatellier]

@matthewmueller I've narrowed it down to apt-get update && apt-get install -y xvfb x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps clang libdbus-1-dev libgtk2.0-dev libnotify-dev libgnome-keyring-dev libgconf2-dev libasound2-dev libcap-dev libcups2-dev libxtst-dev libxss1 libnss3-dev gcc-multilib g++-multilib

Hope that helps!

[matthewmueller]

Thanks man, unless I'm missing something I think this should do it though: https://github.com/segmentio/nightmare/issues/224#issuecomment-141575361. Or are you talking about without node dependencies?

I think the remaining items to sort out are:

• [ ] When do these installation procedures happen? Is this a postinstall script, a node-gyp thing, or is it something that shouldn't even be handled in this library?
• [ ] Verify that we can run xvfb-run --server-args="-screen 0 1024x768x24" across multiple processes (in other words, running two of these spawn commands at once on the same machine)
• [ ] Ensure that we're cleaning up the xvfb server after we execute (when the node process exits)

I plan on looking at this sometime this week or weekend, but any time offered sorting this stuff out would help accelerate this fix.

[dylanvalade]

I'm unable to fix my #223 Heroku deploy. Updated to Nightmare v2.0.6 and tried custom buildpacks but couldn't sort it out. Has anyone used xvfb on Heroku?

[matthewmueller]

Would be good if we could get some sort of post install script going so it just worked on heroku. Sorry guys, I haven't had much time to look into this lately, but hopefully we can get this sorted out soon

[jabinb]

@matthewmueller https://github.com/Rob--W/node-xvfb might be of use for controlling the xvfb server.

[dylanvalade]

@notsentient Jabin, Simply Wall St looks outstanding. Good find on the xvfb module.

[jney]

same issue there. @dylanvalade did you find out how to make it work ?

[dylanvalade]

@jney Unfortunately, I had to change my plan. Using Phantom now and this explains the setup. http://stackoverflow.com/questions/22116673/cannot-find-module-phantomjs-in-heroku/32853675#32853675

After this Linux issue gets sorted out I would much rather use Nightmare.

[matthewmueller]

One thing I just realized is that nightmare is running fine on Circle CI. Probably missing something, but anyone know how that's working? I doubt they have all this additional setup, but maybe...

[edasque]

This

apt-get update && apt-get install -y xvfb x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps clang libdbus-1-dev libgtk2.0-dev libnotify-dev libgnome-keyring-dev libgconf2-dev libasound2-dev libcap-dev libcups2-dev libxtst-dev libxss1 libnss3-dev gcc-multilib g++-multilib

and

DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24" node cnn.js

worked for me, might be worth updating the doc for since this is going to be mostly used on headless servers.

[danhalliday]

+1 I also have it running relatively painlessly using 'xvfb-run', after trying and failing to get it connected to a separate 'xvfb' process.

[punk-dev-robot]

Did not work for me :( Have all packages installed, using master from nightmare, headless Ubuntu 14.04.3. No error messages, just hangs. Works fine on Mac

EDIT: works with sudo, not perfect but it's something

[dickeylth]

Not work for me either after tried install xvfb-run

➜  nightmare  DEBUG=nightmare* xvfb-run --server-args="-screen 0 1024x768x24" node --harmony test.js
  nightmare queueing action "goto" for https://www.taobao.com +0ms
  nightmare queueing action "evaluate" +6ms

[ghost]

sudo yum install xorg-x11-server-Xvfb xvfb-run --server-args="-screen 0 1024x768x24" node --harmony cnn.js

worked for me on CentOS.

[younes200]

@deinwort which version of CentOS and NodeJS are you using ?

[younes200]

On CentOS 7, with xvfb-run is not working :

# xvfb-run --server-args="-screen 0 1024x768x24" node --harmony cnn.js 
  nightmare queueing action "goto" for http://cnn.com +0ms
  nightmare queueing action "evaluate" +2ms

# uname -a
Linux localhost 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

# node -v 
4.2.3

[younes200]

Xvfb need more packages under Centos otherwise it will crash

yum install xorg-x11-server-Xvfb  gtk2 libXtst GConf2 alsa-lib xorg-x11-fonts*

The best way to debug this, is by running electron directly:

xvfb-run --server-args="-screen 0 1024x768x24 -extension RANDR" ./node_modules/nightmare/node_modules/electron-prebuilt/dist/electron

[matthewmueller]

Anyone get this working in a Dockerfile? Build the image just fine and seems to be working inside the container (docker run -it nightmare /bin/bash) but once I try passing in the command or entrypoint it just hangs (not at nightmare queueing action "evaluate" +2ms, but like it's starting the xvfb server without running the script)

Here's the Dockerfile:

FROM node:5

# Updating ubuntu packages
RUN apt-get update

# Installing the packages needed to run Nightmare
RUN apt-get install -y \
  xvfb \
  x11-xkb-utils \
  xfonts-100dpi \
  xfonts-75dpi \
  xfonts-scalable \
  xfonts-cyrillic \
  x11-apps \
  clang \
  libdbus-1-dev \
  libgtk2.0-dev \
  libnotify-dev \
  libgnome-keyring-dev \
  libgconf2-dev \
  libasound2-dev \
  libcap-dev \
  libcups2-dev \
  libxtst-dev \
  libxss1 \
  libnss3-dev \
  gcc-multilib \
  g++-multilib

ENV DEBUG="nightmare"

# Add current directory to /app
ADD . /app

# Set current working directory as /app
WORKDIR /app

# Install npm packages
RUN npm install

# Default command. Assumes our file is index.js and our screen size is 1024x768
CMD DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24" node index.js

Trying to legitimize this a bit for us all, but major strugglez :-P

[edasque]

For those not familiar with Docker, here is the step by step once you've installed it (it needs a 64 bit modern distro & kernel).

In the nightmare source directory (from this repo), create a file named Dockerfile which has the content at the bottom of this post (@matthewmueller I had to make some edits, including replacing FROM node:5 with FROM node:4.2 for version pinning since 5.x is not LTS) then run:

docker build -t nightmare-test .

This takes a little while. Then:

docker run nightmare-test

expected result is:

Thu, 10 Dec 2015 15:03:07 GMT nightmare queueing action "goto" for http://cnn.com
Thu, 10 Dec 2015 15:03:07 GMT nightmare queueing action "evaluate"
Thu, 10 Dec 2015 15:03:07 GMT nightmare running
Breaking News, Daily News and Videos - CNN.com
Thu, 10 Dec 2015 15:03:25 GMT nightmare running

My Dockerfile is:

FROM node:4.2

# Updating ubuntu packages
RUN apt-get update

# Installing the packages needed to run Nightmare
RUN apt-get install -y \
  xvfb \
  x11-xkb-utils \
  xfonts-100dpi \
  xfonts-75dpi \
  xfonts-scalable \
  xfonts-cyrillic \
  x11-apps \
  clang \
  libdbus-1-dev \
  libgtk2.0-dev \
  libnotify-dev \
  libgnome-keyring-dev \
  libgconf2-dev \
  libasound2-dev \
  libcap-dev \
  libcups2-dev \
  libxtst-dev \
  libxss1 \
  libnss3-dev \
  gcc-multilib \
  g++-multilib

ENV DEBUG="nightmare"

# Add current directory to /app
ADD . /app

# Set current working directory as /app
WORKDIR /app

# Install npm packages
RUN npm install

# Default command. Assumes our file is index.js and our screen size is 1024x768
CMD npm install vo ; xvfb-run --server-args="-screen 0 1024x768x24" node index.js

My index.js for the CNN example (in the root project folder is:

var Nightmare = require('nightmare');
var vo = require('vo');

vo(run)(function(err, result) {
  if (err) throw err;
});

function *run() {
  var nightmare = Nightmare();
  var title = yield nightmare
    .goto('http://cnn.com')
    .evaluate(function() {
      return document.title;
    });
  console.log(title);
  yield nightmare.end();
}

@matthewmueller, if you want me to do a PR with the docker file, an example directory and documentation, let me know.

[matthewmueller]

@edasque thanks for the detailed instructions! the command i'm just trying to get working is this:

docker run nightmare xvfb-run --server-args="-screen 0 1024x768x24" node index.js

or

docker run nightmare node index.js

with xvfb-run --server-args="-screen 0 1024x768x24" being an ENTRYPOINT

---

edit: though come to think of it, the dockerfile you provided is deployable to heroku/dokku on it's own. just looking for a couple ways to make this as easy to use as possible.

[edasque]

index.js being the CNN example, right?

Yes, it's a good idea to use an ENTRYPOINT for that. Let me know what I can help with.

Note however that the ENTRYPOINT will require you to install the application dependencies through nightmare's package.json in this case. Use the exec form of ENTRYPOINT so it could be overriden, maybe?

You could also look at a node-onbuild (such as https://github.com/nodejs/docker-node/blob/master/4.2/onbuild/Dockerfile) for your base image.

[matthewmueller]

index.js being the CNN example, right?

Yep!

Let me know what I can help with.

If you have a chance, do you mind testing this command? docker run nightmare xvfb-run --server-args="-screen 0 1024x768x24" node cnn.js

I was able to get it working by keeping everything in the Dockerfile, but it'd be nice to have flexibility in what script you run. For me when I run that command, it doesn't even run the node script, it just hangs, like it started the xvfb server but didn't do anything after. Tested a ton of iterations, but haven't been able to get a version working where you can pass the file in.

Note however that the ENTRYPOINT will require you to install the application dependencies through nightmare's package.json in this case. Use the exec form of ENTRYPOINT so it could be overriden, maybe?

Yah maybe best to stick with CMD, though you can overwrite ENTRYPOINT via docker run --entrypoint, though i'm not sure that's a great idea. I guess it depends how often you need to change the --server-args

[edasque]

I have the same problem, I don't know what the cause is yet. I thought it might be in the encoding of the quotes & such but I don't think so anymore. Might be helpful to tail the logs if you know where the xvfb stuff goes (docker exec nightmare tail -f /var/log/whatever_it_is.log)

[sriducati]

After creating an app in in windows(Worked awesome) .. now its not running in fedora 64-bit.. I seriously think this is waste of time for everyone.. why dont you guyzz shut down nightmare, Its really nightmare for developers... All my time wasted, tried to fix for 2 days.... Unftunately its not giving any error messages... tried every steps....Its totally waste

[edasque]

I have had no problem here. Have you tried the Docker approach @sriducati?

[sriducati]

@edasque thats the only step left bcos iam new to docker... I tried almost all steps.. trust me,each and every dependencies i fixed for electron,scanned whole internet to fix this issue... spent 2-3 days.. but still not working...

[matthewmueller]

it works using docker. we're working on a better solution for linux. keep in mind this is free software, so please be civil. if you're unhappy with nightmare, you're free to try something else.

[sriducati]

@matthewmueller I am sorry if my comment was offencive .I was very happy with nightmare when working on windows but suddenly all my app stopped working without any error messages... its really very hard...just don't know what to do now...

[edasque]

@sriducati did you try using DEBUG=nightmare ? I think there might be a way to alter the verbosity but I can't recall.

[sriducati]

@edasque ya i tried debug mode below is the result...

[root@server ~]# DEBUG=nightmare node --harmony cnn.js nightmare queueing action "goto" for http://cnn.com +0ms nightmare queueing action "evaluate" +21ms nightmare running +3ms

[younes200]

@sriducati nightmare is base on electron which doesn't have a good support on linux speciality with headlessly window. Best way to debug this on linux is to run first electron : ./node_modules/nightmare/node_modules/electron-prebuilt/dist/electron In most case, it's a dep/library missing. Docker solution may be a the best way for now.

[sriducati]

@younes200 thank you, tried your method as well...result is empty ... again the same prob when we try to run nightmare.. no errors.. no output :(

[sriducati]

Good new everyone.. nightmare worked well with ubuntu 12.04 64-bit

[younes200]

@sriducati could you please post your results here to help others too ;)

[sriducati]

@matthewmueller , @edasque , @younes200 Thank you all...After changing many operating sys versions... found simple fix Ubuntu 12.04 64-bit ;)

root@server:~# xvfb-run node cnn.js
Breaking News, Daily News and Videos - CNN.com
root@server:~# node -v 
v5.3.0
root@server:~# npm -v
3.3.12
root@server:~# 

Its running on OpenVz VPS , No docker used... It would be great if anyone could replay to https://github.com/segmentio/nightmare/issues/411 , I can write a complete help guide to install in VPS ...