cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
46.84k stars 3.17k forks source link

`cypress ci` does not issue proper exit code in Codeship CI #328

Closed jennifer-shehane closed 6 years ago

jennifer-shehane commented 7 years ago

I am having an issue when running tests in Codeship. The tests should exit with the number of failures, but instead it times out.

Error Message

--------------------------------------------------------------------------------
This command didn't output anything for 10 minutes, thus we stopped it.
Please make sure your steps regularly print to standard out or standard error.
If the error is on our end please inform us so we can help you to fix this.
--------------------------------------------------------------------------------

Screenshot screen shot 2016-12-06 at 11 18 27 am

Codeship's support team input

I took a look at some of your builds and I'm wondering more about the cypress ci command. It looks like it finishes, but maybe isn't giving an exit code? Codeship relies on scripts/commands providing a meaningful exit code to know when the command is done and also to then know to move on to the next command or if the build should be failed.

brian-mann commented 7 years ago

Cypress exits correctly. It also works on every other CI provider. Either something is terribly wrong with codeship or somehow Cypress behaves differently on them specifically.

To debug this we'd really need ssh access to one of their instances, or replicate their builds locally.

marceldegraaf commented 7 years ago

I'm having the exact same issue on Semaphore CI. Semaphore does have SSH available, and I can reproduce the issue when I manually retry cypress run or cypress ci from the CI instance.

On Gitter, @brian-mann advised me to embed Cypress in a wrapper script that can check for the close callback of Cypress, and can then force-close the process. Unfortunately the sp.on("close", (exitCode) => { ... } callback is never reached so even in this wrapper script the Cypress run never exits. Eventually Semaphore gives up and fails with a timeout.

I suspect this is a similar issue to what's happening on Codeship. I'd be very happy to help you debug this, as I really want Cypress to work on Semaphore. Please let me know what I can do to help you: debug output, testing a nightly build, ... I'm also up for a quick video call with a contributor to look at the Semaphore instance together. Let me know!

marceldegraaf commented 7 years ago

@jennifer-mann have you been able to find a fix/workaround on Codeship?

brian-mann commented 7 years ago

No, we will have to spend some time looking into this. We should be able to figure it out with the ability to ssh into semaphore - but I'll be honest this isn't a super huge priority at the moment.

Our whole team is super slammed with other things, and worst case this may have to wait until Cypress is OS'd, which will likely happen around March or April.

If you're using Cypress on a team and this is a blocker please message @RandallKent on our Gitter channel and we can talk about prioritizing this.

ervinb commented 7 years ago

Ervin from SemaphoreCI here,

If you can provide me the minimal set of steps to reproduce this behavior, I will happily look into it further.

A while back, there was an edge case with an older version of Chrome, where the process hung in some cases, and I wrote a wrapper around it. You might find this approach useful for your issue: https://gist.github.com/ervinb/54e500a3457a659264f891fe197cb66f

mlocher commented 7 years ago

Marko from Codeship here,

same for me. I'd be happy to look into this. I noticed that you used the https://github.com/cypress-io/cypress-example-kitchensink repository in your support request. Is this still the best way to reproduce this?

# setup
nvm install 5.10.0
npm install
npm install -g cypress-cli
npm start -- --silent &

# actual tests (which hang and time out)
cypress ci
marceldegraaf commented 7 years ago

@ervinb @mlocher thanks for getting involved. Your messages led me to have another good look at our Cypress code. I think I've found the root cause for my problem. Note that I cannot reproduce OP's issue with the kitchen sink project; those tests work for me on Semaphore.

But, for my issue, I think this is the cause. In cypress/support/defaults.js we had a before(() => ... ) hook to start our application server and write a PID. Cypress would then kill the application in an after(() => ...) hook. While debugging this on a Semaphore instance, I found out that this before hook is called multiple times, whereas I would expect it to be called exactly once.

This then caused the PID of our application server process to be updated, so Cypress couldn't kill the process in the after hook. I suspect that this zombie process then causes the main Cypress process to hang on completion.

@brian-mann is it expected that a before hook in defaults.js is called more than once? If not, that might be a bug.

I'm updating our Semaphore steps now to start the server before Cypress even runs, that should solve the issue for me.

mlocher commented 7 years ago

@marceldegraaf I saw similar behaviour running the above commands on a SSH debug build on Codeship.

With the npm start ... command the tests would run successfully, but the cypress ci command would hang at the end.

Without npm start all tests would fail, but the command would exit cleanly. I didn't yet look into why cypress didn't start the application server, as I didn't know whether that was supposed to happen or not.

If you want to take a look at this on Codeship as well, let me know which email your account is registered to and I will switch the account to unlimited builds to allow you to debug this.

brian-mann commented 7 years ago

@marceldegraaf hooks come from Mocha and mocha controls when they are called. It is not possible a before hook is called multiple times.

As to using a before hook to start a webserver...

We definitely do not recommend starting a long lived process from within Cypress.

The reason is that while this works on a single headless run, it would instantly break in the GUI. When you're iterating on writing your tests with the GUI you are constantly refreshing, and there is no guarantee that the after hook would ever be called if you happened to refresh mid-test.

Additionally you must be using something like cy.exec to spawn your server, and cy.exec block until it waits for an exit code. I assume them you're backgrounding the process to make cy.exec return.

This is also bad because there is no guarantee your server is up and running and ready to accept connections by the time this command finishes. My guess is that you are also using some kind of arbitrary .wait(Number) in addition to this.

Another problem is that your server logs / stdout would be completely inaccessible in GUI mode since it would not be spawned from a terminal.

Here's what we recommend - boot your webserver outside of Cypress - before running Cypress. Just look at our any of our examples or recipes and you will see us running npm start prior to invoking Cypress. This works well not only for running tests headlessly in CI but it is the only way to be able to use the GUI as well.

Cypress will now error when being opened if you set your baseUrl to be your server and the server is not running. In other words, we really don't want you to boot your server from inside of Cypress specs.

However, putting all that aside for a second, I do see what Cypress can do better in this situation to prevent zombie processes that keep the process from exiting.

We can likely drop in a lib like terminate or tree-kill or possibly even monkey patch child_process itself to keep a running list of child process PID's spawned from inside of Cypress and then automatically nuke them when we close.

This would prevent any kind of zombie process from being created and not cleaned up.

marceldegraaf commented 7 years ago

@brian-mann thanks for your lengthy reply. Yes, you are correct: we use both cy.exec(...) and cy.wait(...) to make sure the server is running. And I agree that doing this in Cypress is Not Good™, so we'll make sure to move that out of Cypress. For me this issue is now solved, but I'm not sure if @jennifer-mann still has this problem on Codeship. Maybe she can chime in, now that we have everybody gathered here? ;-)

@mlocher your message confirms what @brian-mann says: with npm start (or equivalent, for other platforms) is the way to start an application and run tests on it.

jennifer-shehane commented 6 years ago

I updated our project to the latest cypress version 1.0.3 and am still having this problem. Perhaps I've written my config incorrectly? So, I will paste that.

Setup Commands

nvm install 6.5.0
npm install
npm start -- --silent &

Test Commands

cypress run --record

STDOUT

screen shot 2017-10-30 at 9 18 25 am
jennifer-shehane commented 6 years ago

Still failing: Codeship Status for cypress-io/cypress-example-kitchensink

sparda2 commented 6 years ago

Hi @jennifer-shehane, have you found a work around for this by any chance? we use codeship and in order to use cypress we would need to be able to exit builds properly. Thanks for any tips you could give us.

RandallKent commented 6 years ago

@sparda2 - Our builds of the kitchen sink are still failing on Codeship.

@mlocher - Any thoughts on Jennifer's comment?

RandallKent commented 6 years ago

Running with DEBUG=cypress:* we see cypress run --record hang at:

(All Done)

  cypress:server about to exit with code 0 +1ms

Which appears to come from cypress.coffee#L22


Disabled video recording by running cypress via cypress run --record --config videoRecording=false. The run still hung

bahmutov commented 6 years ago

Simple project without a server is working https://github.com/cypress-io/cypress-test-tiny Its test commands on Codeship Basic are just

nvm install 6.5.0
npm install
npm install cypress
npm run cypress:version
npm run cypress:run

So maybe in kitchensink example there is something else. I tried not starting separate server and using single command, but it still keeps hanging. Tried to skip tests in kitchensink

Finally found that if a single test is disabled everything is working in kitchensink. That test has cy.exec

it.skip('cy.exec() - execute a system command', function () {
  // cy.exec allows you to execute a system command.
  // so you can take actions necessary for your test,
  // but outside the scope of Cypress.

  // https://on.cypress.io/exec
  cy.exec('echo Jane Lane')
    .its('stdout').should('contain', 'Jane Lane')

  // we can use Cypress.platform string to
  // select appropriate command
  // https://on.cypress/io/platform
  cy.log(`Platform ${Cypress.platform} architecture ${Cypress.arch}`)

  if (Cypress.platform === 'win32') {
    cy.exec('print cypress.json')
      .its('stderr').should('be.empty')
  } else {
    cy.exec('cat cypress.json')
      .its('stderr').should('be.empty')

    cy.exec('pwd')
      .its('code').should('eq', 0)
  }
})
bahmutov commented 6 years ago

yeah, seems even a single unit test that uses cy.exec on Codeship basic, even something simple like the echo below is hanging.

cy.exec('echo Jane Lane')
   .its('stdout').should('contain', 'Jane Lane')
bahmutov commented 6 years ago

But cy.exec runs fine on Codeship Pro in https://github.com/cypress-io/cypress-example-docker-codeship

bahmutov commented 6 years ago

I enabled all tests in Kitchensink example, but created a branch https://github.com/cypress-io/cypress-test-tiny/tree/codeship-exec with cy.exec test and it is now hanging in Codeship Basic. I wonder how to find why the cy.exec is hanging, maybe some tool like https://www.npmjs.com/package/why-is-node-running could help, but we are going through Electron ...

RandallKent commented 6 years ago

I wonder if something in Codeship's environment disables execution of arbitrary commands. @mlocher Any ideas?

davidsoderberg commented 6 years ago

Is this still a problem?

RandallKent commented 6 years ago

@davidsoderberg - Yes, our Kitchen Sink is still failing

Codeship Status for cypress-io/cypress-example-kitchensink

bgschiller commented 6 years ago

I was bit by this yesterday, and came up with a workaround script. It's not a long-term solution, but I'm posting here because it could be useful to someone: https://gist.github.com/bgschiller/397ad1ca123aa15f331f3f3ec9717e5f

The workaround is:

joesiewert commented 6 years ago

I'm seeing the same behavior on Codeship Basic that @bahmutov found where just a single cy.exec test shows the hanging. Using version 3.0.2.

I'm looking at it directly in a SSH debug build and also monitoring in top. During cypress run --record I see several Cypress processes running though it looks like there are 2 main ones that stay around for the entire run.

At the end after the spec passes we are left with the hanging and this final bit of debug output:

cypress:server:cypress about to exit with code 0 +9ms

What I found interesting is at this point top now notes there are 2 zombie processes. Further if I stop the hang with ctrl + c I get the prompt back, but the 2 Cypress processes remain running. Only way to clear them is to manually kill the pids. 🤔

brian-mann commented 6 years ago

@joesiewert codeship is the only CI behavior that exhibits this behavior. Cypress currently runs on at least a dozen others without this problem (including both OSX + Windows) even.

I'm not sure if this is a codeship problem, or a linux problem, or a Cypress problem. We are exiting very normally in node, but its possible we may need more aggressive measures to kill the pid trees forcibly.

Do you have any insight into this by chance?

brian-mann commented 6 years ago

The CLI itself acts as a script wrapper around the binary, so it may be possible for us to pass messages between that and the binary and then have the CLI kill it once it knows the exit code.

joesiewert commented 6 years ago

Finally managed to track this down. It looks like ~/.bash_profile is getting sourced when cy.exec is called.

Taking a look at ~/.bash_profile on Codeship, eval $(ssh-agent) is getting called and is responsible for causing Cypress to hang.

There are probably a number of ways to workaround this, but I think the simplest thing is to just drop these commands from ~/.bash_profile before running Cypress:

sed -i '/eval $(ssh-agent)/d' ~/.bash_profile
sed -i '/ssh-add -k/d' ~/.bash_profile

Suggested doc updates here: https://github.com/cypress-io/cypress-documentation/pull/759

bahmutov commented 6 years ago

I added printing all running processes on Codeship after all tests, see https://github.com/cypress-io/cypress-example-kitchensink/tree/print-all-procs and the list is below


> cypress-example-kitchensink@1.0.2 test:ci:record /home/rof/src/github.com/cypress-io/cypress-example-kitchensink
> run-p --race start:ci e2e:record

> cypress-example-kitchensink@1.0.2 start:ci /home/rof/src/github.com/cypress-io/cypress-example-kitchensink
> http-server app -c-1 --silent

> cypress-example-kitchensink@1.0.2 e2e:record /home/rof/src/github.com/cypress-io/cypress-example-kitchensink
> cypress run --record

cypress:cli cli starts with arguments ["/home/rof/.nvm/versions/node/v10.6.0/bin/node","/home/rof/src/github.com/cypress-io/cypress-example-kitchensink/node_modules/.bin/cypress","run","--record"] +0ms
cypress:cli NODE_OPTIONS is not set +0ms
cypress:cli program parsing arguments +3ms
cypress:cli running Cypress +1ms
cypress:cli parsed cli options { record: true } +71ms
cypress:cli verifying Cypress app +0ms
cypress:cli Using CYPRESS_CACHE_FOLDER from environment variable +76ms
cypress:cli Using CYPRESS_CACHE_FOLDER from environment variable +0ms
cypress:cli using environment variable CYPRESS_CACHE_FOLDER /home/rof/cache +0ms
cypress:cli checking environment variables +1ms
cypress:cli checking if executable exists /home/rof/cache/3.0.2/Cypress/Cypress +4ms
cypress:cli Binary is executable? : true +2ms
cypress:cli binaryDir is  /home/rof/cache/3.0.2/Cypress +0ms
cypress:cli Reading binary package.json from: /home/rof/cache/3.0.2/Cypress/resources/app/package.json +6ms
cypress:cli Found binary version 3.0.2 installed in: /home/rof/cache/3.0.2/Cypress +4ms
cypress:cli { verified: true } +5ms
cypress:cli is Verified ? true +1ms
cypress:cli processing run options +0ms
cypress:cli --key is not set, looking up environment variable CYPRESS_RECORD_KEY +1ms
cypress:cli Using CYPRESS_CI_KEY from environment variable +12ms
cypress:cli run to spawn.start args ["--run-project","/home/rof/src/github.com/cypress-io/cypress-example-kitchensink","--key","d4c22689-4da1-4773-b501-f2026d4ea708","--record",true] +0ms
cypress:cli Using CYPRESS_CACHE_FOLDER from environment variable +0ms
cypress:cli Using CYPRESS_CACHE_FOLDER from environment variable +0ms
cypress:cli using environment variable CYPRESS_CACHE_FOLDER /home/rof/cache +1ms
cypress:cli needs XVFB? false +0ms
cypress:cli spawning Cypress with executable: /home/rof/cache/3.0.2/Cypress/Cypress +3ms
cypress:cli spawn forcing env overrides { FORCE_COLOR: '1', DEBUG_COLORS: '1', MOCHA_COLORS: '1', FORCE_STDIN_TTY: '1', FORCE_STDOUT_TTY: '1', FORCE_STDERR_TTY: '1' } +0ms
cypress:cli spawn args [ '--run-project', '/home/rof/src/github.com/cypress-io/cypress-example-kitchensink', '--key', 'd4c22689-4da1-4773-b501-f2026d4ea708', '--record', true, '--cwd', '/home/rof/src/github.com/cypress-io/cypress-example-kitchensink' ] { dev: undefined, detached: false, stdio: 'inherit' } +1ms
Xlib:  extension "RANDR" missing on display ":99.0".
Xlib:  extension "RANDR" missing on display ":99.0".

====================================================================================================

  (Run Starting)

   Cypress:    3.0.2                                                                              
   Browser:    Electron 59 (headless)                                                             
   Specs:      19 found (examples/actions.spec.js, examples/aliasing.spec.js, examples/assertion 
   Run URL:    https://dashboard.cypress.io/#/projects/4b7344/runs/1853                           

Running: examples/actions.spec.js...                                                    (1 of 19) 

  Actions
   .type() - type into a DOM element (4416ms)
   .focus() - focus on a DOM element (382ms)
   .blur() - blur off a DOM element (664ms)
   .clear() - clears an input or textarea element (756ms)
   .submit() - submit a form (564ms)
   .click() - click on a DOM element (3614ms)
   .dblclick() - double click on a DOM element (347ms)
   .check() - check a checkbox or radio element (1853ms)
   .uncheck() - uncheck a checkbox element (1744ms)
   .select() - select an option in a <select> element (1386ms)
   .scrollIntoView() - scroll an element into view (320ms)
   cy.scrollTo() - scroll the window or element to a position (2224ms)
   .trigger() - trigger an event on a DOM element (311ms)

All running processes

PID TTY      STAT   TIME COMMAND
1 ?        Ss     0:00 /sbin/init
508 ?        Ss     0:00 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
600 ?        S      0:00 upstart-socket-bridge --daemon
1397 ?        S      0:00 upstart-udev-bridge --daemon
1412 ?        Ss     0:00 /lib/systemd/systemd-udevd --daemon
1417 ?        Ss     0:00 dbus-daemon --system --fork
1480 ?        Ss     0:00 ofonod -P ril
1482 ?        Ss     0:00 /lib/systemd/systemd-logind
1499 ?        S      0:00 avahi-daemon: running [railsonfire4fcf8d60-392b-460b-bf6a-e7ece00aff42ea648b6b4857.local]
1502 ?        S      0:00 avahi-daemon: chroot helper
1508 ?        Ssl    0:00 rsyslogd
1530 ?        S      0:00 upstart-file-bridge --daemon
1670 ?        Ss     0:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
1744 ?        Ss     0:00 su couchdb -c /usr/bin/couchdb
1748 ?        Ssl    0:00 /usr/lib/erlang/erts-6.3/bin/beam.smp -Bd -K true -A 4 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/couchdb -- -noshell -noinput -os_mon start_memsup false start_cpu_sup false disk_space_check_interval 1 disk_almost_full_threshold 1 -sasl errlog_type error -couch_ini /etc/couchdb/default.ini /etc/couchdb/local.ini -s couch
1780 ?        Ss     0:00 runsvdir -P /etc/service log: ...........................................................................................................................................................................................................................................................................................................................................................................................................
1869 pts/3    Ss+    0:00 /sbin/getty -8 38400 tty4
1870 pts/1    Ss+    0:00 /sbin/getty -8 38400 tty2
1871 pts/2    Ss+    0:00 /sbin/getty -8 38400 tty3
1894 ?        Ss     0:00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
1896 ?        Ss     0:00 cron
1897 ?        Ss     0:00 /usr/sbin/sshd -D
1899 ?        Ss     0:00 atd
1916 ?        Ssl    0:00 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg
1926 ?        Ssl    0:00 /usr/bin/mongod --config /etc/mongod.conf
1927 ?        S      0:00 CRON
1934 ?        Ss     0:00 /bin/sh -c Xvfb :99 -ac +extension RANDR -screen 0 2500x2500x16
1936 ?        Sl     0:00 Xvfb :99 -ac +extension RANDR -screen 0 2500x2500x16
2033 ?        Ssl    0:00 /usr/sbin/mysqld
2084 ?        S      0:00 /usr/lib/postgresql/10/bin/postgres -D /var/lib/postgresql/10/main -c config_file=/etc/postgresql/10/main/postgresql.conf
2090 ?        Ss     0:00 sh -s disksup
2092 ?        Ss     0:00 postgres: checkpointer process                                                                                           
2093 ?        Ss     0:00 postgres: writer process                                                                                                 
2094 ?        Ss     0:00 postgres: wal writer process                                                                                             
2095 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                    
2097 ?        Ss     0:00 postgres: stats collector process                                                                                        
2098 ?        Ss     0:00 postgres: bgworker: logical replication launcher                                                                         
2177 ?        S      0:00 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
2179 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2180 ?        Ss     0:00 postgres: writer process                                                                                                    
2181 ?        Ss     0:00 postgres: wal writer process                                                                                                
2182 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2183 ?        Ss     0:00 postgres: stats collector process                                                                                           
2202 ?        S      0:00 /usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/9.4/main -c config_file=/etc/postgresql/9.4/main/postgresql.conf
2204 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2205 ?        Ss     0:00 postgres: writer process                                                                                                    
2206 ?        Ss     0:00 postgres: wal writer process                                                                                                
2207 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2208 ?        Ss     0:00 postgres: stats collector process                                                                                           
2210 ?        Ss     0:00 sshd: rof [priv]    
2262 ?        S      0:00 sshd: rof@pts/9     
2263 pts/9    Ss     0:00 bash -l
2563 ?        S      0:00 /usr/lib/postgresql/9.5/bin/postgres -D /var/lib/postgresql/9.5/main -c config_file=/etc/postgresql/9.5/main/postgresql.conf
2565 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2566 ?        Ss     0:00 postgres: writer process                                                                                                    
2567 ?        Ss     0:00 postgres: wal writer process                                                                                                
2568 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2569 ?        Ss     0:00 postgres: stats collector process                                                                                           
2659 ?        S      0:00 /usr/lib/postgresql/9.6/bin/postgres -D /var/lib/postgresql/9.6/main -c config_file=/etc/postgresql/9.6/main/postgresql.conf
2661 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2662 ?        Ss     0:00 postgres: writer process                                                                                                    
2663 ?        Ss     0:00 postgres: wal writer process                                                                                                
2664 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2665 ?        Ss     0:00 postgres: stats collector process                                                                                           
2745 ?        S      0:00 /usr/bin/beanstalkd -l 127.0.0.1 -p 11300
2760 ?        Sl     0:00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
2790 ?        S      0:00 /usr/lib/erlang/erts-6.3/bin/epmd -daemon
2880 ?        S      0:00 /bin/sh /usr/sbin/rabbitmq-server
2902 ?        Sl     0:10 /usr/lib/erlang/erts-6.3/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@railsonfire_4fcf8d60-392b-460b-bf6a-e7ece00aff42_ea648b6b4857 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@railsonfire_4fcf8d60-392b-460b-bf6a-e7ece00aff42_ea648b6b4857.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@railsonfire_4fcf8d60-392b-460b-bf6a-e7ece00aff42_ea648b6b4857-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@railsonfire_4fcf8d60-392b-460b-bf6a-e7ece00aff42_ea648b6b4857-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@railsonfire_4fcf8d60-392b-460b-bf6a-e7ece00aff42_ea648b6b4857"
3257 ?        Ss     0:00 inet_gethost 4
3258 ?        S      0:00 inet_gethost 4
3267 ?        Ssl    0:00 /usr/bin/redis-server 127.0.0.1:6379       
3599 ?        S      0:00 /usr/lib/riak/erts-5.9.1/bin/run_erl -daemon /tmp/riak// /var/log/riak exec /usr/sbin/riak console
3601 pts/10   Ssl+   0:08 /usr/lib/riak/erts-5.9.1/bin/beam.smp -K true -A 64 -W w -P 256000 -- -root /usr/lib/riak -progname riak -- -home /var/lib/riak -- -boot /usr/lib/riak/releases/1.4.2/riak -config /etc/riak/app.config -pa /usr/lib/riak/lib/basho-patches -name riak@127.0.0.1 -setcookie riak -smp enable -- console
3927 ?        Ss     0:00 sh -s disksup
3928 ?        Ss     0:00 /usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup
3930 ?        Ss     0:00 /usr/lib/riak/lib/os_mon-2.2.9/priv/bin/cpu_sup
4203 ?        S      0:00 /usr/bin/searchd
4204 ?        Sl     0:00 /usr/bin/searchd
4226 ?        Ss     0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 111:116
4234 ?        S      0:00 /bin/sh /etc/init.d/ondemand background
4240 pts/0    Ss+    0:00 /sbin/getty -8 38400 console
4245 pts/0    Ss+    0:00 /sbin/getty -8 38400 tty1
4250 ?        S      0:00 sleep 60
4339 ?        Ss     0:00 ssh-agent
4675 ?        Ss     0:00 ssh-agent
6259 pts/9    S      0:00 dbus-launch --autolaunch=bf2148bf9e379958e2d0d6f15b3ce5b3 --binary-syntax --close-stderr
6260 ?        Ss     0:00 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
6285 pts/9    Sl+    0:00 npm                                                                 
6295 pts/9    S+     0:00 sh -c run-p --race start:ci e2e:record
6296 pts/9    Sl+    0:00 node /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/node_modules/.bin/run-p --race start:ci e2e:record
6306 pts/9    Sl+    0:00 npm                                                                                                                                
6311 pts/9    Sl+    0:00 npm                                                                                                                                  
6326 pts/9    S+     0:00 sh -c http-server app -c-1 --silent
6327 pts/9    Sl+    0:00 node /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/node_modules/.bin/http-server app -c-1 --silent
6333 pts/9    S+     0:00 sh -c cypress run --record
6334 pts/9    Sl+    0:00 node /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/node_modules/.bin/cypress run --record
6344 pts/9    Sl+    0:19 /home/rof/cache/3.0.2/Cypress/Cypress --run-project --key --record --cwd /home/rof/src/github.com/cypress-io/cypress-example-kitchensink d4c22689-4da1-4773-b501-f2026d4ea708 true /home/rof/src/github.com/cypress-io/cypress-example-kitchensink
6346 pts/9    S+     0:00 /home/rof/cache/3.0.2/Cypress/Cypress --type=zygote --no-sandbox
6348 pts/9    Sl+    0:00 /home/rof/cache/3.0.2/Cypress/Cypress /home/rof/cache/3.0.2/Cypress/resources/app/packages/server/timers/child.js
6532 pts/9    Rl+    0:00 /home/rof/cache/3.0.2/Cypress/Cypress /home/rof/cache/3.0.2/Cypress/resources/app/packages/server/lib/plugins/child/index.js --file /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/cypress/plugins/index.js
6569 pts/9    SNl+   0:08 /home/rof/cache/3.0.2/Cypress/resources/app/packages/server/node_modules/@ffmpeg-installer/linux-x64/ffmpeg -f image2pipe -use_wallclock_as_timestamps 1 -i pipe:0 -y -vcodec libx264 -preset ultrafast /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/cypress/videos/examples/actions.spec.js.mp4
6601 pts/9    Rl+    0:08 /home/rof/cache/3.0.2/Cypress/Cypress --type=renderer --force-device-scale-factor=1 --no-sandbox --primordial-pipe-token=F2B5FB49A0C0876B7E184CA5B48E9FC6 --lang=en-US --app-path=/home/rof/cache/3.0.2/Cypress/resources/app --node-integration=false --webview-tag=false --no-sandbox --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553 --disable-accelerated-video-decode --disable-webrtc-hw-vp8-encoding --disable-gpu-compositing --service-request-channel-token=F2B5FB49A0C0876B7E184CA5B48E9FC6 --renderer-client-id=4 --shared-files=v8_natives_data:100,v8_snapshot_data:101
6778 pts/9    R+     0:00 ps -ax

end

  13 passing (20s)

  (Results)

   Tests:        13                       
   Passing:      13                       
   Failing:      0                        
   Pending:      0                        
   Skipped:      0                        
   Screenshots:  0                        
   Video:        true                     
   Duration:     20 seconds               
   Spec Ran:     examples/actions.spec.js 

  (Video)

  - Started processing:   Compressing to 32 CRF
  - Finished processing:  /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/cypress/videos/examples/actions.spec.js.mp4 (1 second)

  (Uploading Results)

- Done Uploading (1/1) /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/cypress/videos/examples/actions.spec.js.mp4

Running: examples/aliasing.spec.js...                                                   (2 of 19) 

  Aliasing
   .as() - alias a DOM element for later use (618ms)
   .as() - alias a route for later use (530ms)

All running processes

PID TTY      STAT   TIME COMMAND
1 ?        Ss     0:00 /sbin/init
508 ?        Ss     0:00 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
600 ?        S      0:00 upstart-socket-bridge --daemon
1397 ?        S      0:00 upstart-udev-bridge --daemon
1412 ?        Ss     0:00 /lib/systemd/systemd-udevd --daemon
1417 ?        Ss     0:00 dbus-daemon --system --fork
1480 ?        Ss     0:00 ofonod -P ril
1482 ?        Ss     0:00 /lib/systemd/systemd-logind
1499 ?        S      0:00 avahi-daemon: running [railsonfire4fcf8d60-392b-460b-bf6a-e7ece00aff42ea648b6b4857.local]
1502 ?        S      0:00 avahi-daemon: chroot helper
1508 ?        Ssl    0:00 rsyslogd
1530 ?        S      0:00 upstart-file-bridge --daemon
1670 ?        Ss     0:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
1744 ?        Ss     0:00 su couchdb -c /usr/bin/couchdb
1748 ?        Ssl    0:00 /usr/lib/erlang/erts-6.3/bin/beam.smp -Bd -K true -A 4 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/couchdb -- -noshell -noinput -os_mon start_memsup false start_cpu_sup false disk_space_check_interval 1 disk_almost_full_threshold 1 -sasl errlog_type error -couch_ini /etc/couchdb/default.ini /etc/couchdb/local.ini -s couch
1780 ?        Ss     0:00 runsvdir -P /etc/service log: ...........................................................................................................................................................................................................................................................................................................................................................................................................
1869 pts/3    Ss+    0:00 /sbin/getty -8 38400 tty4
1870 pts/1    Ss+    0:00 /sbin/getty -8 38400 tty2
1871 pts/2    Ss+    0:00 /sbin/getty -8 38400 tty3
1894 ?        Ss     0:00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
1896 ?        Ss     0:00 cron
1897 ?        Ss     0:00 /usr/sbin/sshd -D
1899 ?        Ss     0:00 atd
1916 ?        Ssl    0:00 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg
1926 ?        Ssl    0:00 /usr/bin/mongod --config /etc/mongod.conf
1927 ?        S      0:00 CRON
1934 ?        Ss     0:00 /bin/sh -c Xvfb :99 -ac +extension RANDR -screen 0 2500x2500x16
1936 ?        Sl     0:00 Xvfb :99 -ac +extension RANDR -screen 0 2500x2500x16
2033 ?        Ssl    0:00 /usr/sbin/mysqld
2084 ?        S      0:00 /usr/lib/postgresql/10/bin/postgres -D /var/lib/postgresql/10/main -c config_file=/etc/postgresql/10/main/postgresql.conf
2090 ?        Ss     0:00 sh -s disksup
2092 ?        Ss     0:00 postgres: checkpointer process                                                                                           
2093 ?        Ss     0:00 postgres: writer process                                                                                                 
2094 ?        Ss     0:00 postgres: wal writer process                                                                                             
2095 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                    
2097 ?        Ss     0:00 postgres: stats collector process                                                                                        
2098 ?        Ss     0:00 postgres: bgworker: logical replication launcher                                                                         
2177 ?        S      0:00 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
2179 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2180 ?        Ss     0:00 postgres: writer process                                                                                                    
2181 ?        Ss     0:00 postgres: wal writer process                                                                                                
2182 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2183 ?        Ss     0:00 postgres: stats collector process                                                                                           
2202 ?        S      0:00 /usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/9.4/main -c config_file=/etc/postgresql/9.4/main/postgresql.conf
2204 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2205 ?        Ss     0:00 postgres: writer process                                                                                                    
2206 ?        Ss     0:00 postgres: wal writer process                                                                                                
2207 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2208 ?        Ss     0:00 postgres: stats collector process                                                                                           
2210 ?        Ss     0:00 sshd: rof [priv]    
2262 ?        S      0:00 sshd: rof@pts/9     
2263 pts/9    Ss     0:00 bash -l
2563 ?        S      0:00 /usr/lib/postgresql/9.5/bin/postgres -D /var/lib/postgresql/9.5/main -c config_file=/etc/postgresql/9.5/main/postgresql.conf
2565 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2566 ?        Ss     0:00 postgres: writer process                                                                                                    
2567 ?        Ss     0:00 postgres: wal writer process                                                                                                
2568 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2569 ?        Ss     0:00 postgres: stats collector process                                                                                           
2659 ?        S      0:00 /usr/lib/postgresql/9.6/bin/postgres -D /var/lib/postgresql/9.6/main -c config_file=/etc/postgresql/9.6/main/postgresql.conf
2661 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2662 ?        Ss     0:00 postgres: writer process                                                                                                    
2663 ?        Ss     0:00 postgres: wal writer process                                                                                                
2664 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2665 ?        Ss     0:00 postgres: stats collector process                                                                                           
2745 ?        S      0:00 /usr/bin/beanstalkd -l 127.0.0.1 -p 11300
2760 ?        Sl     0:00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
2790 ?        S      0:00 /usr/lib/erlang/erts-6.3/bin/epmd -daemon
2880 ?        S      0:00 /bin/sh /usr/sbin/rabbitmq-server
2902 ?        Sl     0:10 /usr/lib/erlang/erts-6.3/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@railsonfire_4fcf8d60-392b-460b-bf6a-e7ece00aff42_ea648b6b4857 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@railsonfire_4fcf8d60-392b-460b-bf6a-e7ece00aff42_ea648b6b4857.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@railsonfire_4fcf8d60-392b-460b-bf6a-e7ece00aff42_ea648b6b4857-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@railsonfire_4fcf8d60-392b-460b-bf6a-e7ece00aff42_ea648b6b4857-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@railsonfire_4fcf8d60-392b-460b-bf6a-e7ece00aff42_ea648b6b4857"
3257 ?        Ss     0:00 inet_gethost 4
3258 ?        S      0:00 inet_gethost 4
3267 ?        Ssl    0:00 /usr/bin/redis-server 127.0.0.1:6379       
3599 ?        S      0:00 /usr/lib/riak/erts-5.9.1/bin/run_erl -daemon /tmp/riak// /var/log/riak exec /usr/sbin/riak console
3601 pts/10   Ssl+   0:09 /usr/lib/riak/erts-5.9.1/bin/beam.smp -K true -A 64 -W w -P 256000 -- -root /usr/lib/riak -progname riak -- -home /var/lib/riak -- -boot /usr/lib/riak/releases/1.4.2/riak -config /etc/riak/app.config -pa /usr/lib/riak/lib/basho-patches -name riak@127.0.0.1 -setcookie riak -smp enable -- console
3927 ?        Ss     0:00 sh -s disksup
3928 ?        Ss     0:00 /usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup
3930 ?        Ss     0:00 /usr/lib/riak/lib/os_mon-2.2.9/priv/bin/cpu_sup
4203 ?        S      0:00 /usr/bin/searchd
4204 ?        Sl     0:00 /usr/bin/searchd
4226 ?        Ss     0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 111:116
4240 pts/0    Ss+    0:00 /sbin/getty -8 38400 console
4245 pts/0    Ss+    0:00 /sbin/getty -8 38400 tty1
4339 ?        Ss     0:00 ssh-agent
4675 ?        Ss     0:00 ssh-agent
6259 pts/9    S      0:00 dbus-launch --autolaunch=bf2148bf9e379958e2d0d6f15b3ce5b3 --binary-syntax --close-stderr
6260 ?        Ss     0:00 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
6285 pts/9    Sl+    0:00 npm                                                                 
6295 pts/9    S+     0:00 sh -c run-p --race start:ci e2e:record
6296 pts/9    Sl+    0:00 node /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/node_modules/.bin/run-p --race start:ci e2e:record
6306 pts/9    Sl+    0:00 npm                                                                                                                                
6311 pts/9    Sl+    0:00 npm                                                                                                                                  
6326 pts/9    S+     0:00 sh -c http-server app -c-1 --silent
6327 pts/9    Sl+    0:00 node /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/node_modules/.bin/http-server app -c-1 --silent
6333 pts/9    S+     0:00 sh -c cypress run --record
6334 pts/9    Sl+    0:00 node /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/node_modules/.bin/cypress run --record
6344 pts/9    Sl+    0:23 /home/rof/cache/3.0.2/Cypress/Cypress --run-project --key --record --cwd /home/rof/src/github.com/cypress-io/cypress-example-kitchensink d4c22689-4da1-4773-b501-f2026d4ea708 true /home/rof/src/github.com/cypress-io/cypress-example-kitchensink
6346 pts/9    S+     0:00 /home/rof/cache/3.0.2/Cypress/Cypress --type=zygote --no-sandbox
6348 pts/9    Sl+    0:00 /home/rof/cache/3.0.2/Cypress/Cypress /home/rof/cache/3.0.2/Cypress/resources/app/packages/server/timers/child.js
6532 pts/9    Sl+    0:00 /home/rof/cache/3.0.2/Cypress/Cypress /home/rof/cache/3.0.2/Cypress/resources/app/packages/server/lib/plugins/child/index.js --file /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/cypress/plugins/index.js
6889 pts/9    RNl+   0:00 /home/rof/cache/3.0.2/Cypress/resources/app/packages/server/node_modules/@ffmpeg-installer/linux-x64/ffmpeg -f image2pipe -use_wallclock_as_timestamps 1 -i pipe:0 -y -vcodec libx264 -preset ultrafast /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/cypress/videos/examples/aliasing.spec.js.mp4
6911 pts/9    Sl+    0:02 /home/rof/cache/3.0.2/Cypress/Cypress --type=renderer --force-device-scale-factor=1 --no-sandbox --primordial-pipe-token=1A6CD019C8CB878572B332FFB0298E35 --lang=en-US --app-path=/home/rof/cache/3.0.2/Cypress/resources/app --node-integration=false --webview-tag=false --no-sandbox --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553 --disable-accelerated-video-decode --disable-webrtc-hw-vp8-encoding --disable-gpu-compositing --service-request-channel-token=1A6CD019C8CB878572B332FFB0298E35 --renderer-client-id=7 --shared-files=v8_natives_data:100,v8_snapshot_data:101
7038 pts/9    R+     0:00 ps -ax

end

So I see two ssh-agent running

4339 ?        Ss     0:00 ssh-agent
4675 ?        Ss     0:00 ssh-agent

I am also printing the bash profile folder

source .bashrc
eval $(ssh-agent)
ssh-add -k
echo ''
echo '= ENV vars'
source /home/rof/.envexport
echo ''
source /home/rof/bin/cs
source /home/rof/bin/jdk/jdk_switcher
source /home/rof/.rvm/scripts/rvm
cs help

After removing ssh-agent commands from the bash_profile using above

sed -i '/eval $(ssh-agent)/d' ~/.bash_profile
sed -i '/ssh-add -k/d' ~/.bash_profile

I see the following procs

PID TTY      STAT   TIME COMMAND
1 ?        Ss     0:00 /sbin/init
465 ?        Ss     0:00 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
525 ?        S      0:00 upstart-socket-bridge --daemon
1392 ?        S      0:00 upstart-udev-bridge --daemon
1408 ?        Ss     0:00 /lib/systemd/systemd-udevd --daemon
1414 ?        Ss     0:00 dbus-daemon --system --fork
1467 ?        Ss     0:00 /lib/systemd/systemd-logind
1469 ?        Ss     0:00 ofonod -P ril
1489 ?        Ssl    0:00 rsyslogd
1499 ?        S      0:00 avahi-daemon: running [railsonfire9ce60c30-56f4-4580-a38d-de83526a7b3d3b7be472d882.local]
1501 ?        S      0:00 avahi-daemon: chroot helper
1525 ?        S      0:00 upstart-file-bridge --daemon
1670 ?        Ss     0:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
1744 ?        Ss     0:00 su couchdb -c /usr/bin/couchdb
1746 ?        Ssl    0:00 /usr/lib/erlang/erts-6.3/bin/beam.smp -Bd -K true -A 4 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/couchdb -- -noshell -noinput -os_mon start_memsup false start_cpu_sup false disk_space_check_interval 1 disk_almost_full_threshold 1 -sasl errlog_type error -couch_ini /etc/couchdb/default.ini /etc/couchdb/local.ini -s couch
1786 ?        Ss     0:00 runsvdir -P /etc/service log: ...........................................................................................................................................................................................................................................................................................................................................................................................................
1837 pts/3    Ss+    0:00 /sbin/getty -8 38400 tty4
1838 pts/1    Ss+    0:00 /sbin/getty -8 38400 tty2
1839 pts/2    Ss+    0:00 /sbin/getty -8 38400 tty3
1856 ?        Ss     0:00 acpid -c /etc/acpi/events -s /var/run/acpid.socket
1857 ?        Ss     0:00 /usr/sbin/sshd -D
1859 ?        Ss     0:00 cron
1861 ?        Ss     0:00 atd
1875 ?        Ssl    0:00 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg
1893 ?        S      0:00 CRON
1894 ?        Ssl    0:00 /usr/bin/mongod --config /etc/mongod.conf
1898 ?        Ss     0:00 /bin/sh -c Xvfb :99 -ac +extension RANDR -screen 0 2500x2500x16
1908 ?        Sl     0:00 Xvfb :99 -ac +extension RANDR -screen 0 2500x2500x16
1981 ?        Ssl    0:00 /usr/sbin/mysqld
2031 ?        S      0:00 /usr/lib/postgresql/10/bin/postgres -D /var/lib/postgresql/10/main -c config_file=/etc/postgresql/10/main/postgresql.conf
2038 ?        Ss     0:00 postgres: checkpointer process                                                                                           
2039 ?        Ss     0:00 postgres: writer process                                                                                                 
2040 ?        Ss     0:00 postgres: wal writer process                                                                                             
2041 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                    
2042 ?        Ss     0:00 postgres: stats collector process                                                                                        
2043 ?        Ss     0:00 postgres: bgworker: logical replication launcher                                                                         
2044 ?        Ss     0:00 sh -s disksup
2125 ?        S      0:00 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
2127 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2128 ?        Ss     0:00 postgres: writer process                                                                                                    
2129 ?        Ss     0:00 postgres: wal writer process                                                                                                
2130 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2131 ?        Ss     0:00 postgres: stats collector process                                                                                           
2150 ?        S      0:00 /usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/9.4/main -c config_file=/etc/postgresql/9.4/main/postgresql.conf
2152 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2153 ?        Ss     0:00 postgres: writer process                                                                                                    
2154 ?        Ss     0:00 postgres: wal writer process                                                                                                
2155 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2156 ?        Ss     0:00 postgres: stats collector process                                                                                           
2158 ?        Ss     0:00 sshd: rof [priv]    
2213 ?        S      0:00 sshd: rof@pts/9     
2214 pts/9    Ss     0:00 bash -l
2453 ?        S      0:00 /usr/lib/postgresql/9.5/bin/postgres -D /var/lib/postgresql/9.5/main -c config_file=/etc/postgresql/9.5/main/postgresql.conf
2455 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2456 ?        Ss     0:00 postgres: writer process                                                                                                    
2457 ?        Ss     0:00 postgres: wal writer process                                                                                                
2458 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2459 ?        Ss     0:00 postgres: stats collector process                                                                                           
2607 ?        S      0:00 /usr/lib/postgresql/9.6/bin/postgres -D /var/lib/postgresql/9.6/main -c config_file=/etc/postgresql/9.6/main/postgresql.conf
2609 ?        Ss     0:00 postgres: checkpointer process                                                                                              
2610 ?        Ss     0:00 postgres: writer process                                                                                                    
2611 ?        Ss     0:00 postgres: wal writer process                                                                                                
2612 ?        Ss     0:00 postgres: autovacuum launcher process                                                                                       
2613 ?        Ss     0:00 postgres: stats collector process                                                                                           
2693 ?        S      0:00 /usr/bin/beanstalkd -l 127.0.0.1 -p 11300
2709 ?        Sl     0:00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
2738 ?        S      0:00 /usr/lib/erlang/erts-6.3/bin/epmd -daemon
2796 ?        S      0:00 /bin/sh /usr/sbin/rabbitmq-server
2815 ?        Sl     0:04 /usr/lib/erlang/erts-6.3/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@railsonfire_9ce60c30-56f4-4580-a38d-de83526a7b3d_3b7be472d882 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@railsonfire_9ce60c30-56f4-4580-a38d-de83526a7b3d_3b7be472d882.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@railsonfire_9ce60c30-56f4-4580-a38d-de83526a7b3d_3b7be472d882-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@railsonfire_9ce60c30-56f4-4580-a38d-de83526a7b3d_3b7be472d882-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@railsonfire_9ce60c30-56f4-4580-a38d-de83526a7b3d_3b7be472d882"
3074 ?        Ss     0:00 inet_gethost 4
3075 ?        S      0:00 inet_gethost 4
3084 ?        Ssl    0:00 /usr/bin/redis-server 127.0.0.1:6379       
3318 ?        S      0:00 /usr/lib/riak/erts-5.9.1/bin/run_erl -daemon /tmp/riak// /var/log/riak exec /usr/sbin/riak console
3320 pts/10   Ssl+   0:05 /usr/lib/riak/erts-5.9.1/bin/beam.smp -K true -A 64 -W w -P 256000 -- -root /usr/lib/riak -progname riak -- -home /var/lib/riak -- -boot /usr/lib/riak/releases/1.4.2/riak -config /etc/riak/app.config -pa /usr/lib/riak/lib/basho-patches -name riak@127.0.0.1 -setcookie riak -smp enable -- console
3550 ?        Ss     0:00 sh -s disksup
3551 ?        Ss     0:00 /usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup
3553 ?        Ss     0:00 /usr/lib/riak/lib/os_mon-2.2.9/priv/bin/cpu_sup
3730 ?        S      0:00 /usr/bin/searchd
3731 ?        Sl     0:00 /usr/bin/searchd
3753 ?        Ss     0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 111:116
3762 ?        S      0:00 /bin/sh /etc/init.d/ondemand background
3768 pts/7    Ss+    0:00 /sbin/getty -8 38400 console
3775 pts/0    Ss+    0:00 /sbin/getty -8 38400 tty1
3776 ?        S      0:00 sleep 60
3867 ?        Ss     0:00 ssh-agent
4203 ?        Ss     0:00 ssh-agent
5782 pts/9    S      0:00 dbus-launch --autolaunch=bf2148bf9e379958e2d0d6f15b3ce5b3 --binary-syntax --close-stderr
5783 ?        Ss     0:00 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
5817 pts/9    Sl+    0:00 npm                                                                 
5827 pts/9    S+     0:00 sh -c run-p --race start:ci e2e:record
5828 pts/9    Sl+    0:00 node /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/node_modules/.bin/run-p --race start:ci e2e:record
5838 pts/9    Sl+    0:00 npm                                                                                                                                
5839 pts/9    Sl+    0:00 npm                                                                                                                                  
5854 pts/9    S+     0:00 sh -c http-server app -c-1 --silent
5859 pts/9    Sl+    0:00 node /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/node_modules/.bin/http-server app -c-1 --silent
5865 pts/9    S+     0:00 sh -c cypress run --record
5866 pts/9    Sl+    0:00 node /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/node_modules/.bin/cypress run --record
5878 pts/9    Rl+    0:16 /home/rof/cache/3.0.2/Cypress/Cypress --run-project --key --record --cwd /home/rof/src/github.com/cypress-io/cypress-example-kitchensink d4c22689-4da1-4773-b501-f2026d4ea708 true /home/rof/src/github.com/cypress-io/cypress-example-kitchensink
5880 pts/9    S+     0:00 /home/rof/cache/3.0.2/Cypress/Cypress --type=zygote --no-sandbox
5882 pts/9    Sl+    0:00 /home/rof/cache/3.0.2/Cypress/Cypress /home/rof/cache/3.0.2/Cypress/resources/app/packages/server/timers/child.js
6067 pts/9    Rl+    0:00 /home/rof/cache/3.0.2/Cypress/Cypress /home/rof/cache/3.0.2/Cypress/resources/app/packages/server/lib/plugins/child/index.js --file /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/cypress/plugins/index.js
6105 pts/9    SNl+   0:07 /home/rof/cache/3.0.2/Cypress/resources/app/packages/server/node_modules/@ffmpeg-installer/linux-x64/ffmpeg -f image2pipe -use_wallclock_as_timestamps 1 -i pipe:0 -y -vcodec libx264 -preset ultrafast /home/rof/src/github.com/cypress-io/cypress-example-kitchensink/cypress/videos/examples/actions.spec.js.mp4
6137 pts/9    Sl+    0:07 /home/rof/cache/3.0.2/Cypress/Cypress --type=renderer --force-device-scale-factor=1 --no-sandbox --primordial-pipe-token=F8C2DA3518135DD830ECF67774D39056 --lang=en-US --app-path=/home/rof/cache/3.0.2/Cypress/resources/app --node-integration=false --webview-tag=false --no-sandbox --enable-pinch --num-raster-threads=4 --enable-main-frame-before-activation --content-image-texture-target=0,0,3553;0,1,3553;0,2,3553;0,3,3553;0,4,3553;0,5,3553;0,6,3553;0,7,3553;0,8,3553;0,9,3553;0,10,3553;0,11,3553;0,12,3553;0,13,3553;0,14,3553;0,15,3553;0,16,3553;1,0,3553;1,1,3553;1,2,3553;1,3,3553;1,4,3553;1,5,3553;1,6,3553;1,7,3553;1,8,3553;1,9,3553;1,10,3553;1,11,3553;1,12,3553;1,13,3553;1,14,3553;1,15,3553;1,16,3553;2,0,3553;2,1,3553;2,2,3553;2,3,3553;2,4,3553;2,5,3553;2,6,3553;2,7,3553;2,8,3553;2,9,3553;2,10,3553;2,11,3553;2,12,3553;2,13,3553;2,14,3553;2,15,3553;2,16,3553;3,0,3553;3,1,3553;3,2,3553;3,3,3553;3,4,3553;3,5,3553;3,6,3553;3,7,3553;3,8,3553;3,9,3553;3,10,3553;3,11,3553;3,12,3553;3,13,3553;3,14,3553;3,15,3553;3,16,3553;4,0,3553;4,1,3553;4,2,3553;4,3,3553;4,4,3553;4,5,3553;4,6,3553;4,7,3553;4,8,3553;4,9,3553;4,10,3553;4,11,3553;4,12,3553;4,13,3553;4,14,3553;4,15,3553;4,16,3553 --disable-accelerated-video-decode --disable-webrtc-hw-vp8-encoding --disable-gpu-compositing --service-request-channel-token=F8C2DA3518135DD830ECF67774D39056 --renderer-client-id=4 --shared-files=v8_natives_data:100,v8_snapshot_data:101
6271 pts/9    R+     0:00 ps -ax

and the only different command present in the hanging build is

3762 ?        S      0:00 /bin/sh /etc/init.d/ondemand background

hmm

bahmutov commented 6 years ago

@Bkucera can we check which procs were started by execa from Cypress during cy.exec command (including sourcing the profile) and make sure we clean up all of them after cy.exec is done? That might resolve this problem, I really don't want to advise users to sed the profile file arbitrarily.

Relevant code: packages/server/lib/exec.coffee where we use execa (should upgrade from ^0.8 to ^0.10) and do not pass cleanup flag https://github.com/sindresorhus/execa#cleanup. Maybe we should pass this flag.

Or maybe just detach process when starting it https://github.com/sindresorhus/execa#detached

If we passing more options to the execa then maybe we should expose these as options object in cy.exec command

brian-mann commented 6 years ago

All that is necessary IMO is simple use { detached: true } when using cy.exec so that sub processes do not hold up the parent Cypress process when exiting.

And also calling .unref().

It may actually be possible to simply call .unref() without needing a detached process as well.

brian-mann commented 6 years ago

Also just wanted to say you are awesome @joesiewert as I don't think we would have ever been able to figure this out, haha.

bahmutov commented 6 years ago

I could not recreate hanging build using "normal" node process, might be due to electron spawn?

so adding this flag in a little bit of a "blind" mode

mitar commented 6 years ago

BTW, the "still failing comments" above now have a green icons in all of them. :-) So it seems things are passing now.

joesiewert commented 6 years ago

I'm still seeing this hang on Codeship Basic without this workaround in place.

sed -i '/eval $(ssh-agent)/d' ~/.bash_profile
sed -i '/ssh-add -k/d' ~/.bash_profile
nvm install 10
npm ci
npm run cy:verify
npm start -- --silent &
npm run cy:run
tcastelli commented 5 years ago

Also doesn't work for us without the workaround (still hangs) so +1 to adding the workaround on a PR again @bahmutov @brian-mann

sunsheeppoplar commented 5 years ago

We're investigating Codeship for use as well, wanted to see if there was any movement or additional information pertaining to this issue, @joesiewert @tcastelli

Thanks!

cc: @benwfreed

jennifer-shehane commented 4 years ago

If you're experiencing a bug similar to this in Cypress, please open a new issue with a fully reproducible example that we can run. There may be a specific edge case with the issue that we need more detail to fix.