Flakey test runs on CircleCI

[jpparsons]

Is anyone having flakey test runs with CircleCI 1.0. I have been running teaspoon with no issues for many months and in the last few weeks I have been getting flakey test runs. Sometimes the tests run ok, other times teaspoon starts and Puma is listening but there is no test output and CircleCI just kills the test after a 10min timeout. Here's what I have:

Rails 4.2.0 Ruby 2.3.1 Puma 3.7.1 Teaspoon 1.1.5 Teaspoon-Jasmine 2.4.1

I have had these versions running for almost 2 months. Assets are pre-compiled (re-stored from cache) every build in a previous CircleCI step.

I run CircleCI builds using 4x parallel nodes. On node 0, I run teaspoon and here is the output when it fails:

case $CIRCLE_NODE_INDEX in 0) bundle exec rake teaspoon ;; 1) bundle exec rake rubocop ;; 2) bundle exec rake coffeelint && bundle exec rake csslint:run ;; 3) bundle exec rake brakeman:check && bundle exec rake eslint:run ;; esac Starting the Teaspoon server... Puma starting in single mode...

• Version 3.7.1 (ruby 2.3.1-p112), codename: Snowy Sagebrush
• Min threads: 0, max threads: 16
• Environment: test
• Listening on tcp://127.0.0.1:42424 Use Ctrl-C to stop Teaspoon running default suite at http://127.0.0.1:42424/testing/teaspoon/default

command case $CIRCLE_NODE_INDEX in 0) bundle exec rake teaspoon ;; 1) bundle exec rake rubocop ;; 2) bundle exec rake coffeelint && bundle exec rake csslint:run ;; 3) bundle exec rake brakeman:check && bundle exec rake eslint:run ;; esac took more than 10 minutes since last output

Iv'e done a CircleCI debug build and I can run teaspoon manually with the same result. Some runs go ok, some just hang. I ran strace on a teaspoon run when it fails and I can see teaspoon just waiting on a read (pipe2). The specs don't appear to be running.

Has anyone else seen this or have any suggestions?

[jejacks0n]

What runner are you setup to use? phantom, selenium, etc.

I've not seen this behavior exactly, but the runner would probably be the biggest factor in that.

[jpparsons]

Im using phantomjs 2.1.1.0

[jpparsons]

I seem to have resolved this issue by running Puma in cluster mode, using 1 worker (was running it in single mode as the output shows above). You can close this issue.

[ilyakatz]

I seem to have resolved this issue by running Puma in cluster mode, using 1 worker (was running it in single mode as the output shows above)

@jpparsons how did you configure that?

[jpparsons]

how did you configure that?

@ilyakatz At the time this issue was filed, not stated above, but we had the Capybara gem pinned at 2.10 and rails was not observing Capybara Puma config. So I think I just made sure the puma config set a worker count of 1.

Since then, I believe rails will now use Capybara Puma config, so the above won't put it in cluster mode. https://github.com/rails/rails/pull/30638

Sorry, my memory on this is a bit old as I have upgraded many parts since then. Long story short, I don't think this is an issue anymore if you are using Rails 4.2+ and a newer version of Capybara gem (2.16+). This issue was probably a combination of a lot of moving parts at the time.

[ilyakatz]

Got, thanks for the info. I'm having similar issues now, and stumbled upon this. Will keep trying