Travis build hasn't been green for 9+ months

aramprice commented 7 years ago

Cookbook version

2.1.2

Chef-client version

chef-client version: 12.16.42

Platform Details

Worker information
hostname: travis-worker-gce-org-prod4-12:22f9717d-713f-472c-8fb8-cbba3ffc3973
version: v2.5.0 https://github.com/travis-ci/worker/tree/da3a43228dffc0fcca5a46569ca786b22991979f
instance: testing-gce-da39621e-82c3-4dbc-83f4-1c23c5b0bda2:travis-ci-sugilite-trusty-1471824869

Scenario:

A green build of master so that one can submit pull-requests with confidence that the test suite is valid.

Steps to Reproduce:

Submit a PR with no code changes, the build fails:

Expected Result:

CI should succeed on master: https://travis-ci.org/chef-cookbooks/homebrew/builds

The last green build of master was: https://travis-ci.org/chef-cookbooks/homebrew/builds/119385581

Actual Result:

Every build in the last 9 months as a PR or on the master branch have failed.

Perhaps the travis CI could be configured to test against macOS: https://docs.travis-ci.com/user/osx-ci-environment/

rmoriz commented 7 years ago

This is really sad, my issue just reported at https://github.com/chef-cookbooks/homebrew/issues/116 is also clearly visible in the latest travis output. Why was it merged?

Chef, Inc. still does not allocate enough ressources to keep the bare minimum cookbooks in good quality. Sure, one can say: "It's OSS, feel free to fork and fix" but also every new org/person introduced to chef will copy+paste from those broken "template" cookbooks with the great excuse of "even the Chef guys screw it up!".

😡

aramprice commented 7 years ago

Is there a reason homebrew is testing against linux? It seems like this should be using macOS since that is the primary target for homebrew.

On https://github.com/pivotal-sprout we use something like the following to run tests on macOS

language: objective-c # force macOS
osx_image: xcode8.2 # force macOS v10.12
rvm: system # use system ruby
install:
  # we expect sprout to provide homebrew so we should ensure it isn't present
  - brew cask list && brew cask zap $(brew cask list)
  - brew list && brew uninstall --force --ignore-dependencies $(brew list)
  - ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall)" -- --force
  - rm -rf /usr/local/var/postgres # clear out Travis-installed Postgres

  - for ruby in $(rvm list strings); do rvm uninstall "${ruby}"; done
  - travis_wait rvm implode --force # We are testing installation of Rubies, so should uninstall everything
script:
# test script(s) here

tas50 commented 7 years ago

@aramprice The reason we don't test against OS X itself is that homebrew is already installed on those nodes and the OS X infrastructure in Travis is often backed up for multiple hours. We're currently looking at getting our own buildings in house so we can run integration testing on the various cookbooks that support OS X.

As for the state of the builds. They're red right now. No doubt about it. There's an outstanding PR to convert this cookbook over to custom resources and part of that work will include getting the build green. After that we're going to focus on moving any useful resources into core chef itself, which eliminates a good chunk of the need for this cookbook. Expect that during the Chef 13 cycle.

aramprice commented 7 years ago

@tas50 thanks for the update. I understand that Travis can be a bit problematic, especially since the pre-install Homebrew on their VMs.

We worked around this in our tests by "imploding" the pre-installed software with an install: stanza in .travis.yml.

Here's an example from one of our cookbooks: https://github.com/pivotal-sprout/sprout-chruby/blob/master/.travis.yml#L5-L13

install:
  # we expect sprout to provide homebrew so we should ensure it isn't present
  - brew cask list && brew cask zap $(brew cask list)
  - brew list && brew uninstall --force --ignore-dependencies $(brew list)
  - ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall)" -- --force
  - rm -rf /usr/local/var/postgres # clear out Travis-installed Postgres 9.4 data

  - for ruby in $(rvm list strings); do rvm uninstall "${ruby}"; done
  - travis_wait rvm implode --force # We are testing installation of Rubies, so should uninstall everything
  - ./sprout exec true # use `exec` to install dependencies

We also remove rvm because our cookbooks expect to run on a fresh macOS system using the (very outdated) system ruby.

Here's a link to the build: https://travis-ci.org/pivotal-sprout/sprout-chruby

We're using the same technique across the https://github.com/pivotal-sprout org.

sous-chefs / homebrew