search

travis-ci / travis-ci

Free continuous integration platform for GitHub projects.
https://travis-ci.org
8.42k stars 723 forks source link

`docker pull` stalls, times out, and error the job #9127

Closed dylandechant closed 5 years ago

dylandechant commented 6 years ago

We have been seeing significant inconsistencies in travis since Saturday 1/20/2018. some builds error with timeouts while when restarting the same build they finish in ~6 minutes.

look at the screen shot of this log:

screen shot 2018-01-23 at 7 34 14 pm

this is getting to the point where it is slowing down development as it is creating a queue of PRs that are constantly failing and when restarted may or may not pass but every once in a while they do.

Please advise.

Thanks

BanzaiMan commented 6 years ago

Do you have a build log URL that shows the problem you are describing here?

BanzaiMan commented 6 years ago

When these build fail, how do they do that? Restarting them will unfortunately wipe out the logs, and we are unable to investigate further.

dylandechant commented 6 years ago

here is one log: https://travis-ci.com/PerkinsAndWill-IO/hub-ui/builds/64121667 and here is the log output:

Worker information
hostname: a950b1a0-5ace-4f6c-8319-c70395ba3bfc@1.production-1-worker-com-f-3-gce
version: v3.3.1 https://github.com/travis-ci/worker/tree/1153e5010ef9863bf96264b75fba506109633460
instance: travis-job-d3f9b627-fe89-45d7-aef0-235e9e92463f travis-ci-garnet-trusty-1512502259-986baf0 (via amqp)
startup: 21.128749146s
mode of ‘/usr/local/clang-5.0.0/bin’ changed from 0777 (rwxrwxrwx) to 0775 (rwxrwxr-x)
system_info
Build system information
Build language: python
Build group: stable
Build dist: trusty
Build id: 64121667
Job id: 107293059
Runtime kernel version: 4.4.0-101-generic
travis-build version: c129335c3
Build image provisioning date and time
Tue Dec  5 19:58:13 UTC 2017
Operating System Details
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:    14.04
Codename:   trusty
Cookbooks Version
7c2c6a6 https://github.com/travis-ci/travis-cookbooks/tree/7c2c6a6
git version
git version 2.15.1
bash version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
gcc version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:38 2017
 OS/Arch:      linux/amd64
Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:20 2017
 OS/Arch:      linux/amd64
 Experimental: false
clang version
clang version 5.0.0 (tags/RELEASE_500/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/clang-5.0.0/bin
jq version
jq-1.5
bats version
Bats 0.4.0
shellcheck version
0.4.6
shfmt version
v2.0.0
ccache version
ccache version 3.1.9
Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2011 Joel Rosdahl
This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; either version 3 of the License, or (at your option) any later
version.
cmake version
cmake version 3.9.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
heroku version
heroku-cli/6.14.39-addc925 (linux-x64) node-v9.2.0
imagemagick version
Version: ImageMagick 6.7.7-10 2017-07-31 Q16 http://www.imagemagick.org
md5deep version
4.2
mercurial version
Mercurial Distributed SCM (version 4.2.2)
(see https://mercurial-scm.org for more information)
Copyright (C) 2005-2017 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
mysql version
mysql  Ver 14.14 Distrib 5.6.33, for debian-linux-gnu (x86_64) using  EditLine wrapper
openssl version
OpenSSL 1.0.1f 6 Jan 2014
packer version
Packer v1.0.2
Your version of Packer is out of date! The latest version
is 1.1.2. You can update by downloading from www.packer.io
postgresql client version
psql (PostgreSQL) 9.6.6
ragel version
Ragel State Machine Compiler version 6.8 Feb 2013
Copyright (c) 2001-2009 by Adrian Thurston
subversion version
svn, version 1.8.8 (r1568071)
   compiled Aug 10 2017, 17:20:39 on x86_64-pc-linux-gnu
Copyright (C) 2013 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/
The following repository access (RA) modules are available:
* ra_svn : Module for accessing a repository using the svn network protocol.
  - with Cyrus SASL authentication
  - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
  - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using serf.
  - using serf 1.3.3
  - handles 'http' scheme
  - handles 'https' scheme
sudo version
Sudo version 1.8.9p5
Configure options: --prefix=/usr -v --with-all-insults --with-pam --with-fqdn --with-logging=syslog --with-logfac=authpriv --with-env-editor --with-editor=/usr/bin/editor --with-timeout=15 --with-password-timeout=0 --with-passprompt=[sudo] password for %p:  --without-lecture --with-tty-tickets --disable-root-mailer --enable-admin-flag --with-sendmail=/usr/sbin/sendmail --with-timedir=/var/lib/sudo --mandir=/usr/share/man --libexecdir=/usr/lib/sudo --with-sssd --with-sssd-lib=/usr/lib/x86_64-linux-gnu --with-selinux
Sudoers policy plugin version 1.8.9p5
Sudoers file grammar version 43
Sudoers path: /etc/sudoers
Authentication methods: 'pam'
Syslog facility if syslog is being used for logging: authpriv
Syslog priority to use when user authenticates successfully: notice
Syslog priority to use when user authenticates unsuccessfully: alert
Send mail if the user is not in sudoers
Use a separate timestamp for each user/tty combo
Lecture user the first time they run sudo
Root may run sudo
Allow some information gathering to give useful error messages
Require fully-qualified hostnames in the sudoers file
Visudo will honor the EDITOR environment variable
Set the LOGNAME and USER environment variables
Length at which to wrap log file lines (0 for no wrap): 80
Authentication timestamp timeout: 15.0 minutes
Password prompt timeout: 0.0 minutes
Number of tries to enter a password: 3
Umask to use or 0777 to use user's: 022
Path to mail program: /usr/sbin/sendmail
Flags for mail program: -t
Address to send mail to: root
Subject line for mail messages: *** SECURITY information for %h ***
Incorrect password message: Sorry, try again.
Path to authentication timestamp dir: /var/lib/sudo
Default password prompt: [sudo] password for %p: 
Default user to run commands as: root
Value to override user's $PATH with: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Path to the editor for use by visudo: /usr/bin/editor
When to require a password for 'list' pseudocommand: any
When to require a password for 'verify' pseudocommand: all
File descriptors >= 3 will be closed before executing a command
Environment variables to check for sanity:
    TZ
    TERM
    LINGUAS
    LC_*
    LANGUAGE
    LANG
    COLORTERM
Environment variables to remove:
    RUBYOPT
    RUBYLIB
    PYTHONUSERBASE
    PYTHONINSPECT
    PYTHONPATH
    PYTHONHOME
    TMPPREFIX
    ZDOTDIR
    READNULLCMD
    NULLCMD
    FPATH
    PERL5DB
    PERL5OPT
    PERL5LIB
    PERLLIB
    PERLIO_DEBUG 
    JAVA_TOOL_OPTIONS
    SHELLOPTS
    GLOBIGNORE
    PS4
    BASH_ENV
    ENV
    TERMCAP
    TERMPATH
    TERMINFO_DIRS
    TERMINFO
    _RLD*
    LD_*
    PATH_LOCALE
    NLSPATH
    HOSTALIASES
    RES_OPTIONS
    LOCALDOMAIN
    CDPATH
    IFS
Environment variables to preserve:
    JAVA_HOME
    TRAVIS
    CI
    DEBIAN_FRONTEND
    XAUTHORIZATION
    XAUTHORITY
    PS2
    PS1
    PATH
    LS_COLORS
    KRB5CCNAME
    HOSTNAME
    HOME
    DISPLAY
    COLORS
Locale to use while parsing sudoers: C
Directory in which to store input/output logs: /var/log/sudo-io
File in which to store the input/output log: %{seq}
Add an entry to the utmp/utmpx file when allocating a pty
PAM service name to use
PAM service name to use for login shells
Create a new PAM session for the command to run in
Maximum I/O log sequence number: 0
Local IP address and netmask pairs:
    10.240.0.28/255.255.255.255
    172.17.0.1/255.255.0.0
Sudoers I/O plugin version 1.8.9p5
gzip version
gzip 1.6
Copyright (C) 2007, 2010, 2011 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License .
There is NO WARRANTY, to the extent permitted by law.
Written by Jean-loup Gailly.
zip version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.
Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.
Compiled with gcc 4.8.2 for Unix (Linux ELF) on Oct 21 2013.
Zip special compilation options:
    USE_EF_UT_TIME       (store Universal Time)
    BZIP2_SUPPORT        (bzip2 library version 1.0.6, 6-Sept-2010)
        bzip2 code and library copyright (c) Julian R Seward
        (See the bzip2 license for terms of use)
    SYMLINK_SUPPORT      (symbolic links supported)
    LARGE_FILE_SUPPORT   (can read and write large files on file system)
    ZIP64_SUPPORT        (use Zip64 to store large files in archives)
    UNICODE_SUPPORT      (store and read UTF-8 Unicode paths)
    STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field)
    UIDGID_NOT_16BIT     (old Unix 16-bit UID/GID extra field not used)
    [encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3)
Encryption notice:
    The encryption code of this program is not copyrighted and is
    put in the public domain.  It was originally written in Europe
    and, to the best of our knowledge, can be freely distributed
    in both source and object forms from any country, including
    the USA under License Exception TSU of the U.S. Export
    Administration Regulations (section 740.13(e)) of 6 June 2002.
Zip environment options:
             ZIP:  [none]
          ZIPOPT:  [none]
vim version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Nov 24 2016 16:43:18)
Included patches: 1-52
Extra patches: 8.0.0056
Modified by pkg-vim-maintainers@lists.alioth.debian.org
Compiled by buildd@
Huge version without GUI.  Features included (+) or not (-):
+acl             +farsi           +mouse_netterm   +syntax
+arabic          +file_in_path    +mouse_sgr       +tag_binary
+autocmd         +find_in_path    -mouse_sysmouse  +tag_old_static
-balloon_eval    +float           +mouse_urxvt     -tag_any_white
-browse          +folding         +mouse_xterm     -tcl
++builtin_terms  -footer          +multi_byte      +terminfo
+byte_offset     +fork()          +multi_lang      +termresponse
+cindent         +gettext         -mzscheme        +textobjects
-clientserver    -hangul_input    +netbeans_intg   +title
-clipboard       +iconv           +path_extra      -toolbar
+cmdline_compl   +insert_expand   -perl            +user_commands
+cmdline_hist    +jumplist        +persistent_undo +vertsplit
+cmdline_info    +keymap          +postscript      +virtualedit
+comments        +langmap         +printer         +visual
+conceal         +libcall         +profile         +visualextra
+cryptv          +linebreak       +python          +viminfo
+cscope          +lispindent      -python3         +vreplace
+cursorbind      +listcmds        +quickfix        +wildignore
+cursorshape     +localmap        +reltime         +wildmenu
+dialog_con      -lua             +rightleft       +windows
+diff            +menu            -ruby            +writebackup
+digraphs        +mksession       +scrollbind      -X11
-dnd             +modify_fname    +signs           -xfontset
-ebcdic          +mouse           +smartindent     -xim
+emacs_tags      -mouseshape      -sniff           -xsmp
+eval            +mouse_dec       +startuptime     -xterm_clipboard
+ex_extra        +mouse_gpm       +statusline      -xterm_save
+extra_search    -mouse_jsbterm   -sun_workshop    -xpm
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H     -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1      
Linking: gcc   -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed -o vim        -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl    -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions      
iptables version
iptables v1.4.21
curl version
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
wget version
GNU Wget 1.15 built on linux-gnu.
rsync version
rsync  version 3.1.0  protocol version 31
gimme version
v1.2.0
nvm version
0.33.6
perlbrew version
/home/travis/perl5/perlbrew/bin/perlbrew  - App::perlbrew/0.80
phpenv version
rbenv 1.1.1-25-g6aa70b6
rvm version
rvm 1.29.3 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
default ruby version
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
CouchDB version
couchdb 1.6.1
ElasticSearch version
5.5.0
Installed Firefox version
firefox 56.0.2
MongoDB version
MongoDB 3.4.10
PhantomJS version
2.1.1
Pre-installed PostgreSQL versions
9.2.24
9.3.20
9.4.15
9.5.10
9.6.6
RabbitMQ Version
3.6.14
Redis version
redis-server 4.0.6
riak version
2.2.3
Pre-installed Go versions
1.7.4
ant version
Apache Ant(TM) version 1.9.3 compiled on April 8 2014
mvn version
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
Maven home: /usr/local/maven-3.5.2
Java version: 1.8.0_151, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-98-generic", arch: "amd64", family: "unix"
gradle version
------------------------------------------------------------
Gradle 4.0.1
------------------------------------------------------------
Build time:   2017-07-07 14:02:41 UTC
Revision:     38e5dc0f772daecca1d2681885d3d85414eb6826
Groovy:       2.4.11
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.8.0_151 (Oracle Corporation 25.151-b12)
OS:           Linux 4.4.0-98-generic amd64
lein version
Leiningen 2.8.1 on Java 1.8.0_151 Java HotSpot(TM) 64-Bit Server VM
Pre-installed Node.js versions
v4.8.6
v6.12.0
v6.12.1
v8.9
v8.9.1
phpenv versions
  system
  5.6
* 5.6.32 (set by /home/travis/.phpenv/version)
  7.0
  7.0.25
  7.1
  7.1.11
  hhvm
  hhvm-stable
composer --version
Composer version 1.5.2 2017-09-11 16:59:25
Pre-installed Ruby versions
ruby-2.2.7
ruby-2.3.4
ruby-2.4.1
removed ‘/etc/apt/sources.list.d/basho_riak.list’
W: http://ppa.launchpad.net/couchdb/stable/ubuntu/dists/trusty/Release.gpg: Signature by key 15866BAFD9BCC4F3C1E0DFC7D69548E1C17EAB57 uses weak digest algorithm (SHA1)
W: Failed to fetch https://packagecloud.io/computology/apt-backport/ubuntu/dists/trusty/InRelease  Failed to connect to packagecloud-repositories.s3.dualstack.us-west-1.amazonaws.com port 443: Connection timed out
W: Failed to fetch http://repo.mongodb.org/apt/ubuntu/dists/trusty/mongodb-org/3.4/InRelease  Could not connect to repo.mongodb.org:80 (52.84.5.22), connection timed out
W: Some index files failed to download. They have been ignored, or old ones used instead.
127.0.0.1 localhost nettuno travis vagrant
127.0.1.1 travis-job-d3f9b627-fe89-45d7-aef0-235e9e92463f travis-job-d3f9b627-fe89-45d7-aef0-235e9e92463f ip4-loopback trusty64
Installing an SSH key from: default repository key
Key fingerprint: a0:d2:2c:6d:bc:8e:db:f6:c2:4a:13:0c:1f:2d:61:1d
git.checkout
1.02s$ git clone --depth=50 --branch=development git@github.com:PerkinsAndWill-IO/hub-ui.git PerkinsAndWill-IO/hub-ui
Cloning into 'PerkinsAndWill-IO/hub-ui'...
Warning: Permanently added the RSA host key for IP address '192.30.253.112' to the list of known hosts.
$ cd PerkinsAndWill-IO/hub-ui
$ git checkout -qf 63fd7bf81de96d2d4d94613c578a0aeb619f44d0
Setting environment variables from repository settings
[redacted]
0.01s$ source ~/virtualenv/python2.7/bin/activate
cache.1
Setting up build cache
$ export CASHER_DIR=$HOME/.casher
0.14s$ Installing caching utilities
0.01s
5.43sattempting to download cache archive
fetching development/cache-linux-trusty-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855--python-2.7.tgz
found cache
0.01s
6.91sadding /home/travis/build/PerkinsAndWill-IO/hub-ui/node_modules to cache
creating directory /home/travis/build/PerkinsAndWill-IO/hub-ui/node_modules
cache.pip
0.00s
5.12sadding /home/travis/.cache/pip to cache
$ python --version
Python 2.7.14
$ pip --version
pip 9.0.1 from /home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages (python 2.7)
install.1
31.85s$ npm install -g @angular/cli
/home/travis/.nvm/versions/node/v8.9.1/bin/ng -> /home/travis/.nvm/versions/node/v8.9.1/lib/node_modules/@angular/cli/bin/ng
> node-sass@4.7.2 install /home/travis/.nvm/versions/node/v8.9.1/lib/node_modules/@angular/cli/node_modules/node-sass
> node scripts/install.js
Downloading binary from https://github.com/sass/node-sass/releases/download/v4.7.2/linux-x64-57_binding.node
Download complete
Binary saved to /home/travis/.nvm/versions/node/v8.9.1/lib/node_modules/@angular/cli/node_modules/node-sass/vendor/linux-x64-57/binding.node
Caching binary to /home/travis/.npm/node-sass/4.7.2/linux-x64-57_binding.node
> uglifyjs-webpack-plugin@0.4.6 postinstall /home/travis/.nvm/versions/node/v8.9.1/lib/node_modules/@angular/cli/node_modules/webpack/node_modules/uglifyjs-webpack-plugin
> node lib/post_install.js
> node-sass@4.7.2 postinstall /home/travis/.nvm/versions/node/v8.9.1/lib/node_modules/@angular/cli/node_modules/node-sass
> node scripts/build.js
Binary found at /home/travis/.nvm/versions/node/v8.9.1/lib/node_modules/@angular/cli/node_modules/node-sass/vendor/linux-x64-57/binding.node
Testing binary
Binary is fine
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.1.3 (node_modules/@angular/cli/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
+ @angular/cli@1.6.5
added 1258 packages in 31.057s
install.2
9.58s$ npm install
npm WARN @angular/flex-layout@2.0.0-beta.12 requires a peer of @angular/core@~5.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN @angular/flex-layout@2.0.0-beta.12 requires a peer of @angular/common@~5.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN ng2-img-cropper@0.9.0 requires a peer of @angular/common@^4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN ng2-img-cropper@0.9.0 requires a peer of @angular/compiler@^4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN ng2-img-cropper@0.9.0 requires a peer of @angular/core@^4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.1.3 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
added 115 packages, removed 2 packages and updated 1 package in 9.176s
install.3
11.95s$ npm install typescript@'>=2.4.2 <2.5.0'
npm WARN @angular/flex-layout@2.0.0-beta.12 requires a peer of @angular/core@~5.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN @angular/flex-layout@2.0.0-beta.12 requires a peer of @angular/common@~5.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN ng2-img-cropper@0.9.0 requires a peer of @angular/common@^4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN ng2-img-cropper@0.9.0 requires a peer of @angular/compiler@^4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN ng2-img-cropper@0.9.0 requires a peer of @angular/core@^4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.1.3 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
+ typescript@2.4.2
added 117 packages and updated 1 package in 11.562s
install.4
4.24s$ pip install awscli
Collecting awscli
  Using cached awscli-1.14.30-py2.py3-none-any.whl
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting PyYAML<=3.12,>=3.10 (from awscli)
Collecting s3transfer<0.2.0,>=0.1.12 (from awscli)
  Using cached s3transfer-0.1.12-py2.py3-none-any.whl
Collecting botocore==1.8.34 (from awscli)
  Using cached botocore-1.8.34-py2.py3-none-any.whl
Collecting colorama<=0.3.7,>=0.2.5 (from awscli)
  Using cached colorama-0.3.7-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
  Using cached docutils-0.14-py2-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
  Using cached pyasn1-0.4.2-py2.py3-none-any.whl
Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.12->awscli)
  Using cached futures-3.2.0-py2-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.8.34->awscli)
  Using cached jmespath-0.9.3-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.8.34->awscli)
  Using cached python_dateutil-2.6.1-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5 in /home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.8.34->awscli)
Installing collected packages: pyasn1, rsa, PyYAML, futures, jmespath, docutils, python-dateutil, botocore, s3transfer, colorama, awscli
Successfully installed PyYAML-3.12 awscli-1.14.30 botocore-1.8.34 colorama-0.3.7 docutils-0.14 futures-3.2.0 jmespath-0.9.3 pyasn1-0.4.2 python-dateutil-2.6.1 rsa-3.4.2 s3transfer-0.1.12
$ if [ "$TRAVIS_BRANCH" = "development" ]; then ng build --prod --aot --env=development --build-optimizer=false; fi
Node#moveTo was deprecated. Use Container#append.
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated
dylandechant commented 6 years ago

@BanzaiMan ^^ see above

andrewsalveson commented 6 years ago

It is worth noting that even locally this project sits at Node#moveTo was deprecated. Use Container#append. for quite some time with no console output

dylandechant commented 6 years ago

@andrewsalveson i'm not sure i agree. builds are sometimes 6 minutes, and other times they time out. most of the time this is with the exact same source.

BanzaiMan commented 6 years ago

Since we don’t have access to the code, it’s difficult to guess. Have you tried running it in the debug mode? Can you surmise what might be causing the hang?

dylandechant commented 6 years ago

we are now seeing this in a completely different place:

Worker information
hostname: 551fa88c-7360-44e0-ae13-b8268a42f684@1.production-1-worker-com-c-4-gce
version: v3.4.0 https://github.com/travis-ci/worker/tree/ce0440bc30c289a49a9b0c21e4e1e6f7d7825101
instance: travis-job-bb32ba49-6fe7-44ae-a61f-fb92f3599381 travis-ci-garnet-trusty-1512502259-986baf0 (via amqp)
startup: 21.104721038s
mode of ‘/usr/local/clang-5.0.0/bin’ changed from 0777 (rwxrwxrwx) to 0775 (rwxrwxr-x)
system_info
Build system information
Build language: python
Build group: stable
Build dist: trusty
Build id: 64219546
Job id: 107465646
Runtime kernel version: 4.4.0-101-generic
travis-build version: c129335c3
Build image provisioning date and time
Tue Dec  5 19:58:13 UTC 2017
Operating System Details
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:    14.04
Codename:   trusty
Cookbooks Version
7c2c6a6 https://github.com/travis-ci/travis-cookbooks/tree/7c2c6a6
git version
git version 2.15.1
bash version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
gcc version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:38 2017
 OS/Arch:      linux/amd64
Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:20 2017
 OS/Arch:      linux/amd64
 Experimental: false
clang version
clang version 5.0.0 (tags/RELEASE_500/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/clang-5.0.0/bin
jq version
jq-1.5
bats version
Bats 0.4.0
shellcheck version
0.4.6
shfmt version
v2.0.0
ccache version
ccache version 3.1.9
Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2011 Joel Rosdahl
This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation; either version 3 of the License, or (at your option) any later
version.
cmake version
cmake version 3.9.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
heroku version
heroku-cli/6.14.39-addc925 (linux-x64) node-v9.2.0
imagemagick version
Version: ImageMagick 6.7.7-10 2017-07-31 Q16 http://www.imagemagick.org
md5deep version
4.2
mercurial version
Mercurial Distributed SCM (version 4.2.2)
(see https://mercurial-scm.org for more information)
Copyright (C) 2005-2017 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
mysql version
mysql  Ver 14.14 Distrib 5.6.33, for debian-linux-gnu (x86_64) using  EditLine wrapper
openssl version
OpenSSL 1.0.1f 6 Jan 2014
packer version
Packer v1.0.2
Your version of Packer is out of date! The latest version
is 1.1.2. You can update by downloading from www.packer.io
postgresql client version
psql (PostgreSQL) 9.6.6
ragel version
Ragel State Machine Compiler version 6.8 Feb 2013
Copyright (c) 2001-2009 by Adrian Thurston
subversion version
svn, version 1.8.8 (r1568071)
   compiled Aug 10 2017, 17:20:39 on x86_64-pc-linux-gnu
Copyright (C) 2013 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/
The following repository access (RA) modules are available:
* ra_svn : Module for accessing a repository using the svn network protocol.
  - with Cyrus SASL authentication
  - handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
  - handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using serf.
  - using serf 1.3.3
  - handles 'http' scheme
  - handles 'https' scheme
sudo version
Sudo version 1.8.9p5
Configure options: --prefix=/usr -v --with-all-insults --with-pam --with-fqdn --with-logging=syslog --with-logfac=authpriv --with-env-editor --with-editor=/usr/bin/editor --with-timeout=15 --with-password-timeout=0 --with-passprompt=[sudo] password for %p:  --without-lecture --with-tty-tickets --disable-root-mailer --enable-admin-flag --with-sendmail=/usr/sbin/sendmail --with-timedir=/var/lib/sudo --mandir=/usr/share/man --libexecdir=/usr/lib/sudo --with-sssd --with-sssd-lib=/usr/lib/x86_64-linux-gnu --with-selinux
Sudoers policy plugin version 1.8.9p5
Sudoers file grammar version 43
Sudoers path: /etc/sudoers
Authentication methods: 'pam'
Syslog facility if syslog is being used for logging: authpriv
Syslog priority to use when user authenticates successfully: notice
Syslog priority to use when user authenticates unsuccessfully: alert
Send mail if the user is not in sudoers
Use a separate timestamp for each user/tty combo
Lecture user the first time they run sudo
Root may run sudo
Allow some information gathering to give useful error messages
Require fully-qualified hostnames in the sudoers file
Visudo will honor the EDITOR environment variable
Set the LOGNAME and USER environment variables
Length at which to wrap log file lines (0 for no wrap): 80
Authentication timestamp timeout: 15.0 minutes
Password prompt timeout: 0.0 minutes
Number of tries to enter a password: 3
Umask to use or 0777 to use user's: 022
Path to mail program: /usr/sbin/sendmail
Flags for mail program: -t
Address to send mail to: root
Subject line for mail messages: *** SECURITY information for %h ***
Incorrect password message: Sorry, try again.
Path to authentication timestamp dir: /var/lib/sudo
Default password prompt: [sudo] password for %p: 
Default user to run commands as: root
Value to override user's $PATH with: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Path to the editor for use by visudo: /usr/bin/editor
When to require a password for 'list' pseudocommand: any
When to require a password for 'verify' pseudocommand: all
File descriptors >= 3 will be closed before executing a command
Environment variables to check for sanity:
    TZ
    TERM
    LINGUAS
    LC_*
    LANGUAGE
    LANG
    COLORTERM
Environment variables to remove:
    RUBYOPT
    RUBYLIB
    PYTHONUSERBASE
    PYTHONINSPECT
    PYTHONPATH
    PYTHONHOME
    TMPPREFIX
    ZDOTDIR
    READNULLCMD
    NULLCMD
    FPATH
    PERL5DB
    PERL5OPT
    PERL5LIB
    PERLLIB
    PERLIO_DEBUG 
    JAVA_TOOL_OPTIONS
    SHELLOPTS
    GLOBIGNORE
    PS4
    BASH_ENV
    ENV
    TERMCAP
    TERMPATH
    TERMINFO_DIRS
    TERMINFO
    _RLD*
    LD_*
    PATH_LOCALE
    NLSPATH
    HOSTALIASES
    RES_OPTIONS
    LOCALDOMAIN
    CDPATH
    IFS
Environment variables to preserve:
    JAVA_HOME
    TRAVIS
    CI
    DEBIAN_FRONTEND
    XAUTHORIZATION
    XAUTHORITY
    PS2
    PS1
    PATH
    LS_COLORS
    KRB5CCNAME
    HOSTNAME
    HOME
    DISPLAY
    COLORS
Locale to use while parsing sudoers: C
Directory in which to store input/output logs: /var/log/sudo-io
File in which to store the input/output log: %{seq}
Add an entry to the utmp/utmpx file when allocating a pty
PAM service name to use
PAM service name to use for login shells
Create a new PAM session for the command to run in
Maximum I/O log sequence number: 0
Local IP address and netmask pairs:
    10.240.0.28/255.255.255.255
    172.17.0.1/255.255.0.0
Sudoers I/O plugin version 1.8.9p5
gzip version
gzip 1.6
Copyright (C) 2007, 2010, 2011 Free Software Foundation, Inc.
Copyright (C) 1993 Jean-loup Gailly.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License .
There is NO WARRANTY, to the extent permitted by law.
Written by Jean-loup Gailly.
zip version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.
Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.
Compiled with gcc 4.8.2 for Unix (Linux ELF) on Oct 21 2013.
Zip special compilation options:
    USE_EF_UT_TIME       (store Universal Time)
    BZIP2_SUPPORT        (bzip2 library version 1.0.6, 6-Sept-2010)
        bzip2 code and library copyright (c) Julian R Seward
        (See the bzip2 license for terms of use)
    SYMLINK_SUPPORT      (symbolic links supported)
    LARGE_FILE_SUPPORT   (can read and write large files on file system)
    ZIP64_SUPPORT        (use Zip64 to store large files in archives)
    UNICODE_SUPPORT      (store and read UTF-8 Unicode paths)
    STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field)
    UIDGID_NOT_16BIT     (old Unix 16-bit UID/GID extra field not used)
    [encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3)
Encryption notice:
    The encryption code of this program is not copyrighted and is
    put in the public domain.  It was originally written in Europe
    and, to the best of our knowledge, can be freely distributed
    in both source and object forms from any country, including
    the USA under License Exception TSU of the U.S. Export
    Administration Regulations (section 740.13(e)) of 6 June 2002.
Zip environment options:
             ZIP:  [none]
          ZIPOPT:  [none]
vim version
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Nov 24 2016 16:43:18)
Included patches: 1-52
Extra patches: 8.0.0056
Modified by pkg-vim-maintainers@lists.alioth.debian.org
Compiled by buildd@
Huge version without GUI.  Features included (+) or not (-):
+acl             +farsi           +mouse_netterm   +syntax
+arabic          +file_in_path    +mouse_sgr       +tag_binary
+autocmd         +find_in_path    -mouse_sysmouse  +tag_old_static
-balloon_eval    +float           +mouse_urxvt     -tag_any_white
-browse          +folding         +mouse_xterm     -tcl
++builtin_terms  -footer          +multi_byte      +terminfo
+byte_offset     +fork()          +multi_lang      +termresponse
+cindent         +gettext         -mzscheme        +textobjects
-clientserver    -hangul_input    +netbeans_intg   +title
-clipboard       +iconv           +path_extra      -toolbar
+cmdline_compl   +insert_expand   -perl            +user_commands
+cmdline_hist    +jumplist        +persistent_undo +vertsplit
+cmdline_info    +keymap          +postscript      +virtualedit
+comments        +langmap         +printer         +visual
+conceal         +libcall         +profile         +visualextra
+cryptv          +linebreak       +python          +viminfo
+cscope          +lispindent      -python3         +vreplace
+cursorbind      +listcmds        +quickfix        +wildignore
+cursorshape     +localmap        +reltime         +wildmenu
+dialog_con      -lua             +rightleft       +windows
+diff            +menu            -ruby            +writebackup
+digraphs        +mksession       +scrollbind      -X11
-dnd             +modify_fname    +signs           -xfontset
-ebcdic          +mouse           +smartindent     -xim
+emacs_tags      -mouseshape      -sniff           -xsmp
+eval            +mouse_dec       +startuptime     -xterm_clipboard
+ex_extra        +mouse_gpm       +statusline      -xterm_save
+extra_search    -mouse_jsbterm   -sun_workshop    -xpm
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
  fall-back for $VIM: "/usr/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H     -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1      
Linking: gcc   -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed -o vim        -lm -ltinfo -lnsl  -lselinux  -lacl -lattr -lgpm -ldl    -L/usr/lib/python2.7/config-x86_64-linux-gnu -lpython2.7 -lpthread -ldl -lutil -lm -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions      
iptables version
iptables v1.4.21
curl version
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
wget version
GNU Wget 1.15 built on linux-gnu.
rsync version
rsync  version 3.1.0  protocol version 31
gimme version
v1.2.0
nvm version
0.33.6
perlbrew version
/home/travis/perl5/perlbrew/bin/perlbrew  - App::perlbrew/0.80
phpenv version
rbenv 1.1.1-25-g6aa70b6
rvm version
rvm 1.29.3 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
default ruby version
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
CouchDB version
couchdb 1.6.1
ElasticSearch version
5.5.0
Installed Firefox version
firefox 56.0.2
MongoDB version
MongoDB 3.4.10
PhantomJS version
2.1.1
Pre-installed PostgreSQL versions
9.2.24
9.3.20
9.4.15
9.5.10
9.6.6
RabbitMQ Version
3.6.14
Redis version
redis-server 4.0.6
riak version
2.2.3
Pre-installed Go versions
1.7.4
ant version
Apache Ant(TM) version 1.9.3 compiled on April 8 2014
mvn version
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
Maven home: /usr/local/maven-3.5.2
Java version: 1.8.0_151, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-98-generic", arch: "amd64", family: "unix"
gradle version
------------------------------------------------------------
Gradle 4.0.1
------------------------------------------------------------
Build time:   2017-07-07 14:02:41 UTC
Revision:     38e5dc0f772daecca1d2681885d3d85414eb6826
Groovy:       2.4.11
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.8.0_151 (Oracle Corporation 25.151-b12)
OS:           Linux 4.4.0-98-generic amd64
lein version
Leiningen 2.8.1 on Java 1.8.0_151 Java HotSpot(TM) 64-Bit Server VM
Pre-installed Node.js versions
v4.8.6
v6.12.0
v6.12.1
v8.9
v8.9.1
phpenv versions
  system
  5.6
* 5.6.32 (set by /home/travis/.phpenv/version)
  7.0
  7.0.25
  7.1
  7.1.11
  hhvm
  hhvm-stable
composer --version
Composer version 1.5.2 2017-09-11 16:59:25
Pre-installed Ruby versions
ruby-2.2.7
ruby-2.3.4
ruby-2.4.1
removed ‘/etc/apt/sources.list.d/basho_riak.list’
W: http://ppa.launchpad.net/couchdb/stable/ubuntu/dists/trusty/Release.gpg: Signature by key 15866BAFD9BCC4F3C1E0DFC7D69548E1C17EAB57 uses weak digest algorithm (SHA1)
127.0.0.1 localhost nettuno travis vagrant
127.0.1.1 travis-job-bb32ba49-6fe7-44ae-a61f-fb92f3599381 travis-job-bb32ba49-6fe7-44ae-a61f-fb92f3599381 ip4-loopback trusty64
Installing an SSH key from: default repository key
Key fingerprint: a0:d2:2c:6d:bc:8e:db:f6:c2:4a:13:0c:1f:2d:61:1d
git.checkout
1.28s$ git clone --depth=50 --branch=pre-dev git@github.com:PerkinsAndWill-IO/hub-ui.git PerkinsAndWill-IO/hub-ui
Cloning into 'PerkinsAndWill-IO/hub-ui'...
Warning: Permanently added the RSA host key for IP address '192.30.255.112' to the list of known hosts.
$ cd PerkinsAndWill-IO/hub-ui
$ git checkout -qf 544d3039afd4a6a73853aa3b0926c6937e9ea3bb
Setting environment variables from repository settings
[redeacted]
0.01s$ source ~/virtualenv/python2.7/bin/activate
Setting up build cache
$ export CASHER_DIR=$HOME/.casher
0.20s$ Installing caching utilities
0.00s
5.21sattempting to download cache archive
fetching pre-dev/cache-linux-trusty-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855--python-2.7.tgz
found cache
0.00s
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated
BanzaiMan commented 6 years ago

That is really unnerving. Could you collect the host information for these bad builds (under "Worker information") and send that to us? Maybe there is an emerging pattern.

Thank you!

dylandechant commented 6 years ago

absolutely @BanzaiMan 

Worker information
hostname: 0e446963-556d-4fc6-b0df-ab951b39b102@1.production-1-worker-com-c-2-gce
version: v3.4.0 https://github.com/travis-ci/worker/tree/ce0440bc30c289a49a9b0c21e4e1e6f7d7825101
instance: travis-job-aa4001ee-7d63-42f5-a878-dfa5f4d8aaea travis-ci-garnet-trusty-1512502259-986baf0 (via amqp)
startup: 21.15878866s
mode of ‘/usr/local/clang-5.0.0/bin’ changed from 0777 (rwxrwxrwx) to 0775 (rwxrwxr-x)
lilic commented 6 years ago

We are seeing similar problems but only on the new Trusty env, not on the old images. It often times out for us as well, noticed this since December, but it has been worse lately as it seems the builds use the new images more often?

Let me know if you want me to open a new issue? Also our project is open sourced so let me know if you need any additional info. Thanks!

BanzaiMan commented 6 years ago

@LiliC Thanks for the comments.

@dylandechant How about your case? Does the old image exhibit the same problem?

dylandechant commented 6 years ago

@BanzaiMan yes, that is correct. its becoming quite a problem

soulshake commented 6 years ago

@dylandechant @LiliC Are your affected repos specifying any caches in their .travis.yml? If so, does clearing them help?

jongear commented 6 years ago

To add to this. We periodically see rake failures on trusty. Restarting the build usually resolves the issue.

vm settings

language: node_js
node_js:
  - "8"
cache:
  directories:
    - "node_modules"

error

4.81s$ rvm use default
Using /home/travis/.rvm/gems/ruby-2.4.1
** Updating RubyGems to the latest version for security reasons. **
** If you need an older version, you can downgrade with 'gem update --system OLD_VERSION'. **
Updating rubygems-update
Successfully installed rubygems-update-2.7.4
Installing RubyGems 2.7.4
Bundler 1.16.0 installed
RubyGems 2.7.4 installed
Regenerating binstubs
=== 2.6.14 / 2017-10-09
Security fixes:
* Whitelist classes and symbols that are in loaded YAML.
  See CVE-2017-0903 for full details.
  Fix by Aaron Patterson.
=== 2.6.13 / 2017-08-27
Security fixes:
* Fix a DNS request hijacking vulnerability. (CVE-2017-0902)
  Discovered by Jonathan Claudius, fix by Samuel Giddins.
* Fix an ANSI escape sequence vulnerability. (CVE-2017-0899)
  Discovered by Yusuke Endoh, fix by Evan Phoenix.
* Fix a DOS vulnerability in the `query` command. (CVE-2017-0900)
  Discovered by Yusuke Endoh, fix by Samuel Giddins.
* Fix a vulnerability in the gem installer that allowed a malicious gem
  to overwrite arbitrary files. (CVE-2017-0901)
  Discovered by Yusuke Endoh, fix by Samuel Giddins.
=== 2.6.12 / 2017-04-30
Bug fixes:
* Fix test_self_find_files_with_gemfile to sort expected files. Pull
  request #1880 by Kazuaki Matsuo.
* Fix issue for MinGW / MSYS2 builds and testing. Pull request #1879 by
  MSP-Greg.
* Fix gem open to open highest version number rather than lowest. Pull
  request #1877 by Tim Pope.
* Add a test for requiring a default spec as installed by the ruby
  installer. Pull request #1899 by Samuel Giddins.
* Fix broken --exact parameter to gem command. Pull request #1873 by Jason
  Frey.
* [Installer] Generate backwards-compatible binstubs. Pull request #1904
  by Samuel Giddins.
* Fix pre-existing source recognition on add action. Pull request #1883 by
  Jonathan Claudius.
* Prevent negative IDs in output of #inspect. Pull request #1908 by Vít
  Ondruch.
* Allow Gem.finish_resolve to respect already-activated specs. Pull
  request #1910 by Samuel Giddins.
------------------------------------------------------------------------------
RubyGems installed the following executables:
    /home/travis/.rvm/rubies/ruby-2.4.1/bin/gem
    /home/travis/.rvm/rubies/ruby-2.4.1/bin/bundle
RubyGems system software updated
$ ruby --version
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
$ rvm --version
rvm 1.29.3 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
$ bundle --version
Bundler version 1.16.0
$ gem --version
2.7.4
0.16s$ rake
rake aborted!
No Rakefile found (looking for: rakefile, Rakefile, rakefile.rb, Rakefile.rb)
/home/travis/.rvm/gems/ruby-2.4.1@global/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
(See full trace by running task with --trace)
The command "rake" exited with 1.
Done. Your build exited with 1.
dylandechant commented 6 years ago

@soulshake we cleared the cache over the weekend. it seems it worked for a bit but we are back to where we are.

we can kind of anticipate a failed build as passing ones usually build in 6 minutes, once we get near 10 we are learning to just cancel and restart

lilic commented 6 years ago

@soulshake Nope, we do not use cache at all. https://github.com/kinvolk/habitat-operator/blob/master/.travis.yml

It could be a completely different problem to what the others are experiencing. What we have been seeing is the same msg as the above mentioned one:

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

Or these kind of errors https://travis-ci.org/kinvolk/habitat-operator/builds/332325306#L579 and https://travis-ci.org/kinvolk/habitat-operator/builds/332787305#L584 and many more. At first I was thinking it's just minikube, but then I noticed it only happened when it was using the new images. When restarting and if the image was not using Trusty it passed. I tested a branch the other day by using the deprecated image and the builds passed every time it seemed.

Hope that was of some help!

soulshake commented 6 years ago

@dylandechant @LiliC Of the jobs that fail with the log silence timeout (No output has been received in the last 10m0s...), do they always fail at the exact same place?

dylandechant commented 6 years ago

@soulshake 99% fail right here:

$ if [ "$TRAVIS_BRANCH" = "development" ]; then ng build --prod --aot --env=development --build-optimizer=false; fi
Node#moveTo was deprecated. Use Container#append.
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

but we have had one outlier:

5.21sattempting to download cache archive
fetching pre-dev/cache-linux-trusty-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855--python-2.7.tgz
found cache
0.00s
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

also @soulshake do i need to get my repo flagged to use group: deprecated-2017Q4?

edit: I should also add simply restarting the build will sometimes pass, this is without any code editions/modifications to the repo

lilic commented 6 years ago

@soulshake Two different places from what I remember, some in the test setup (minikube waiting for nodes to be ready I think) and some when running the tests itself. But don't have the logs of those jobs sadly. Today has been very stable, haven't seen any problems so far and we did not change anything in our test setup or touched part of the code that effects the e2e tests.

soulshake commented 6 years ago

Can anyone point me to a repo with active builds that are regularly encountering this problem? I'd love to investigate further, and having a running job to look at would be very helpful.

lilic commented 6 years ago

@soulshake I tried restarting a build to reproduce the problem, but it seems to be stuck in booting status for 5+ minutes now.... https://travis-ci.org/kinvolk/habitat-operator/builds/289930904

jongear commented 6 years ago

My builds are also stuck in booting. I'm hoping that means the travis team is rolling out a patch

dylandechant commented 6 years ago

@soulshake https://travis-ci.com/PerkinsAndWill-IO/hub-ui/builds/64222197

lilic commented 6 years ago

@soulshake seems to have passed now, but next time I find a failing one I can keep it and not trigger a new if that helps you?

jongear commented 6 years ago

@soulshake

random rake fail from travis bootup https://travis-ci.com/jdktech/hub-mobile-api/builds/64316198

build stuck booting https://travis-ci.com/jdktech/hub-mobile-api/jobs/107637969

soulshake commented 6 years ago

We have a theory that this may be related to networking issues faced by our upstream provider for sudo-enabled builds (Google Cloud). Their status page mentions a networking issue today, though most reports I've seen indicate that this behavior started before today.

Does anyone have a case where a build suddenly started failing with No output has been received in the last 10m0s which did not run on gce?

You can see this (gce, ec2, etc) under Worker information in the job logs (e.g. here):

Worker information
hostname: 57012187-2608-4011-89fd-9a59573b63df@1.production-3-worker-org-c-1-gce  <----

Also, when including a link to a repo with this problem, please let me know if I can restart your failed builds for testing purposes.

alexmclean commented 6 years ago

We worked around this by adding "travis_wait 30" (or however long your tests normally take to run) before the command that ran our tests in travis.yml.

This seemed to be happening for us because our tests took 13-14 minutes, and Travis seems to not write to STD OUT anymore while the tests are happening, only after they finish. Some warnings would normally be printed in the logs which would be considered "output", but because these were being printed at the end, the job would see nothing for 10 minutes and fail.

lilic commented 6 years ago

@soulshake Thanks for the info! I guess that could indeed be the problem, and the new images be just a coincidence.

Will let you know if we run into more problems that are not related to the builds using gce. Thanks again!

lilic commented 6 years ago

@soulshake Ok we found one, it is using gce, but in case you want to have a look, this is the build https://travis-ci.org/kinvolk/habitat-operator/builds/333786017?utm_source=email&utm_medium=notification

cotsog commented 6 years ago

@dylandechant @LiliC: based on @alexmclean's comment, could you also try adding travis_wait 30 in front of the commands that are stalling e.g.

- if [ "$TRAVIS_BRANCH" = "development" ]; then travis_wait 30 ng build --prod --aot --env=development --build-optimizer=false; fi

Please let us know if it helps.

@jongear your issue looks different, could you send an e-mail including build links to support [at] travis-ci [dot] com so we can have a look?

torokati44 commented 6 years ago

I've also been getting these 10-minute timeout errors in the last couple of days. For me it happens while pulling an image from Docker Hub, so it can definitely be caused by subpar network conditions.

See for example: https://travis-ci.org/inet-framework/inet/builds/334144957 https://travis-ci.org/inet-framework/inet/builds/334144923

cotsog commented 6 years ago

@dylandechant @LiliC @torokati44 I've made a backend change specific to your organizations that will hopefully fix your issue. In brief, I've put your org on a different filtering mechanism for your log.

Hence, could you try to restart a failing build or start a new one? Please let us know if you are still seeing the behavior or not and we will continue from there.

Thank you for your patience with this.

lilic commented 6 years ago

@cotsog Thanks, much appreciated! We will let you know if we experience the problem again from now on.

torokati44 commented 6 years ago

Thank you, @cotsog. I haven't experienced this issue since the builds linked above. The docker pulls seemed to just complete faster. I'll update if it occurs again.

torokati44 commented 6 years ago

Here it is again, @cotsog: https://travis-ci.org/inet-framework/inet/builds/336990829

hwdsl2 commented 6 years ago

This timeout issue has been observed starting around 2017 across all my GitHub repos. It had annoyed me to the point that I disabled the (originally scheduled) CRON builds because I had to restart them every time and they keep erroring.

Recently with the latest Travis CI update the builds fail consistently for one of my repos. For this repo, the timeout always occurs around the line "Generating OBJ.linux.x86_64/.../Makefile". Even after switching back to "deprecated-2017Q4" I still had to restart the build more than 10 times before one becomes successful.

Example build logs: https://travis-ci.org/hwdsl2/setup-ipsec-vpn/builds/337114659 https://travis-ci.org/hwdsl2/setup-ipsec-vpn/builds/337078414 https://travis-ci.org/hwdsl2/setup-ipsec-vpn/builds/337066099 https://travis-ci.org/hwdsl2/setup-ghost-blog/builds/315692990 https://travis-ci.org/hwdsl2/setup-ghost-blog/builds/318979235

kratsg commented 6 years ago

Hi, here's an example job that fails on the docker pulls consistently (https://travis-ci.org/UCATLAS/xAODAnaHelpers/jobs/327270108) and it is a gce. In particular, this is seeing consistent failures that restarting jobs fix: https://travis-ci.org/UCATLAS/xAODAnaHelpers/. It's screwing up deployments for us.

Other jobs:

cotsog commented 6 years ago

Hey all,

I'm terribly sorry for the lack of update here. This issue seems to be a moving target and we are still trying to get to the bottom of it.

That being said, could you also try to add the following to your .travis.yml file and let me know if it helps?

filter_secrets: false

Be advised that doing so will disable our feature that's currently preventing commands from leaking environment variables. In general, commands can leak when they return an error. If your commands are used properly right now, they shouldn't leak anything. It is meant as a temporary measure to help us pinpoint the issue at play here.

Thanks again for your continued collaboration and please report the results you get in this issue.

torokati44 commented 6 years ago

It happened again today, this time after a successful docker pull. Which is weird. https://travis-ci.org/inet-framework/inet/builds/338115796 I'm gonna try your suggestion soon - am watching the Falcon Heavy launch right now... :D :rocket:

torokati44 commented 6 years ago

I triggered two builds with the .travis.yml adjustment. One by pushing, one by cron. The first one succeeded, the second failed the exact same way as last time. This might be only a coincidence though.

https://travis-ci.org/torokati44/inet/builds/338233473 https://travis-ci.org/torokati44/inet/builds/338234253

torokati44 commented 6 years ago

Just an update: Suspiciously similar error again today, in the mainline repo, without the filter_secrets: false addition. Only the last job errored, at the same part of our script, when testing was supposed to start.

https://travis-ci.org/inet-framework/inet/builds/339091213

kratsg commented 6 years ago

The build for the repository seems to went through the first time -- I'll do a restart a few more times and see if it happens. But it naively seems like filter_secrets is fixing it.

kratsg commented 6 years ago

I'm not seeing the errors anymore... was there a fix propagated?

torokati44 commented 6 years ago

I still do, both with and without filter_secrets: false:

https://travis-ci.org/torokati44/inet/builds/340269935 https://travis-ci.org/inet-framework/inet/builds/340188344

zoq commented 6 years ago

We are seeing similar problems, could be a completely different problem to what the others are experiencing. However, the message is the same:

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

Example build logs:

https://travis-ci.org/mlpack/mlpack/jobs/341477198 https://travis-ci.org/mlpack/mlpack/jobs/341432576 https://travis-ci.org/mlpack/mlpack/jobs/340971086 https://travis-ci.org/mlpack/mlpack/jobs/340954040

At least one of the three builds fails every time:

https://travis-ci.org/mlpack/mlpack/builds/341477194

torokati44 commented 6 years ago

We did not experience this error in our repo in the last 10 days. Today it returned. https://travis-ci.org/inet-framework/inet/builds/345355584 Strangely, the very last job seems to be the most likely to fail. Can this have something to do with it being the last, or just with what it's doing?

epiclabsDASH commented 6 years ago

Same problem here. One of the steps of our build process involves deploying some files to an ftp server. After uploading the first 1-2 small files, process is stalled until a time out issue is raised.

https://travis-ci.org/Dash-Industry-Forum/dash.js/builds/350581541

BanzaiMan commented 6 years ago

Thanks for the patience, everyone. We continue to investigate this issue as our priority.

henryruhs commented 6 years ago

I can report same timeout issues to all of my lftp powered jobs...

https://travis-ci.org/redaxscript/redaxscript-demo-sync