`docker pull` stalls, times out, and error the job

dylandechant commented 6 years ago

We have been seeing significant inconsistencies in travis since Saturday 1/20/2018. some builds error with timeouts while when restarting the same build they finish in ~6 minutes.

look at the screen shot of this log:

this is getting to the point where it is slowing down development as it is creating a queue of PRs that are constantly failing and when restarted may or may not pass but every once in a while they do.

Please advise.

Thanks

epiclabsDASH commented 6 years ago

@BanzaiMan. Sorry to bother, is there any news regarding this? In our case it has turned out from being something that happened from time to time to be an issue that happens always. In fact we had to deactivate our CI workflow to avoid bigger issues.

BanzaiMan commented 6 years ago

@redaxmedia I mentioned in https://github.com/travis-ci/travis-ci/issues/9112#issuecomment-373031537 that FTP traffic appears to be especially unreliable (with an estimated 50% failures).

@epiclabsDASH We continue to investigate the issue. We apologize for not having better news to share yet.

kratsg commented 6 years ago

Thanks a lot @BanzaiMan -- I'm continuing to see this still with my docker pulls:

Let me know what other information you need.

BanzaiMan commented 6 years ago

@kratsg Where are you pulling Docker images from, and how big are they?

henryruhs commented 6 years ago

Sorry @BanzaiMan - but at this point I start to get angry... no process on this issue since months!

A lot of people cannot run their deployment jobs - so why is nobody putting this on high priority?

kratsg commented 6 years ago

@BanzaiMan - hi. So I'm pulling multiple tags (corresponding to multiple releases of physics software) from two repositories on docker hub:

These images are O(2GB) and are based on top of CERN's SLC6 images here (https://hub.docker.com/r/cern/slc6-base/).

BanzaiMan commented 6 years ago

@redaxmedia We apologize for the frustration. Have you had a chance to test filter_secrets: false to see if that mitigates the problem in your case?

BanzaiMan commented 6 years ago

@redaxmedia Actually, your issue is the ftp traffic issue, so that's being investigated separately. https://github.com/travis-ci/travis-ci/issues/9112#issuecomment-375092252

epiclabsDASH commented 6 years ago

Also our one, dash.js project build process, is related with ftp. Doesn't seem to be related with long transfers, it fails after uploading two small files. I will follow up the progress on #9112

BanzaiMan commented 6 years ago

@kratsg How often do you see issues with docker pull these past few days?

kratsg commented 6 years ago

@kratsg How often do you see issues with docker pull these past few days?

None so far for the past 5 days.

BanzaiMan commented 6 years ago

@dylandechant What kind of operation does your ng perform? Is it trying to pull a Docker image from somewhere?

I see that recent builds are passing. Are you experiencing the error recently?

BanzaiMan commented 6 years ago

I've changed the subject, so that the issue has a good focus on what we are dealing with. Thank you.

dylandechant commented 6 years ago

@BanzaiMan ng is not related to docker, its a build process for us. no errors on our end as of recent

taqtiqa-mark commented 6 years ago

@dylandechant, @LiliC, @kratsg, To try to eliminate the Docker daemon as the source could you try the following rkt fetch in the builds that use the Docker images that reliably fail:

rkt fetch --insecure-options=image docker://<your-container-image>

This might isolate the issue to docker-hub, AWS, GCE, and eliminate the docker client/daemon as the source of the behaviour.

@zoq, @torokati44: In the builds you report as reliably failing, is it possible the up/download volume for the whole >4GB by the time it fails? It'd be nice to eliminate throttling or hard limits as the source of the pull's halting, but so far it is not clear (see #9384) if there are Travis-CI.org throttles or hard limits on bandwidth.

torokati44 commented 6 years ago

is it possible the up/download volume for the whole >4GB by the time it fails?

It very much is, however, all the errors I've seen in the past couple of weeks (in our repo) happened after all network activity (git clone, docker pull) was done, successfully. There used to be different errors before, during docker pull, but not anymore. I think what we have now is not related to networking, nor Docker, so the current topic of this issue is not that relevant to us anymore, I think.

I'm actually starting to suspect that there might be a deadlock caused by a race condition in our test runner script, but I'll have to try a few more things to confirm this. It's weird though, that it only ever happens on Travis, not even that rarely, and almost exclusively on one particular job in each build. (And all jobs in each build do the exact same network operations as far as I know.)

blimmer commented 6 years ago

We're seeing this same problem. Same worker config:

travis_fold:start:worker_info^MWorker information
hostname: 9590591c-6388-4576-bd18-6092a49cbce1@1.production-1-worker-com-a-1-gce
version: v3.6.0 https://github.com/travis-ci/worker/tree/170b2a0bb43234479fd1911ba9e4dbcc36dadfad
instance: travis-job-171153ff-de95-438f-b737-c12dc46c8a2c travis-ci-garnet-trusty-1512502259-986baf0 (via amqp)
startup: 26.837803504s
travis_fold:end:worker_info^Mtravis_fold:start:system_info^MBuild system information
Build language: ruby
Build group: stable
Build dist: trusty
Build id: 74753960
Job id: 126467433
Runtime kernel version: 4.4.0-101-generic
travis-build version: e12e2fc01
Build image provisioning date and time
Tue Dec  5 19:58:13 UTC 2017
Operating System Details
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:        14.04
Codename:       trusty
Cookbooks Version
7c2c6a6 https://github.com/travis-ci/travis-cookbooks/tree/7c2c6a6
git version
git version 2.15.1
bash version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
gcc version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:38 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:20 2017
 OS/Arch:      linux/amd64
 Experimental: false

Our build runs a docker pull (from our Amazon ECR repo) and a bunch of git clones (of terraform modules located in a private repo).

Over the past few weeks, it has become unusably slow (over 25 minutes to pull and clone all the pieces we need).

cc @endzyme @erindatkinson

blimmer commented 6 years ago

I also just saw this extreme slowness with github in a debug session. I turned up the logging a ton and here's what I saw.

The command is:

curl -sSfL https://github.com/docker/compose/releases/download/1.11.2/docker-compose-`uname -s`-`uname -m` > docker-compose

This curl took over 4 minutes! Most of the time spent was in DNS resolution, which could be the source of the problem.

Really slow logs

travis@travis-job-db364705-789c-4323-8b27-625424a28749:~/build/Ibotta/microservice_scorecard_generator$ time curl -sSfLv https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker- compose-`uname -s`-`uname -m` > docker-compose * Hostname was NOT found in DNS cache * Trying 192.30.253.113... * connect to 192.30.253.113 port 443 failed: Connection timed out * Trying 192.30.253.112... * Connected to github.com (192.30.253.112) port 443 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): } [data not shown] * SSLv3, TLS handshake, Server hello (2): { [data not shown] * SSLv3, TLS handshake, CERT (11): { [data not shown] * SSLv3, TLS handshake, Server key exchange (12): { [data not shown] * SSLv3, TLS handshake, Server finished (14): { [data not shown] * SSLv3, TLS handshake, Client key exchange (16): } [data not shown] * SSLv3, TLS change cipher, Client hello (1): } [data not shown] * SSLv3, TLS handshake, Finished (20): } [data not shown] * SSLv3, TLS change cipher, Client hello (1): { [data not shown] * SSLv3, TLS handshake, Finished (20): { [data not shown] * SSL connection using ECDHE-RSA-AES128-GCM-SHA256 * Server certificate: * subject: businessCategory=Private Organization; 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.6 0.2.1.2=Delaware; serialNumber=5157550; C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github .com * start date: 2018-05-08 00:00:00 GMT * expire date: 2020-06-03 12:00:00 GMT * subjectAltName: github.com matched * issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA * SSL certificate verify ok. > GET /docker/compose/releases/download/1.11.2/docker-compose-Linux-x86_64 HTTP/1.1 > User-Agent: curl/7.35.0 > Host: github.com > Accept: */* > < HTTP/1.1 302 Found * Server GitHub.com is not blacklisted < Server: GitHub.com < Date: Tue, 29 May 2018 16:46:54 GMT < Content-Type: text/html; charset=utf-8 < Transfer-Encoding: chunked < Status: 302 Found < Cache-Control: no-cache < Vary: X-PJAX < Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/15045751/c9e8f01e-f84e-11e6 -8a03-4108467e422d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180529%2F us-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180529T164654Z&X-Amz-Expires=300&X-Amz-Signature=750c26c7fb3 a836abe69d5b003c306783d4a426289d15711bdc9a4da1d382a43&X-Amz-SignedHeaders=host&actor_id=0&response-cont ent-disposition=attachment%3B%20filename%3Ddocker-compose-Linux-x86_64&response-content-type=applicatio n%2Foctet-stream < Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Sat, 29 May 2038 16:46:54 -0000; secure ; HttpOnly < Set-Cookie: _gh_sess=R05keXo0T3lORkFtK0paVUUwVmZ2VGs4eU5wN1I0dWhMdnRhdFRpWG03SWV4NE9IdVdmS2lpTkxpUHFR SzZ0TzRDQ3dtMm53ZGx5S1IrbW5UaThSK2s5WEJlWm9Qb1lvcXQxMWhlZ0dOS3oxRjJjdjZHM1lrM1pIeXlwNHB0SjF4SzcyQVRVbEJ wcEdsQm5IRHArVFZEem13QS9VOU9LNEFNYUtrMmFlRHk1Z04xQXQvc3Bndmh5UlpZbU1lWmppLS1nWEpvKzQ0T3JYamhZaHVKdC9KU2 J3PT0%3D--e12ce2ff8794c5cc4899bd227c021ad21a2994e6; path=/; secure; HttpOnly < X-Request-Id: c4dd2fd1-faba-4977-8d95-4d51daf418e7 < X-Runtime: 0.089137 < Strict-Transport-Security: max-age=31536000; includeSubdomains; preload < X-Frame-Options: deny < X-Content-Type-Options: nosniff < X-XSS-Protection: 1; mode=block < Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors" < Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 's elf' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.c om github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-produ ction-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src assets-cdn.github.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: assets-cdn.github. com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com ; manifest-src 'self'; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' as sets-cdn.github.com < X-Runtime-rack: 0.103631 < X-GitHub-Request-Id: B9B4:0FF6:36D1E74:6AB297F:5B0D83FE < * Ignoring the response-body { [data not shown] * Connection #0 to host github.com left intact * Issue another request to this URL: 'https://github-production-release-asset-2e65be.s3.amazonaws.com/1 5045751/c9e8f01e-f84e-11e6-8a03-4108467e422d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJ YAX4CSVEH53A%2F20180529%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180529T164654Z&X-Amz-Expires=300&X -Amz-Signature=750c26c7fb3a836abe69d5b003c306783d4a426289d15711bdc9a4da1d382a43&X-Amz-SignedHeaders=hos t&actor_id=0&response-content-disposition=attachment%3B%20filename%3Ddocker-compose-Linux-x86_64&respon se-content-type=application%2Foctet-stream' * Hostname was NOT found in DNS cache * Trying 52.216.133.115... * connect to 52.216.133.115 port 443 failed: Connection timed out * Failed to connect to github-production-release-asset-2e65be.s3.amazonaws.com port 443: Connection tim ed out * Closing connection 1 curl: (7) Failed to connect to github-production-release-asset-2e65be.s3.amazonaws.com port 443: Connec tion timed out real 4m14.836s user 0m0.012s sys 0m0.008s

Then, after that 4 minute failure, I ran it again and it completed in 1 second.

Fast logs

travis@travis-job-db364705-789c-4323-8b27-625424a28749:~/build/Ibotta/microservice_scorecard_ge[81/292] time curl -sSfLv https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker- compose-`uname -s`-`uname -m` > docker-compose * Hostname was NOT found in DNS cache * Trying 192.30.253.112... * Connected to github.com (192.30.253.112) port 443 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): } [data not shown] * SSLv3, TLS handshake, Server hello (2): { [data not shown] * SSLv3, TLS handshake, CERT (11): { [data not shown] * SSLv3, TLS handshake, Server key exchange (12): { [data not shown] * SSLv3, TLS handshake, Server finished (14): { [data not shown] * SSLv3, TLS handshake, Client key exchange (16): } [data not shown] * SSLv3, TLS change cipher, Client hello (1): } [data not shown] * SSLv3, TLS handshake, Finished (20): } [data not shown] * SSLv3, TLS change cipher, Client hello (1): { [data not shown] * SSLv3, TLS handshake, Finished (20): { [data not shown] * SSL connection using ECDHE-RSA-AES128-GCM-SHA256 * Server certificate: * subject: businessCategory=Private Organization; 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.6 0.2.1.2=Delaware; serialNumber=5157550; C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github .com * start date: 2018-05-08 00:00:00 GMT * expire date: 2020-06-03 12:00:00 GMT * subjectAltName: github.com matched * issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA * SSL certificate verify ok. > GET /docker/compose/releases/download/1.11.2/docker-compose-Linux-x86_64 HTTP/1.1 > User-Agent: curl/7.35.0 > Host: github.com > Accept: */* > < HTTP/1.1 302 Found * Server GitHub.com is not blacklisted < Server: GitHub.com < Date: Tue, 29 May 2018 16:50:46 GMT < Content-Type: text/html; charset=utf-8 < Transfer-Encoding: chunked < Status: 302 Found < Cache-Control: no-cache < Vary: X-PJAX < Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/15045751/c9e8f01e-f84e-11e6 -8a03-4108467e422d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20180529%2F us-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180529T165046Z&X-Amz-Expires=300&X-Amz-Signature=7610fda91eb a8b32dae968930ad750b1bad32e2d8263049513f5d2221f1e0f15&X-Amz-SignedHeaders=host&actor_id=0&response-cont ent-disposition=attachment%3B%20filename%3Ddocker-compose-Linux-x86_64&response-content-type=applicatio n%2Foctet-stream < Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Sat, 29 May 2038 16:50:46 -0000; secure ; HttpOnly < Set-Cookie: _gh_sess=d1ZTVDBQSVQzK0Z6ZXRLc2dvczdQRjNFb0FUbEpsbFZWZVRpVFQ4elh3TjhwVU92SFlBcGovVjBmV3Zm d0R0THpJc3QxT2VBV3l1K1dlK0xqamxKWUExbE5aZFdBTWt1aUR0R3JUa0FwVXM3M3orUFg5SlV5VFN2YkRtSEdJL1A0QUJIYUFJZUZ FK3JqYWpDYmdyR3ZkMmxmays0TElvaktyYTRZMUdZYzdtQ1IxOWIzbi9RNCtQY1UyZm5rVmN5LS1XM3V1ZDFzSC9MS0JhRlhPODNtZV B3PT0%3D--ea8ec4a80e7cdb272e07980cfd9d171d5de051c4; path=/; secure; HttpOnly < X-Request-Id: 9e875f9b-5019-497d-91ee-a63cd3a40eb3 < X-Runtime: 0.075747 < Strict-Transport-Security: max-age=31536000; includeSubdomains; preload < X-Frame-Options: deny < X-Content-Type-Options: nosniff < X-XSS-Protection: 1; mode=block < Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors" < Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 's elf' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.c om github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-produ ction-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.co com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com ; manifest-src 'self'; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' as sets-cdn.github.com < X-Runtime-rack: 0.086479 < X-GitHub-Request-Id: B9CE:0264:2939309:4C17CCA:5B0D84E6 < * Ignoring the response-body { [data not shown] * Connection #0 to host github.com left intact * Issue another request to this URL: 'https://github-production-release-asset-2e65be.s3.amazonaws.com/1 5045751/c9e8f01e-f84e-11e6-8a03-4108467e422d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJ YAX4CSVEH53A%2F20180529%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180529T165046Z&X-Amz-Expires=300&X -Amz-Signature=7610fda91eba8b32dae968930ad750b1bad32e2d8263049513f5d2221f1e0f15&X-Amz-SignedHeaders=hos t&actor_id=0&response-content-disposition=attachment%3B%20filename%3Ddocker-compose-Linux-x86_64&respon se-content-type=application%2Foctet-stream' * Hostname was NOT found in DNS cache * Trying 52.216.224.152... * Connected to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.224.152) port 443 (#1) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): } [data not shown] * SSLv3, TLS handshake, Server hello (2): { [data not shown] * SSLv3, TLS handshake, CERT (11): { [data not shown] * SSLv3, TLS handshake, Server key exchange (12): { [data not shown] * SSLv3, TLS handshake, Server finished (14): { [data not shown] * SSLv3, TLS handshake, Client key exchange (16): } [data not shown] * SSLv3, TLS change cipher, Client hello (1): } [data not shown] * SSLv3, TLS handshake, Finished (20): } [data not shown] * SSLv3, TLS change cipher, Client hello (1): { [data not shown] * SSLv3, TLS handshake, Finished (20): { [data not shown] * SSL connection using ECDHE-RSA-AES128-GCM-SHA256 * Server certificate: * subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=*.s3.amazonaws.com * start date: 2017-09-22 00:00:00 GMT * expire date: 2019-01-03 12:00:00 GMT * subjectAltName: github-production-release-asset-2e65be.s3.amazonaws.com matched * issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2 * SSL certificate verify ok. > GET /15045751/c9e8f01e-f84e-11e6-8a03-4108467e422d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential= AKIAIWNJYAX4CSVEH53A%2F20180529%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180529T165046Z&X-Amz-Expir es=300&X-Amz-Signature=7610fda91eba8b32dae968930ad750b1bad32e2d8263049513f5d2221f1e0f15&X-Amz-SignedHea ders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Ddocker-compose-Linux-x86_6 4&response-content-type=application%2Foctet-stream HTTP/1.1 > User-Agent: curl/7.35.0 > Host: github-production-release-asset-2e65be.s3.amazonaws.com > Accept: */* > < HTTP/1.1 200 OK < x-amz-id-2: JNNhbc3QPnCrf6yuykZgDyVprXQbKaymFzxITBgUgqXzA9LzI8p4yarZeYzi4KW0l2MXzRLcdig= < x-amz-request-id: 3822E695EA17EE9B < Date: Tue, 29 May 2018 16:50:47 GMT < Last-Modified: Thu, 25 May 2017 10:29:59 GMT < ETag: "585e87810bccd49b441274e377ead7b6" < Content-Disposition: attachment; filename=docker-compose-Linux-x86_64 < Accept-Ranges: bytes < Content-Type: application/octet-stream < Content-Length: 8260272 * Server AmazonS3 is not blacklisted < Server: AmazonS3 < { [data not shown] * Connection #1 to host github-production-release-asset-2e65be.s3.amazonaws.com left intact real 0m1.876s user 0m0.020s sys 0m0.032s

Here's the machine details:

Details

travis_fold:start:worker_info [0K[33;1mWorker information[0m hostname: 4412d8ab-f62e-4610-9cec-a2f6b9ee9ec7@1.production-1-worker-com-b-4-gce version: v3.6.0 https://github.com/travis-ci/worker/tree/170b2a0bb43234479fd1911ba9e4dbcc36dadfad instance: travis-job-db364705-789c-4323-8b27-625424a28749 travis-ci-garnet-trusty-1513287054-2ffda03 (via amqp) startup: 26.379122154s travis_fold:end:worker_info [0Ktravis_fold:start:system_info [0K[33;1mBuild system information[0m Build language: ruby Build group: edge Build dist: trusty Build id: 74754075 Job id: 126467645 Runtime kernel version: 4.4.0-104-generic travis-build version: e12e2fc01 [34m[1mBuild image provisioning date and time[0m Thu Dec 14 21:58:09 UTC 2017 [34m[1mOperating System Details[0m Distributor ID: Ubuntu Description: Ubuntu 14.04.5 LTS Release: 14.04 Codename: trusty [34m[1mCookbooks Version[0m c58cb9c https://github.com/travis-ci/travis-cookbooks/tree/c58cb9c [34m[1mgit version[0m git version 2.15.1 [34m[1mbash version[0m GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu) [34m[1mgcc version[0m gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4 Copyright (C) 2013 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

tsteven4 commented 6 years ago

Docker stalls that lead to no log output in 10m have been plaguing me for the last day or so. Sometimes restarting works, and sometimes it just stalls again. One that stalled: https://travis-ci.org/tsteven4/gpsbabel/jobs/389432206 One that worked: https://travis-ci.org/tsteven4/gpsbabel/jobs/389432207

tsteven4 commented 6 years ago

I don't know if it mattered but I rebuild my docker image to use more but smaller layers. One of the original layers was 1.4GB.

stale[bot] commented 6 years ago

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue in 24 hours. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

Spongman commented 6 years ago

why is this closed? this is still very much an issue.

a search for '10m0s' will show a whole bunch of users having the same issue.

https://github.com/travis-ci/travis-ci/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+10m0s

travis-ci / travis-ci

`docker pull` stalls, times out, and error the job #9127