search

travis-ci / dpl

Dpl (dee-pee-ell) is a deploy tool made for continuous deployment.
MIT License
1.29k stars 367 forks source link

Timeout while uploading to GitHub Releases: Net::ReadTimeout (Faraday::TimeoutError) #796

Closed cxbrooks closed 6 years ago

cxbrooks commented 6 years ago

I'm uploading fairly large files to GitHub Releases using deploy and I'm seeing 

/home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/protocol.rb:158:in `rescue in rbuf_fill': Net::ReadTimeout (Faraday::TimeoutError)

Any ideas?

Details below.

https://travis-ci.org/icyphy/ptII/jobs/359403072 failed during deploy.

The deploy part of my .travis.yml looks like

env:
   # [Elided]
  global:
     # [Elided]
    - TRAVIS_TAG=nightly

#...

deploy:
  # [Elided]
  - provider: releases
    api_key: $GITHUB_TOKEN
    file_glob: true
    file:
      - $PTII/doc/codeDoc*.jar
      - $PTII/adm/gen-11.0/capecode0.2.devel.setup.linux.tar.gz
      - $PTII/adm/gen-11.0/capecode0.2.devel.setup.mac.app.tar.gz
      - $PTII/adm/gen-11.0/capecode0_2_devel_setup_windows_64.exe
      - $PTII/adm/gen-11.0/ptII11.0.devel.setup.linux.tar.gz
      - $PTII/adm/gen-11.0/ptII11.0.devel.setup.mac.app.tar.gz
      - $PTII/adm/gen-11.0/ptII11_0_devel_setup_windows_64.exe
      - $PTII/adm/gen-11.0/ptII11_0_devel_setup_windows.exe
      - $PTII/adm/gen-11.0/ptII11.0.devel.src.tar
      - $PTII/adm/gen-11.0/ptII11.0.devel.src.zip
    on:
      tags: true
      condition: $PT_TRAVIS_INSTALLERS = true
    overwrite: true
    skip_cleanup: true
    verbose: true

The files that are to be uploaded are large (between 261Mb and 613Mb)

-rwxrwxr-x 1 travis travis  263470977 Mar 28 15:22 capecode0.2.devel.setup.linux.jar
-rw-rw-r-- 1 travis travis  452221788 Mar 28 15:21 capecode0.2.devel.setup.mac.app.tar.gz
-rwxrwxr-x 1 travis travis  391279040 Mar 28 15:16 capecode0_2_devel_setup_windows_64.exe
-rw-rw-r-- 1 travis travis  463312188 Mar 28 15:14 ptII11.0.devel.setup.linux.tar.gz
-rw-rw-r-- 1 travis travis  613702279 Mar 28 15:18 ptII11.0.devel.setup.mac.app.tar.gz
-rwxrwxr-x 1 travis travis  551971002 Mar 28 15:12 ptII11_0_devel_setup_windows_64.exe
-rwxrwxr-x 1 travis travis  540590804 Mar 28 15:10 ptII11_0_devel_setup_windows.exe
-rw-rw-r-- 1 travis travis  549898240 Mar 28 15:06 ptII11.0.devel.src.tar
-rw-rw-r-- 1 travis travis  261495314 Mar 28 15:07 ptII11.0.devel.src.zip

The failure from https://travis-ci.org/icyphy/ptII/jobs/359403072 is below:

Logged in as Christopher Brooks
Deploying to repo: icyphy/ptII
Current tag is: nightly
codeDocCapeCode.jar already exists, overwriting.
codeDocViptos.jar already exists, overwriting.
codeDocHyVisual.jar already exists, overwriting.
codeDocVisualSense.jar already exists, overwriting.
codeDocBcvtb.jar already exists, overwriting.
codeDoc.jar already exists, overwriting.
capecode0.2.devel.setup.linux.tar.gz already exists, overwriting.
capecode0.2.devel.setup.mac.app.tar.gz already exists, overwriting.
/home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/protocol.rb:158:in `rescue in rbuf_fill': Net::ReadTimeout (Faraday::TimeoutError)
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/protocol.rb:152:in `rbuf_fill'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/protocol.rb:134:in `readuntil'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/protocol.rb:144:in `readline'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/http/response.rb:39:in `read_status_line'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/http/response.rb:28:in `read_new'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/http.rb:1414:in `block in transport_request'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/http.rb:1411:in `catch'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/http.rb:1411:in `transport_request'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/http.rb:1384:in `request'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/http.rb:1377:in `block in request'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/http.rb:853:in `start'
     from /home/travis/.rvm/rubies/ruby-2.2.7/lib/ruby/2.2.0/net/http.rb:1375:in `request'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.14.0/lib/faraday/adapter/net_http.rb:80:in `perform_request'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.14.0/lib/faraday/adapter/net_http.rb:38:in `block in call'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.14.0/lib/faraday/adapter/net_http.rb:85:in `with_net_http_connection'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.14.0/lib/faraday/adapter/net_http.rb:33:in `call'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.14.0/lib/faraday/response.rb:8:in `call'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.14.0/lib/faraday/response.rb:8:in `call'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/octokit-4.6.2/lib/octokit/middleware/follow_redirects.rb:73:in `perform_with_redirection'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/octokit-4.6.2/lib/octokit/middleware/follow_redirects.rb:61:in `call'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.14.0/lib/faraday/rack_builder.rb:143:in `build_response'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.14.0/lib/faraday/connection.rb:387:in `run_request'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/faraday-0.14.0/lib/faraday/connection.rb:175:in `post'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/sawyer-0.8.1/lib/sawyer/agent.rb:94:in `call'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/octokit-4.6.2/lib/octokit/connection.rb:154:in `request'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/octokit-4.6.2/lib/octokit/client/releases.rb:94:in `upload_asset'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/octokit-4.6.2/lib/octokit/client/releases.rb:94:in `upload_asset'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-releases-1.9.5/lib/dpl/provider/releases.rb:138:in `upload_file'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-releases-1.9.5/lib/dpl/provider/releases.rb:123:in `block in push_app'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-releases-1.9.5/lib/dpl/provider/releases.rb:109:in `each'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-releases-1.9.5/lib/dpl/provider/releases.rb:109:in `push_app'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.5/lib/dpl/provider.rb:194:in `block in deploy'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.5/lib/dpl/cli.rb:41:in `fold'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.5/lib/dpl/provider.rb:194:in `deploy'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.5/lib/dpl/cli.rb:32:in `run'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.5/lib/dpl/cli.rb:7:in `run'
     from /home/travis/.rvm/gems/ruby-2.2.7/gems/dpl-1.9.5/bin/dpl:5:in `<top (required)>'
     from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in `load'
     from /home/travis/.rvm/gems/ruby-2.2.7/bin/dpl:23:in `<main>'
dpl.2
Preparing deploy
dpl.3
Deploying application
failed to deploy

I saw a similar failure with another job kepler-build #11

I'm going to attempt to use dpl from the command line to reproduce this, but I was wondering if anyone had any ideas?

cxbrooks commented 6 years ago

Under Ubuntu 17.10, I was able to replicate a slightly different error by trying to upload a 607Mb file to GitHub Releases

1) I created an test repo: dpl-test

2) I downloaded dpl-1.9.5 and installed it

3) I had previously set up an api-key

4) In a local copy of my dpl-test repo, I downloaded a large test file wget https://chess.eecs.berkeley.edu/ptexternal/nightly/builds/builds/ptII11_0_devel_setup_windows-2018-03-10.exe

5) I ran the command:

cxh@wessel:~/src/dpl-test$ dpl --provider=releases --file=ptII11_0_devel_setup_windows_64-2018-03-10.exe --api-key=Elided --repo=cxbrooks/dpl-test --skip-cleanup=true
Installing deploy dependencies
Preparing deploy
Logged in as Christopher Brooks
Deploying to repo: cxbrooks/dpl-test
Current tag is:
Deploying application
 (Octokit::ServerError)
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
        from /var/lib/gems/2.3.0/gems/faraday-0.14.0/lib/faraday/response.rb:9:in `block in call'
        from /var/lib/gems/2.3.0/gems/faraday-0.14.0/lib/faraday/response.rb:61:in `on_complete'
        from /var/lib/gems/2.3.0/gems/faraday-0.14.0/lib/faraday/response.rb:8:in `call'
        from /var/lib/gems/2.3.0/gems/octokit-4.6.2/lib/octokit/middleware/follow_redirects.rb:73:in `perform_with_redirection'
        from /var/lib/gems/2.3.0/gems/octokit-4.6.2/lib/octokit/middleware/follow_redirects.rb:61:in `call'
        from /var/lib/gems/2.3.0/gems/faraday-0.14.0/lib/faraday/rack_builder.rb:143:in `build_response'
        from /var/lib/gems/2.3.0/gems/faraday-0.14.0/lib/faraday/connection.rb:387:in `run_request'
        from /var/lib/gems/2.3.0/gems/faraday-0.14.0/lib/faraday/connection.rb:175:in `post'
        from /var/lib/gems/2.3.0/gems/sawyer-0.8.1/lib/sawyer/agent.rb:94:in `call'
        from /var/lib/gems/2.3.0/gems/octokit-4.6.2/lib/octokit/connection.rb:154:in `request'
        from /var/lib/gems/2.3.0/gems/octokit-4.6.2/lib/octokit/client/releases.rb:94:in `upload_asset'
        from /var/lib/gems/2.3.0/gems/dpl-releases-1.9.5/lib/dpl/provider/releases.rb:138:in `upload_file'
        from /var/lib/gems/2.3.0/gems/dpl-releases-1.9.5/lib/dpl/provider/releases.rb:119:in `block in push_app'
        from /var/lib/gems/2.3.0/gems/dpl-releases-1.9.5/lib/dpl/provider/releases.rb:109:in `each'
        from /var/lib/gems/2.3.0/gems/dpl-releases-1.9.5/lib/dpl/provider/releases.rb:109:in `push_app'
        from /var/lib/gems/2.3.0/gems/dpl-1.9.5/lib/dpl/provider.rb:194:in `block in deploy'
        from /var/lib/gems/2.3.0/gems/dpl-1.9.5/lib/dpl/cli.rb:41:in `fold'
        from /var/lib/gems/2.3.0/gems/dpl-1.9.5/lib/dpl/provider.rb:194:in `deploy'
        from /var/lib/gems/2.3.0/gems/dpl-1.9.5/lib/dpl/cli.rb:32:in `run'
        from /var/lib/gems/2.3.0/gems/dpl-1.9.5/lib/dpl/cli.rb:7:in `run'
        from /var/lib/gems/2.3.0/gems/dpl-1.9.5/bin/dpl:5:in `<top (required)>'
        from /usr/local/bin/dpl:22:in `load'
        from /usr/local/bin/dpl:22:in `<main>'
cxh@wessel:~/src/dpl-test$

One thing of note is that this errored after "Deploying application", whereas the Travis-ci build failure errored before that.

benanhalt commented 6 years ago

I've been seeing the same issue. https://travis-ci.org/specify/specify6/builds/362769264

cxbrooks commented 6 years ago

@benanhalt How large are the files that you are deploying? Mine are fairly large, the test case above failed for a 579Mb file.

Interestingly, the deploy seems to work at night, see https://travis-ci.org/icyphy/ptII/jobs/361903257

See that build for the sizes of files that worked.

https://travis-ci.org/icyphy/ptII/jobs/362769335 failed at or near this file:

 470155196 Apr  5 18:52 ptII11.0.devel.setup.linux.tar.gz
benanhalt commented 6 years ago

There are ten pretty large files, about 140MB each.

antonym commented 6 years ago

I'm seeing the same thing periodically, uploading three files with the larger one being around 600mb. If I retry the builds several times it'll eventually go through.

https://travis-ci.org/rcbops/rpc-deploy-utility-image/builds/370152780

iriberri commented 6 years ago

I've been taking a look at this and I found this "related" issue in Octokit: https://github.com/octokit/octokit.rb/issues/769, which doesn't bring much extra information, though.

Also, according to Ruby's Net::HTTP:

read_timeout[R]
Number of seconds to wait for one block to be read (via one read(2) call).
Any number may be used, including Floats for fractional seconds.
If the HTTP object cannot read data in this many seconds, it raises a
Net::ReadTimeout exception. **The default value is 60 seconds.**

(http://ruby-doc.org/stdlib-2.1.2/libdoc/net/http/rdoc/Net/HTTP.html)

Not sure if this timeout is being overwritten through any of the middle layers, or if all the cases here are indeed stumbling upon the 60 second limit 🤔. It's a bit difficult to say, since the timing for these requests is not being logged AFAIK.

@BanzaiMan what do you think about adding some debugging messages on time around the lines that use Octokit's upload_asset, and/or maybe try playing with the connection options through Octokit (see: https://github.com/octokit/octokit.rb/pull/235)?

0reo27 commented 6 years ago

Getting the same issue. The binaries trying to be uploaded in my case are over 1GB.

Deilan commented 6 years ago

Having the same issue for almost a month. Sadly, going to switch from Travis to something else because of tons of issues with testing and deployment.

cxbrooks commented 6 years ago

@Deilan Please feel free to submit issues for things that are interfering with your usage. Also, what sorts of alternatives are you going to try? Travis is missing some of the features of Jenkins, which I regret, but Travis-ci.org is free, and I need to move off some non-free servers, hence my choice of Travis. I've had reasonable success getting my multihour build working, my notes are at https://wiki.eecs.berkeley.edu/ptexternal/Main/Travis

About this bug, it seems like the bug is intermittent and somewhat based on time of day. Travis cron jobs seem to work more reliably, see my cron jobs at https://travis-ci.org/icyphy/ptII/builds

0reo27 commented 6 years ago

For those having issues with releases, you may find this useful:

https://gist.github.com/stefanbuck/ce788fee19ab6eb0b4447a85fc99f447

IanSavchenko commented 6 years ago

We are also hit by this issue having two not so big files uploaded (63 MB and 274 MB). It seems like files are actually uploaded but a release stays in draft.

tesch1 commented 6 years ago

Same here, fails every time, large release file (319 M) : https://travis-ci.org/tesch1/OpenVnmrJ/jobs/382637038

Deilan commented 6 years ago

@cxbrooks

Please feel free to submit issues for things that are interfering with your usage.

I do all the time. The vast majority of them became stale and were closed without getting fixed.

Also, what sorts of alternatives are you going to try?

Some solutions available on-premise (like Jenkins or TeamCity) or other SaaS with GitHub integration (like CircleCI). We have private repositories hosted on GitHub.

cxbrooks commented 6 years ago

This seems to be fixed, or at least it has not bitten me for 17 days since https://travis-ci.org/icyphy/ptII/jobs/382777902

I'm not sure what changed, but I'll close this and reopen it if it reappears.

If anyone else is still being bitten by this, then by all means please reopen it.

cxbrooks commented 6 years ago

The error occurred again, see https://travis-ci.org/icyphy/ptII/jobs/392745520

cxbrooks commented 6 years ago

And again, see https://travis-ci.org/icyphy/ptII/jobs/394816449

antonym commented 6 years ago

Still an issue for me.

joepvd commented 6 years ago

A potential fix has been released. I am closing this issue for now, let us know if the problems would persist.