search

hashicorp / vagrant

Vagrant is a tool for building and distributing development environments.
https://www.vagrantup.com
Other
26.28k stars 4.43k forks source link

Vagrant box downloads extremely slow #5319

Closed spkane closed 6 years ago

spkane commented 9 years ago

When relying on vagrant to download a box I frequently see connection speeds like this:

default: Downloading: http://boxes.example.com/vagrant/boxes/c6/packer_c6_2.5.2_virtualbox.box
default: Progress: 20% (Rate: 179k/s, Estimated time remaining: 0:41:37)

(Rate: 179k/s)

Yet when I use wget to the same URL:

wget http://boxes.example.com/vagrant/boxes/c6/packer_c6_2.5.2_virtualbox.box
--2015-02-10 09:52:12--  http://boxes.example.com/vagrant/boxes/c6/packer_c6_2.5.2_virtualbox.box
Resolving boxes.example.com... 10.1.0.17
Connecting to boxes.example.com|10.1.0.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 830674320 (792M) [text/plain]
Saving to: 'packer_c6_2.5.2_virtualbox.box'

packer_c6_2.5.2_virtualbox.bo   0%[                                                         ]   7.12M   696KB/s   eta 19m 50s

(Rate: 696KB/s) or often higher.

This particular example was pulled when on Wifi and connected to an IPSEC VPN.

DeadlySystem commented 7 years ago

Trying to download ubuntu/xenial64. Download speed maxes out at 150 KB/s on a 1 Gbps symmetrical fiber connection. WTF. Remaining time 1 hour? I could probably download the ISO, read the guide on how to set up my own box, and finish earlier.

EDIT: Interestingly, speed went up by factor 10 when I tried to download the same box in the browser simultaneously.

matti commented 7 years ago

use latest devops tools to speed things up spend days watching max 420k/s download speeds

minhphuc429 commented 7 years ago

Same here, I have a 30Mbps connection

default: Adding box 'ubuntu/trusty64' (v20170313.0.5) for provider: virtualbox default: Downloading: https://atlas.hashicorp.com/ubuntu/boxes/trusty64/versions/20170313.0.5/providers/virtualbox.box default: Box download is resuming from prior download progress default: Progress: 0% (Rate: 80568/s, Estimated time remaining: 1:26:22)

tersmitten commented 7 years ago

@DeadlySystem We have the same experience, when I download the same box (url) using curl from the commandline (during the vagrant up)

dqlopez commented 7 years ago

Any update on this, fetching box from Hashicorp is painfully slow.

screenshot_2017-04-02_21-08-15

faddat commented 7 years ago

Hey, quick thought:

If this uses curl (not libcurl) through some sort of ruby-controlled, bash-mediated process, why not just remove curl for one of:

Both would do the job better than curl.

voycey commented 7 years ago

It honestly looks like they dont give a shit, rules this out as an option for me!

On 3 Apr 2017 8:15 AM, "Jacob Gadikian" notifications@github.com wrote:

Hey, quick thought:

If this uses curl (not libcurl) through some sort of ruby-controlled, bash-mediated process, why not just remove curl for one of:

  • ipfs
  • aria2

Both would do the job better than curl.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mitchellh/vagrant/issues/5319#issuecomment-291019093, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBAihR9ng4t2Jq1XTmAjMyMCnlEFtxRks5rsB4WgaJpZM4Deq5d .

bobflorian commented 7 years ago

I'm on a 150Mbps line.

vagrant up = HOURS vagrant box add = HOURS browser download /wget = HOURS

May not be a vagrant issue per se, BUT IT IS. If your infrastructure can't handle it then your product is broken.

BAD UX

richard-scott commented 7 years ago

I opened a ticket about customizing the download tool, but it got rejected as too complicated to impliment ;-(

That said, my recent download speeds have been ok from the UK for a while. This example downloaded just now:

    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'bento/ubuntu-16.04'
    default: URL: https://atlas.hashicorp.com/bento/ubuntu-16.04
==> default: Adding box 'bento/ubuntu-16.04' (v2.3.4) for provider: virtualbox
    default: Downloading: https://atlas.hashicorp.com/bento/boxes/ubuntu-16.04/versions/2.3.4/providers/virtualbox.box
    default: Progress: 12% (Rate: 8940k/s, Estimated time remaining: 0:01:06)

Maybe you guys are just too far from their AWS instances for a good download speed? All their download servers look to be in New York with no CDN to distribute content.

$ dig atlas.hashicorp.com
; <<>> DiG 9.10.3-P4-Ubuntu <<>> atlas.hashicorp.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44169
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;atlas.hashicorp.com.           IN      A

;; ANSWER SECTION:
atlas.hashicorp.com.    120     IN      CNAME   atlas.hashi.co.
atlas.hashi.co.         60      IN      A       52.206.86.0
atlas.hashi.co.         60      IN      A       52.200.255.5
atlas.hashi.co.         60      IN      A       52.55.203.197

;; Query time: 42 msec
;; SERVER: 10.1.1.14#53(10.1.1.14)
;; WHEN: Mon Apr 24 09:20:24 DST 2017
;; MSG SIZE  rcvd: 124

http://geoiplookup.net/ip/52.55.203.197 http://geoiplookup.net/ip/52.206.86.0 http://geoiplookup.net/ip/52.200.255.5

Maybe its worth the people getting slow downloads detailing what ISP they are using? Maybe you are all on an ISP with a high contention ratio?

chrisroberts commented 7 years ago

https://github.com/mitchellh/vagrant/issues/8434#issuecomment-291979521

bradisbell commented 7 years ago

Maybe its worth the people getting slow downloads detailing what ISP they are using?

Any ISP, in any state I've traveled to in the last couple years.

This problem is squarely on whatever Hashicorp is doing for hosting, and that's where it needs to be fixed. If they're unable or unwilling to fix it, a torrent solution would certainly help without taking their resources aside from development for adding a torrent client to the tool. If all this stuff is on S3 anyway, AWS provides torrent seeding out of the box.

colin-marshall commented 7 years ago

I was getting painfully slow speeds and so I decided to see if upgrading Vagrant would change anything. Before I updated I was getting speeds of around 50-200kbps. After the update I was using the full 70mbps of my connection.

So for those of you that have slow speeds, try updating to v1.9.4 if you aren't already running it.

sXe79 commented 7 years ago

Hi, I'm on the latest Vagrant 1.9.4 and the download speed is so slow that keeps disconnect. No way to download latest laravel homestead 2.1.0 box... When I downloaded the 2.0 with Vagrant 1.9.3 (or previous, can't remember) it was flawless

da-kami commented 7 years ago

same issue here, very slow download rates Rate: 35033/s (keeps jumping between 20000 and 60000...) Estimated time remaining: 9:26:13 :( (on vagrant 1.9.5)

da-kami commented 7 years ago

Now it's fast again... really depends on time of day and luck (as stated above...).

oncet commented 7 years ago

I'm having this same problem when trying to download Bento boxes. I'm using Vagrant 1.9.5 with 5.1.22. I've tried several times, during various days. My normal download speed is 1.7mb~ but when dowloading boxes I can't get pass 40kb.

If I try downloading with wget I get the same low speed.

richard-scott commented 7 years ago

Maybe the move will help:

https://www.vagrantup.com/docs/vagrant-cloud/vagrant-cloud-migration.html

oncet commented 7 years ago

Gonna try again tonight to see if the migration helped. Anyway I noted that with a different ISP the download rates were fine, so I guess it's not a problem with Vagrant itself (like many users already reported).

prabhu89wali commented 7 years ago

Well, nothing has changed, it still downloads at a snail's pace given that i am on a 125 Mbps connection!

acerest commented 7 years ago

Sooooooooooooooo slowly！！！！:(

matti commented 7 years ago

create a new issue, this has been closed for ages

iron-cherep commented 7 years ago

i just downloaded latest scotch-box vagrant box, spent ~1hour for that, and then i found an issue which could be fixed only by downgrading to previous version of box, so i'm here again downloading second box for next hour of my working time. So.. it's good since i got per-time payment, but on the other hand probably i'll have some problems explaining why i spen last day downloading stuff from the internet

otjutt commented 7 years ago

I'm also facing the slow download issue.

prabhu89wali commented 7 years ago

Try downloading from a wired connection, it's exponentially faster!

On 4 September 2017 at 17:02, Omar Tariq notifications@github.com wrote:

I'm also facing the slow download issue.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mitchellh/vagrant/issues/5319#issuecomment-326939886, or mute the thread https://github.com/notifications/unsubscribe-auth/ABmbHxWHtAnCnEw34FB4oXX8lTQrOsNRks5se9-_gaJpZM4Deq5d .

-- Best regards, Shivaprabhu Wali.

CaelanStewart commented 7 years ago

Typical issue, been around for ages, everybody's moaning about it, and nothing is being done.

PascalAOMS commented 7 years ago

Came here via Google and this might help some people:

When I run vagrant init [box] && vagrant up, it loads super slow. But when I run vagrant box add [box], it's faster.

Going from 120 kb/s to >1000 kb/s.

lukewang1024 commented 7 years ago

@PascalAOMS Thanks for the tip. At least it worked for me.

oleg-andreyev commented 7 years ago

Today I've tried aria2 with 16 simultaneous connections to download laravel/homestead managed to get up to 9.6MiB

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
80fa38|ERR |   7.8MiB/s|C:/development/homestead/47dce273-9892-4691-a746-c4f351ae44a5
sandeshchaudhari commented 7 years ago

From last two days i'm trying to download vigrant box but every time i try to download it gives me error and downloading spedd is very slow. I also tried to install it manually but it also didn't worked.For newbie to Laravel it pretty frustrating and i'm still not able to download it.Don't know what should i do now.

stevenrombauts commented 7 years ago

I'm seeing the same problem. Running latest Vagrant 2.0.0. When downloading a box (configured using external URL on Vagrant Cloud, so it's actually redirecting to Rackspace Cloud Files) it's painfully slow.

If I use cURL or Chrome to download the same URL (https://vagrantcloud.com/joomlatools/boxes/box/versions/1.5.0/providers/virtualbox.box) from the same machine it only takes a couple of seconds to complete the download. What could be the issue here?

marcoflorian commented 6 years ago

Using terminal to add a box is extremely slow, more than 1 hour with 100kb/s. If I use a browser to directly download the box it is much faster, just a few minutes with 600kb/s. What's the reason for having such a huge difference? Seems as adding the box through terminal won't use all the available bandwidth.

panozzaj commented 6 years ago

I am trying to download boxes more quickly, and following the advice of some of the above comments, I used aria2 through homebrew to download much faster in parallel. Sample command (url was from original box add attempt from Vagrant):

$ aria2c -x16 https://vagrantcloud.com/ubuntu/boxes/xenial64/versions/20171118.0.0/providers/virtualbox.box

My speed was around 100k/s with Vagrant's download, up to 1 MB/s with aria2.

Then when you finish downloading that, you can add the box using:

$ vagrant box add ubuntu/xenial64 ../xenial64.box

I think you can remove the downloaded box after this since it will be copied to the standard box storage path after adding it. Hope this helps save someone else some time.

madmatt commented 6 years ago

@panozzaj's comment above doesn't quite work if a Vagrantfile expects a specific version of a box (that command will add a box without a version). Instead, you can do the below.

Steps to get a specific box at a specific version without using vagrant to perform the download

I'm using laravel/homestead in this example because it's what I was trying to get.

  1. Download the box using a better download client (e.g. your browser, curl, wget, whatever).

  2. Create a new json file (anywhere), add this to it (note, tweak the name, version and url keys to match the box you want, don't worry about the checksum key yet. For url, use the path to the file you just downloaded, example below):

    {
"name": "laravel/homestead",
"description": "Whatever you want",
"versions": [{
    "version": "4.0.0",
    "providers": [{
            "name": "virtualbox",
            "url": "file://c:/users/madmatt/Downloads/47dce273-9892-4691-a746-c4f351ae44a5",
            "checksum_type": "sha1",
            "checksum": "abc123"
    }]
}]
}

  3. In your Vagrantfile, add the following lines vm.box and vm.box_url keys):

    Vagrant.configure("2") do |config|
config.vm.box = "laravel/homestead"
config.vm.box_url = "file://c:/users/madmatt/path/to/the/json-file-you-created-in-step-2.json"
end

  4. Run vagrant up as normal.

  5. vagrant will complain that the sha hash doesn't match (The checksum of the downloaded box did not match...). Take the string that appears next to 'Expected', copy and paste that into the checksum key of the json file you created in step 2.

  6. Run vagrant up again, this time it should load from the local file, store it as the correct name and version, and successfully run it.

Hopefully someone finds these steps useful... funnily enough I have done all this research (having never used vagrant before), have tested it a bunch of times, and the original vagrant box add laravel/homestead command that I started running 3 hours ago is still only 8% complete, even though in that time I've downloaded the box file 8 times outside of vagrant.

The rest of this is just my experience, no need to read further ;)

My experience (aka. why is vagrant so slow to download boxes)

Was getting 420b/sec (yes, that's bytes per second) on a gigabit connection downloading https://vagrantcloud.com/laravel/boxes/homestead/versions/4.0.0/providers/virtualbox.box.

I downloaded the same file via browser, curl and wget with speeds varying between 12 and 27MB/sec. I then tried doing both at the same time - I was able to download via Chrome, Firefox, curl and wget before vagrant box add laravel/homestead had downloaded 1%.

The URL I got in the browser was https://vagrantcloud-files-production.s3.amazonaws.com/archivist/boxes/47dce273-9892-4691-a746-c4f351ae44a5?X-Amz-Algorithm=<snip>&X-Amz-Credential=<snip>&X-Amz-Date=<snip>&X-Amz-Expires=<snip>&X-Amz-SignedHeaders=<snip>&X-Amz-Signature=<snip>

I don't know what the problem here is, but I can think of a couple:

TimOgilvy commented 6 years ago

how is this closed? still a major issue

bradisbell commented 6 years ago

Perhaps it's time to fork. This project is core to a lot of development environments, leaving us all subject to the whims of HashiCorp... and on this issue we seemingly cannot even get a reasonable official response.

Vagrant is MIT-licensed. Boxes could easily be distributed via torrent, or we can even just specify URLs in our Vagrantfile. We don't need the Vagrant Cloud dependency if some basic enhancements around box handling are made. Are there any maintained forks already in existence?

Any thoughts from others?

fredngo commented 6 years ago

Definitely still an issue!

atb00ker commented 6 years ago

Are there any trusted torrent links available? I am trying to download bentoo/ubuntu16.04 and i am getting 12 kbps and upto 15kbps when i try to do wget, i am sure that my connection is fine!

DenzoNL commented 6 years ago

Still an issue here, have to update my box at ~150k/s, not able to do a lot of useful work in the meanwhile..

dogweather commented 6 years ago

Still an issue.

dogweather commented 6 years ago

@bradisbell The real solution is probably a docker-based development environment. Just waiting for that ecosystem to settle down .

TimOgilvy commented 6 years ago

🍺 🍺 🍺 🍺 🍺 🍺

I'm just going to leave these here for any poor sod waiting for this download.

dogweather commented 6 years ago

I found a root cause and a solution for my situation. I increased my download speed 100x:

Root cause: Slow speed of ubuntu.com cloud box servers Solution: Switch to another trusted source for Ubuntu boxes, Puppet Labs.

I've been using ubuntu/xenial64 for developing apps that must deploy to Ubuntu 16. (Heroku) And download speeds have become ridiculously slow, or failed entirely. I used wget and watched it follow the redirects and saw that it's actually hitting ubuntu.com. I tried several ways to download from Ubuntu, but they all resulted in speeds around 50 KB/s.

So I went back to the Vagrant box listing to see who else might have a Xenial 64 box, and if I'd trust them. Turns out Puppet does, and I trust them as much as (or even more than) Ubuntu. My Vagrantfile now has the config line:

config.vm.box = "puppetlabs/ubuntu-16.04-64-nocm"

And it downloaded in just a few seconds, as opposed to possibly an hour or never.

tristanmason commented 6 years ago

@dogweather Thanks for this! My download time estimate went from 3 hours to 3 minutes when I switched to a box from the list at https://app.vagrantup.com/puppetlabs/

dogweather commented 6 years ago

@tristanmason You're welcome!

Ironically: In the past week I switched to Docker Compose, and it's AMAZING. I can't believe I've waited this long to use it. I used this Quickstart for Rails doc.

I was able to setup a Docker-based dev environment on four computers in a fraction of the time it takes me to create a single Vagrant setup from scratch.

I'm going to write up a blog with detailed metrics - I'll post it here.

oncet commented 6 years ago

@dogweather Vagrant and Docker seems similar but they are different things. Also, you can use a Docker provider within Vagrant.

dogweather commented 6 years ago

@oncet Yep, very different! But for my use case, they're equivalent and commodified: Ways I can create and launch an isolated dev environment with just one or two commands.

Docker Compose works great and is far simpler than Vagrant (note, this not just "Docker" per se, which only launches containers one by one) and I'd need to hear about a compelling reason to try a Vagrant+Docker solution, which sounds pretty damn complex. ;-)

sXe79 commented 6 years ago

Pain. This is a pain. I'll never be able to download the 5.0.1 I guess

jdunk commented 6 years ago

On a 100mbps connection in Manila, with Vagrant 2.0.1, I was trying to download:

https://vagrantcloud.com/ubuntu/boxes/trusty64/versions/20180110.0.0/providers/virtualbox.box

...and I was getting speeds of anywhere from 1k-150k via vagrant up or even wget. Total download time: 12 hours!

Then I did one thing: I VPN'd to California.

Suddenly, my download took only 3 mins. So that's something worth trying possibly.

racterub commented 6 years ago

Same here, I thought this is a personal problem lol. Even update box still get really poor download speed, hashicorp please fix :(

==> default: Updating 'ubuntu/xenial64' with provider 'virtualbox' from version
==> default: '20171221.0.0' to '20180122.0.0'...
==> default: Loading metadata for box 'https://vagrantcloud.com/ubuntu/xenial64'
==> default: Adding box 'ubuntu/xenial64' (v20180122.0.0) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/ubuntu/boxes/xenial64/versions/20180122.0.0/providers/virtualbox.box
==> default: Box download is resuming from prior download progress
    default: Progress: 13% (Rate: 52999/s, Estimated time remaining: 0:31:46)
dqlopez commented 6 years ago

Ignoring such persistent issue means something on Hashicorp's end, I believe.