when first project creation is unacceptably slow due to s3 download speed - Githubissues

tilemill-project / tilemill

TileMill is a modern map design studio

https://tilemill-project.github.io/tilemill/

BSD 3-Clause "New" or "Revised" License

3.11k stars 527 forks source link

when first project creation is unacceptably slow due to s3 download speed #660

Closed springmeyer closed 13 years ago

springmeyer commented 13 years ago

Discussed with @yahn last week about bundling the default project shapefile. At the time I was on a rural connection. now I'm back in the city of Seattle and have basically the same download speeds: pathetic speeds around 30-40 KB/sec.

So, I'm back proposing that we bundle this 9 MB shapefile because on a connection like this creating a new project (if the shapefile is not cached) takes > 2 minutes.

For new users that download the tilemill.app and hit this its going to appear that tilemill is broken out of the box because they'll have no idea tilemill is even trying to download anything when they create a project for the first time.

Here is how bad things are:

time wget http://mapbox-geodata.s3.amazonaws.com/natural-earth-1.3.0/cultural/10m-admin-0-countries.zip
--2011-08-23 19:55:53--  http://mapbox-geodata.s3.amazonaws.com/natural-earth-1.3.0/cultural/10m-admin-0-countries.zip
Resolving mapbox-geodata.s3.amazonaws.com... 72.21.214.42
Connecting to mapbox-geodata.s3.amazonaws.com|72.21.214.42|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7232484 (6.9M) [binary/octet-stream]
Saving to: `10m-admin-0-countries.zip.2'

100%[=====================================================================================>] 7,232,484   46.7K/s   in 2m 26s  

2011-08-23 19:58:20 (48.2 KB/s) - `10m-admin-0-countries.zip.2' saved [7232484/7232484]

real    2m26.800s
user    0m0.053s
sys 0m0.359s

This is despite speakeasy reporting my download speed as:

chrisforbes commented 13 years ago

It would be really nice to have more feedback from tilemill about exactly what it's doing, and how much more work there is to do. Downloading the initial shapefile isn't the only slow thing.

willwhite commented 13 years ago

Progress indicator is similar to #427. I think bundling the basic shapefile makes sense. What are the downsides?

willwhite commented 13 years ago

So the main concern I guess is where to draw the line once we start storing data in this repo. It would be better to keep all data centrally as resources like we have done now.

We also haven't hear of many people running into problems with this.

We have tried to address this in the past using other approaches. What other options do we have? Should we just work on a progress indicator system?

springmeyer commented 13 years ago

I agree we need to draw the line - but the default shapefile used for a new project is well within that line in my opinion. #639 was a great fix for the experienced user. This ticket is about the first run experience and it being potentially awful for those with slow or spotty connections. I think the only way to solve this is to bundle. For people with a good connection first project creation does not need a progress (or will not benefit from a) bar since it is likely be so fast, though a progress bar would be an awesome thing in other parts of the app.

willwhite commented 13 years ago

Fixed with https://github.com/mapbox/tilemill/commit/0e6cdb3066d9ab0f2e970dd34761a57e6d4a2a62

springmeyer commented 13 years ago

This appears not to be working in the 0.5.1 ubuntu package. Here at foss4g testing trying to upgrade the VMs. Net is horrid. Creating a new project took about 5 minutes of wait on the spinner.

springmeyer commented 13 years ago

/usr/share/mapbox/data/ is empty.

willwhite commented 13 years ago

@springmeyer, is this on a fresh install? Can you check your cache directory? TileMill should create a directory called 1aa2d236-10m-admin-0-countries on first run, but only if the cache directory doesn't exist already.

willwhite commented 13 years ago

It should be in /usr/share/mapbox/cache.

springmeyer commented 13 years ago

ya, it is there now, after the initial wait. Maybe I am misunderstanding? Should there still be the first run download wait? I am on my first machine of 25, so I'll have a chance to check the cache directory on the next machine(s).

willwhite commented 13 years ago

After installing, TileMill will start and the file will be copied over to the cache directory on this first run. This is what happens on my system. Right after installing and before opening TileMill in the browser, look in the cache directory for the shapefile.

If it's there, it should get used. Could any slowness be related to downloading files for the sample projects? Also, maybe there is a problem with this on natty.

willwhite commented 13 years ago

An easy way to rule out the sample project thing would be to delete them before opening TileMill in your browser.

willwhite commented 13 years ago

Huh something else is going on. It looks like tilemill isn't satisfied with just the shapefiles and it gets upset that there isn't an actual zipfile. In hindsight this makes sense because it's the zipfile that is referenced by the project. I'll work on a fix.

springmeyer commented 13 years ago

ya, so the cache is there upon installation. the zipfile thing sounds likely. we should also not rule out that the downloading of the non-cached projects is blocking something around the project creation. It it looking like it will take these machines more than 30 minutes to get the projects downloaded.

willwhite commented 13 years ago

Pushing a fix for the zip thing now. Not sure what to do about the sample projects for the immediate besides just deleting them manually after installation.

willwhite commented 13 years ago

This should be fixed in the lastest ubuntu build.