search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

Dataset at ftp:/podaac.jpl.nasa.gov/OceanTemperature/ghrsst #210

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

ftp:/podaac.jpl.nasa.gov/OceanTemperature/ghrsst.

Suggested in a large email containing many urls

TheWug commented 7 years ago

I'm pulling this one. I have no idea what I'm doing, this is the first I've contributed to something like this, will attempt to get up to speed as it downloads. Have ~20TB of free space total

donbright commented 7 years ago

if .inventory/index_GDS2.txt is a list of files here, then there are about 3 million files.
alot of them are about 5 megabytes. i gave up after an hour, i dont have the space.

TheWug commented 7 years ago

Space I can do, I'm not sure I have the time, the box seems pretty loaded and I'm only getting about 10mbits average transfer, it could take days to download it all...

ghost commented 7 years ago

Don't want to intrude, but some dataset transfers the Azimuth Backup Project did, at backbone speeds, took weeks.

On Wed, Jan 25, 2017, at 22:55, TheWug wrote:

Space I can do, I'm not sure I have the time, the box seems pretty loaded and I'm only getting about 10mbits average transfer, it could take days to download it all... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub[1], or mute the thread[2].

Links:

  1. https://github.com/climate-mirror/datasets/issues/210#issuecomment-275302493
  2. https://github.com/notifications/unsubscribe-auth/AD3HB9c3JqPDZt7aDWHAKk6ZmVUEQWT9ks5rWBkpgaJpZM4LtK7u
mskallisti commented 7 years ago

Pulling OceanTemperature/*

TheWug commented 7 years ago

Yeah, but I'm only seeing between 5-10% bandwidth utilization, I suspect the backend is heavily IO loaded. I would expect the whole dataset, using donbright's estimation of its volume, to take about as week to download, but it's obviously going to be longer given the poor utilization.

Does nasa really pay for infinite bandwidth on this thing, or are we all fighting over a single 100mbit pipe?

bkirkbri commented 7 years ago

@mskallisti @TheWug Were you able to get this one? If so, please update #241 since this is a duplicate.

Closing as a duplicate of #241.