search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

ftp.epa.gov #124

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

Including directory listing of the FTP server - a checked box means that item is mirrored by someone. These can be split out into separate issues, but I didn't have time right now.

ghost commented 7 years ago

Pulling via Internet2, The Azimuth Backup Project. All of it.

nickrsan commented 7 years ago

I'm pulling coal-combustion-residues, Emisinventory, and nceaftp1 right now

lrehmann commented 7 years ago

Also pulling a copy of both ftp.epa.gov and newftp.epa.gov

nbyloff commented 7 years ago

Pulling the amd folder right now. Some others in this list are empty though.

ghost commented 7 years ago

Progress from The Azimuth Backup Project is that as of this morning, have 2/3 of the 391G this FTP folder contains.

gabefair commented 7 years ago

Any update @empirical-bayesian ?

ghost commented 7 years ago

Actually, yes, it's finished. I'll post final details in morning. It's on a public mirror. Off to a climate resiliency tech forum tonight.

On Wed, Jan 25, 2017, at 16:45, Gabriel Fair wrote:

Any update @empirical-bayesian[1] ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub[2], or mute the thread[3].

Links:

  1. https://github.com/empirical-bayesian
  2. https://github.com/climate-mirror/datasets/issues/124#issuecomment-275242565
  3. https://github.com/notifications/unsubscribe-auth/AD3HB6uWmon_1cdn1CGNZbBh27Afpcb3ks5rV8KHgaJpZM4LtIEL
TwistTheNeil commented 7 years ago

I'm also pulling down ftp.epa.gov. Since @empirical-bayesian has a public mirror, i'll keep mine offline for later should it be needed.

ghost commented 7 years ago

The Azimuth Backup Project respectfully submits the FTP site of the EPA, mirrored at:

https://mirror.math.princeton.edu/pub/climatemirror/ftp.epa.gov/

for your copy if you make one, consider making life easy and using rsync to the same path:

rsync -Hhartv --progress rsync://mirror.math.princeton.edu/pub/climatemirror/ftp.epa.gov/ /var/local/whereever

ghost commented 7 years ago

And if someone could make a copy, we'd appreciate it!

blitzdesigner commented 7 years ago

I started to do a copy...

blitzdesigner commented 7 years ago

copy done, received 418,68GB. I will make it available soon at ftp://ftp.blitzdesigner.info/pub/climatemirror/ftp.epa.gov

StephWo commented 7 years ago

@blitzdesigner How did you do the download? Most federal servers don't let me pull that much in only 12 hours.

blitzdesigner commented 7 years ago

@BauerPiepenbrink I did an rsync from the Azimuth Project as described by @empirical-bayesian , so i didn't pulled it directly from epa.gov

JeremiahCurtis commented 7 years ago

was there any data in ftp://ftp.epa.gov/pub/r8/ ? (not mirrored above)