search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

Dataset at ftp:/airbornescience.nsstc.nasa.gov/grip/HIWRAP #134

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

ftp:/airbornescience.nsstc.nasa.gov/grip/HIWRAP.

Suggested in a large email containing many urls

stevelord commented 7 years ago

There's no grip folder on this FTP server. An ls of the / folder reveals:

drwxr-xr-x 22 ftp ftp 23 Jan 24 18:55 . drwxr-xr-x 6 ftp ftp 7 Jun 24 2016 .. drwxr-xr-x 6 ftp ftp 6 Jun 24 2014 amsre drwxr-xr-x 5 ftp ftp 6 Jun 29 2015 amsu-a drwxr-xr-x 4 ftp ftp 4 Mar 12 2013 amsutemps drwxr-xr-x 8 ftp ftp 8 Jun 29 2015 browse drwxr-xr-x 6 ftp ftp 6 Dec 7 21:58 browse_sample drwxr-xr-x 3 ftp ftp 3 May 25 2016 case_studies drwxr-xr-x 11 ftp ftp 11 Mar 1 2016 data drwxr-xr-x 36 ftp ftp 36 Oct 25 16:31 doc drwxr-xr-x 11 ftp ftp 11 Nov 2 21:46 fieldCampaigns drwxr-xr-x 3 ftp ftp 4 Aug 7 2013 globalir drwxr-xr-x 4 ftp ftp 4 Aug 28 2014 lis drwxr-xr-x 6 ftp ftp 6 Feb 2 2015 lma drwxr-xr-x 6 ftp ftp 6 Jan 25 2016 msu drwxr-xr-x 5 ftp ftp 5 Dec 4 2014 rasi lrw-r--r-- 1 ftp ftp 10 Nov 14 2011 sandbox -> ../sandbox drwxr-xr-x 8 ftp ftp 8 Jul 26 2012 ssmi drwxr-xr-x 4 ftp ftp 4 Sep 21 2012 ssmis drwxr-xr-x 4 ftp ftp 4 Jun 7 2013 storms drwxr-xr-x 3 ftp ftp 3 Dec 10 2015 tcpf drwxr-xr-x 4 ftp ftp 4 Apr 26 2016 tpw drwxr-xr-x 4 ftp ftp 4 Apr 26 2016 wind_climatology

donbright commented 7 years ago

As with issue #133 this seems to be behind a username / password wall from Earthdata (free but it complicates fetching)

I attempted mirror from https://fcportal.nsstc.nasa.gov/pub/grip/HIWRAP but am not 100% positive due to the complicated fetching required over http:

attempted mirror: offline, contact me if data is needed

hashdeep: offline, contact me if data is needed

Size: ~ 12G.

I attempted to mirror it using the same method as issue #133 but with slightly different curl/wget

curl  -k -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://fcportal.nsstc.nasa.gov/pub/grip/HIWRAP/data

wget -m -np --user=xxxxx --password=xxxxxx  --no-check-certificate --load-cookies ~/.urs_cookies  https://fcportal.nsstc.nasa.gov:/pub/grip/HIWRAP/

update i had previously used wget -r ,, updated this to wget -m per @wantonwonton suggestion on #133

redownloaded 2017-2-19 due to hard drive failure