climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

NASA PODAAC Ocean Temperature Data #268

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

IGNORE, THIS IS A SUBSET OF ANOTHER ISSUE

ftp:/podaac.jpl.nasa.gov/OceanTemperature/amsre.

Suggested in a large email containing many urls

gabefair commented 7 years ago

Downloading now

DiabolicalTMcD commented 7 years ago

Pulling

bkirkbri commented 7 years ago

This is a subset of #282.

@gabefair @DiabolicalTMcD Any luck?

DiabolicalTMcD commented 7 years ago

My download is still running. Going slowly. I'm at about 60% completion

bkirkbri commented 7 years ago

@DiabolicalTMcD Did your download complete?

DiabolicalTMcD commented 7 years ago

Yes. 531GB. Working on making it available somehow.

bkirkbri commented 7 years ago

@DiabolicalTMcD Great! If you would, please post output of hashdeep -rl podaac.jpl.nasa.gov/OceanTemperature/amsre

DiabolicalTMcD commented 7 years ago

That particular directory isn't very large - it's only around 40MB. What's really big is the allData/topex directory. That's the 531GB chunk. podaac.jpl.nasa.gov/OceanTemperature/amsre/L3/sst_ldeg_lmo is a sym link back to ../../../allData/amsre/L3/sst_ldeg_lmo. The hashdeep output for that path is:

hashdeep -rl podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo 
%%%% HASHDEEP-1.0
%%%% size,md5,sha256,filename
## Invoked from: /localdata/jpl
## # hashdeep -rl podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo
## 
218,37ee8ff3292bfa3eeece20f63f58a32e,5c5985a4f1ce1dcb573fd16374af57021440060e5921c3ad47ad8c70b6172f98,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/docs/.listing
70,c915f290c6a566a8fee608c661a7a25f,85ce5e415e3c4e8a15d76f0cc2be09d82c68d7efe08446da5e858e7e81cc96f2,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/tosNobs_AMSRE_L3_v7_200206-201012.nc.md5
788,45eb2ea0c4644b69c4d9f04590649b0a,3bdaddc3e9d33fbba82f34be079cb91dff75fe83d3246843c3c698bce7513f09,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/README
66,bbf68d05a57ceb953171a7951fde0f98,f44c1ba5bff0103990176727c64603ece0c22eeae2ea424c2a217908c642ae2b,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/tos_AMSRE_L3_v7_200206-201012.nc.md5
1949273,b245802c1bd95a8094d579526cf984d3,a5271c055243cc22c7eb4d702ebc3db6e143668bc51f3362f5776cf2712465ad,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/docs/tosTechNote_AMSRE_L3_v7_200206-201012.pdf
244,b5d45cdf454f85c6bd034bf34f926a57,81912f3b516da54e298d36488edca7382d04d46c8ff0b3774d7578d65bb21a2f,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/sw/.listing
416,17374635e0b7d341267fa25c534d72e9,17b2acea2734a5778205eb13430717d6f5566d3114e87391abfa334090749789,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/sw/idl/.listing
489,793e2dc683e493700f517b9520460662,e215a762d4ba500b994ad4bd3ebc2ac8e77bf82a154d208ab83604503bdfd31b,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/sw/idl/README
3419,ba27e6ab9ef2b876051a43fd11a064d0,b61ff5e9b746b2c96b425c6453e63f82c9ba815e72f748bb00964d588b49540e,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/sw/idl/get_netcdf_vars.pro
2364,c652d7bc7c3eefa7d6135f64f9126b61,f1c98aa3e62ff7908926970a96bfff6df9b06a0f42e5e10268fdfa95aff76e4b,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/sw/idl/get_netcdf_global_atts.pro
1652,37a8b9dfc5e7dc8e1d68b3d0509bfbad,ee7b858709b4e01ace8faf11499ac5ab800f63a20766f734003b283705497a03,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/sw/idl/read_pcmdi.pro
2160,d844e7d7a37e66cf5a411f749bd0d6fa,2e536ccd79cb1e606dcaf37a20cdd507fc30493c0dde6444440217530af61baf,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/sw/matlab/read_nc_file_struct.m
198,32f54c6dd7b2051ea2727b7ffb209d8d,fd9033c804d83820cdbf348628e49fac6775d520576481fa2c9dfa21e486be30,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/sw/matlab/.listing
72,c82df12d2f0b08a0c50333f06aeda51d,6dcdec067f5fb304681f6245458480b900a22511a996d540d3145f1466f4657e,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/tosStderr_AMSRE_L3_v7_200206-201012.nc.md5
877,677edbad12625fea6b005b5812bd7782,3f810ca03c4a314886d9b1bc11d3798591b4c870975398343c0eb371e06e8a06,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/.listing
26722112,fb6d2175846a3aee8ba006bf84a357d9,2409d2bc5abfb6f735a1a88b27f605c30f7832d3f1fcf65a006b9d7ff14ff2bc,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/tosNobs_AMSRE_L3_v7_200206-201012.nc
26721732,2714a239aa3e6159b9acef0995b0b668,4cbacacddd523723bb1cf7c559f714c034df240e1d1654f9307daad0d9182b33,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/tosStderr_AMSRE_L3_v7_200206-201012.nc
26722020,9afef341c242b8b7d7ce13da0f1463e7,fad0891c79cc9f9ee41d05965ff61174f74d688f7fff49cba09554820c4882cb,podaac.jpl.nasa.gov/allData/amsre/L3/sst_1deg_1mo/tos_AMSRE_L3_v7_200206-201012.nc
DiabolicalTMcD commented 7 years ago

@bkirkbri I have a feeling I must not have done the pull correctly for it to be so small. Or that there are mostly symlinks in the hierarchy that lead back to hard files inside allData/topex.

bkirkbri commented 7 years ago

@DiabolicalTMcD Ok. I see that /OceanTemperature/amsre/L3/sst_1deg_1mo is a symlink to /allData/amsre/L3/sst_1deg_1mo. That directory is about 79MB, according to lftp du. Where does topex come into it? I don't see symlinks to that. Thanks!

DiabolicalTMcD commented 7 years ago

@bkirkbri Under /allData I have amsre/ and topex/

DiabolicalTMcD commented 7 years ago

@bkirkbri Just did another pull and again got 40MB.

bkirkbri commented 7 years ago

@DiabolicalTMcD Ok. /allData/topex/ is in issue #267 - please update there if you have a complete and public miror.