Allow input files to be grabbed via THREDDS

[rsignell-usgs]

Per Rick's request, we need a way for folks to get the input files. One way would be ftp, but a easy short term solution is to zip the input files and add .zip to the THREDDS datasetScan list.

The zip files could then be downloaded using the TDS direct download service.

[rsignell-usgs]

I've modified the thredds catalog /var/www/thredds_instance/content/thredds/comt_1_archive_full.xml to include zip files:

 <filter>
<!-- Exclude .svn directories -->
 <exclude wildcard=".svn" atomic="false" collection="true"/>

 <include wildcard="*.nc"/>
 <include wildcard="*.nc"/>
 <include wildcard="*.cdf"/>
 <include wildcard="*.txt"/>
 <include wildcard="*.hdf"/>
 <include wildcard="*.ncml"/>
 <include wildcard="*.grib"/>
 <include wildcard="*.grib2"/>
 <include wildcard="*.docx"/>
 <include wildcard="*.zip"/>
 </filter>

As a test, I zipped the input in this one directory:

[testbed@testbed2 Input]$ pwd
/data/comt_1_archive/inundation_tropical/UND_ADCIRC/Hurricane_Ike_2D_final_run_with_waves/Inpu
[testbed@testbed2 Input]$ ls -sailh
total 2.0G
173277220 4.0K drwxrwxr-x. 2 testbed testbed 4.0K Dec 16  2015 .
173277191 4.0K drwxrwxr-x. 4 testbed testbed 4.0K Nov 24 12:45 ..
173277402  86M -rw-rwxr--. 1 testbed testbed  86M Feb  6  2013 fort.13
173277403  51M -rwxrwxr-x. 1 testbed testbed  51M Feb  6  2013 fort.14
173277404  64K -rwxrwxr-x. 1 testbed testbed  61K Feb  6  2013 fort.15
173277405 4.0K -rwxrwxr-x. 1 testbed testbed  154 Feb  6  2013 fort.22
173277406 400M -rwxrwxr-x. 1 testbed testbed 400M Feb  6  2013 fort.221
173277407 799M -rwxrwxr-x. 1 testbed testbed 799M Feb  6  2013 fort.222
173277408 193M -rwxrwxr-x. 1 testbed testbed 193M Feb  6  2013 fort.223
173277409 386M -rwxrwxr-x. 1 testbed testbed 386M Feb  6  2013 fort.224
173277410 4.0K -rw-rwxr-x. 1 testbed testbed 1.2K Feb  6  2013 fort.26
173277411  26M -rw-rwxr--. 1 testbed testbed  26M Feb  6  2013 fort.68
173277412  41M -rw-rwxr--. 1 testbed testbed  41M Feb  6  2013 fort.68.nc
173277413 4.0K -rwxrwxr-x. 1 testbed testbed  935 Feb  6  2013 swaninit

[testbed@testbed2 Input]$ zip input.zip fort.?? fort.??? swaninit
  adding: fort.13 (deflated 77%)
  adding: fort.14 (deflated 75%)
  adding: fort.15 (deflated 71%)
  adding: fort.22 (deflated 29%)
  adding: fort.26 (deflated 61%)
  adding: fort.68 (deflated 69%)
  adding: fort.221 (deflated 69%)
  adding: fort.222 (deflated 68%)
  adding: fort.223 (deflated 74%)
  adding: fort.224 (deflated 72%)
  adding: swaninit (deflated 73%)

[testbed@testbed2 Input]$ ls -sailh input.zip
173277399 579M -rw-rw-r--. 1 testbed testbed 579M Dec 16  2015 input.zip

and I was able to download successfully by starting at the comt_1 full catalog and then click,click, clicking down to the input folder for the specific run where you find the zipfile "dataset" and finally you need to select the HTTPServer link to download the zip file.

I tried downloading the file from COMT to my laptop here in Woods Hole, and I was able to get he 606MB file in just under two minutes. And everything looks good inside.

[rsignell-usgs]

So if it wasn't clear from all that, the URL that will download a zip file of the input data (that we could share via e-mail, or put behind a link on a webpage) was: http://comt.sura.org/thredds/fileServer/comt_1_archive_full/inundation_tropical/UND_ADCIRC/Hurricane_Ike_2D_final_run_with_waves/Input/input.zip

[rsignell-usgs]

I didn't zip the fort.68.nc file that was in that input directory, partly because it's already exposed via THREDDS and partly because there was a fort.68 ascii file as well, so it seemed like perhaps some experiment?

[rluettich]

Fort.67 and fort.68 files are not asci files but rather are native binary re-start files. They are of limited value since binary files are not necessarily compatible between machines. We have started writing / reading these in netCDF to get around this issue, but I don’t recall how carefully this was followed at the time.

I would say don’t bother zipping fort.67 or fort.68 files as they are of limited value anyway.

Thanks again for the help with this, Rick

From: Rich Signell [mailto:notifications@github.com] Sent: Wednesday, December 16, 2015 3:31 PM To: ioos/comt Subject: Re: [comt] Allow input files to be grabbed via THREDDS (#39)

I didn't zip the fort.68.nc file that was in that input directory, partly because it's already exposed via THREDDShttp://comt.sura.org/thredds/catalog/comt_1_archive_full/inundation_tropical/UND_ADCIRC/Hurricane_Ike_2D_final_run_with_waves/Input/catalog.html and partly because there was a fort.68 ascii file as well, so it seemed like perhaps some experiment?

— Reply to this email directly or view it on GitHubhttps://github.com/ioos/comt/issues/39#issuecomment-165230533.