search

NOAA-PMEL / uafnciso

The Unlicense
4 stars 4 forks source link

Only getting one ISO record from crawling a catalog with six datasets #11

Closed rsignell-usgs closed 3 years ago

rsignell-usgs commented 6 years ago

I'm ran nciso on a THREDDS catalog with six global datasets but only got one ISO record back.

It turns out each of the six records is being named gamone.xml and overwriting the previous record.

My ./do_bathy bash script is:

#!/bin/bash
for full_cat in https://gamone.whoi.edu/thredds/global_bathy.xml
do
  echo $full_cat
  java -Xms1024m -Xmx1024m -jar ncISO-2.3.5.jar \
    -ts ${full_cat} -num 1000 -depth 20 -iso true \
    -waf ./data/global_bathy
done

I'm getting this response:

(IOOS3) rsignell@gamone:/usgs/data2/rsignell/waf> ./do_bathy
https://gamone.whoi.edu/thredds/global_bathy.xml
 INFO [main] (ServiceController.java:29) - Running createWaf in service controller...
 INFO [main] (CatalogCrawlerImpl.java:78) - maxDepth: 20 depth: 0 dataset.getFullName():Global Digital Elevation Models/ETOPO1 (1 arc minute - Worldwide); dataset.hasAccess(): true maxLeaves: 1000 leafcnt: 0
 INFO [main] (CatalogCrawlerImpl.java:82) - allowable service
 INFO [main] (CatalogCrawlerImpl.java:99) - adding mdc
 INFO [main] (CatalogCrawlerImpl.java:78) - maxDepth: 20 depth: 0 dataset.getFullName():Global Digital Elevation Models/ETOPO2  Version  2c  (2 arc minute - Worldwide); dataset.hasAccess(): true maxLeaves: 1000 leafcnt: 1
 INFO [main] (CatalogCrawlerImpl.java:82) - allowable service
 INFO [main] (CatalogCrawlerImpl.java:99) - adding mdc
 INFO [main] (CatalogCrawlerImpl.java:78) - maxDepth: 20 depth: 0 dataset.getFullName():Global Digital Elevation Models/SRTM30+  Version 1.0  (30 arc second - Worldwide); dataset.hasAccess(): true maxLeaves: 1000 leafcnt: 2
 INFO [main] (CatalogCrawlerImpl.java:82) - allowable service
 INFO [main] (CatalogCrawlerImpl.java:99) - adding mdc
 INFO [main] (CatalogCrawlerImpl.java:78) - maxDepth: 20 depth: 0 dataset.getFullName():Global Digital Elevation Models/SRTM30+  Version 6.0  (30 arc second - Worldwide); dataset.hasAccess(): true maxLeaves: 1000 leafcnt: 3
 INFO [main] (CatalogCrawlerImpl.java:82) - allowable service
 INFO [main] (CatalogCrawlerImpl.java:99) - adding mdc
 INFO [main] (CatalogCrawlerImpl.java:78) - maxDepth: 20 depth: 0 dataset.getFullName():Global Digital Elevation Models/Smith and Sandwell v9.1  (~ 60 arc second - Worldwide); dataset.hasAccess(): true maxLeaves: 1000 leafcnt: 4
 INFO [main] (CatalogCrawlerImpl.java:82) - allowable service
 INFO [main] (CatalogCrawlerImpl.java:99) - adding mdc
 INFO [main] (CatalogCrawlerImpl.java:78) - maxDepth: 20 depth: 0 dataset.getFullName():Global Digital Elevation Models/Smith and Sandwell v11.1  (~ 60 arc second - Worldwide); dataset.hasAccess(): true maxLeaves: 1000 leafcnt: 5
 INFO [main] (CatalogCrawlerImpl.java:82) - allowable service
 INFO [main] (CatalogCrawlerImpl.java:99) - adding mdc
 INFO [main] (WafService.java:45) - ncmlFilePath=./data/global_bathy/ncml/gamone.xml
 INFO [main] (WafService.java:45) - ncmlFilePath=./data/global_bathy/ncml/gamone.xml
 INFO [main] (WafService.java:45) - ncmlFilePath=./data/global_bathy/ncml/gamone.xml
 INFO [main] (WafService.java:45) - ncmlFilePath=./data/global_bathy/ncml/gamone.xml
 INFO [main] (WafService.java:45) - ncmlFilePath=./data/global_bathy/ncml/gamone.xml
 INFO [main] (WafService.java:45) - ncmlFilePath=./data/global_bathy/ncml/gamone.xml

and only one file in ./data/global_bathy/iso, gamone.xml, which contains the ISO of the last dataset (the others being overwritten).

Why am I getting gamone.xml every time instead of the dataset name?

kevin-obrien commented 6 years ago

Hi Rich -

I ran into this as well and it has to do with http vs https servers. I have a fix which I've tested on the UAF catalog and it seems to work. So, I will consult with Roland and check this into the repository...

noaaroland commented 3 years ago

Fixed in https://github.com/NOAA-PMEL/uafnciso/commit/16023217bdc6f2b617f13127da13a0ac7b2e3372 and released in 2.3.7.