Modify ncWMS requests for ncWMS 2.x

[rod-glover]

Resolves #187

Demo:

• This version of PDP making requests to ...
• The translation service, which is using the pcic_meta database, and makes requests to ...
• The ncWMS 2.5.1 dynamic datasets instance, which has been updated to mount the PDP (pcic_meta) datasets.

Testing:

• [x] BC Station Data - PCDS: does not use ncWMS
• [x] High-Resolution PRISM Climatology
  • presently fails because of dataset file issues (non-montonic time dimension)
  • [x] Get input from Faron re. datasets and UI
  • [x] Update datasets and reindex them
  • [x] Implement UI changes
• [x] Statistically Downscaled GCM Scenarios
  • map layers work
  • fails on GetLegendGraphic - needs analysis.
  • Problem: NUMCOLORBANDS=254, > 249
  • Fixed
• [x] Daily Gridded Meteorological Datasets
  • map layers work
  • fails on GetLegendGraphic - fixed
  • palette = default is different
  • x-Rainbow appears to be identical ncWMS 2 predefined palettes: fixed

• [x] Gridded Hydrologic Model Output

  • [x] needs access to /storage/data/projects/hydrology/dataportal/CMIP5/VICGL
  • [x] file not found
  • confirmed mount of /storage/data/projects/hydrology/dataportal
  • confirmed file exists
  • executing ls in container shows (a) file exists, (b) permission denied.
  • VICGL directory lacks x permission for all users. Changed permission. Files now accessible.
  • [x] GetLegendGraphic and GetMap requests undefined palette - error for ncWMS 2 (ncWMS 1 evidently used default)
  • [x] GetMap requests SRS=EPSG:3005 - error:

    
    The CRS EPSG:3005 is not supported by this server
    Cause:
        No ‘CoordinateReferenceSystem’ object found for code “3005”. However only a subset of the EPSG geodetic dataset has been queried. See http://sis.apache.org/epsg.html for instruction about how to install the full EPSG database.

• [x] Modelled Streamflow Data: does not use ncWMS

Fixes required to PDP:

• [x] STYLES parameter: boxfill renamed default
• [x] NUMCOLORBANDS parameter: Values limited to 1 ... 249: Use 249 (not 254)
• [x] Use x-Rainbow palette instead of default
• [x] Replace undefined palette with x-Rainbow (Gridded Hydrologic Model Output)
• [x] Improve configuration for BC PRISM portal to handle new datasets.

Fixes required externally:

• [x] Add custom palettes to ncWMS 2 to support absent ncWMS 1 palettes used in PDP.
• [x] Load full EPSG geodetic dataset into ncWMS 2 instances.
• [x] Split up PRISM datasets into single-resolution files.

[rod-glover]

Discrepancies in PRISM portal

Below are side-by-side screenshots of current (production) PDP and demo of new PDP.

TL;DR: The climatology layers look quite different in the new, although the monthly timeseries are unchanged. I suspect a data issue, but am not certain. Thoughts?

Monthly timeseries (OK)

The monthly timeseries displays are unchanged (this is good):

Monthly timeseries 1950-2007 Precip - current (L), new (R)

Monthly timeseries 1950-2007 Tmax - current (L), new (R)

Monthly timeseries 1950-2007 Tmax - current (L), new (R)

Climatologies

The climatologies look quite different. I suspect some kind of data problem, since the scales are identical and the monthly timeseries look identical.

1970-2000

1970-2000 Precip - current (L), new (R)

1970-2000 Tmax - current (L), new (R)

1970-2000 Tmin - current (L), new (R)

1981-2010

1981-2010 Precip - current (L), new (R)

1981-2010 Tmax - current (L), new (R)

1981-2010 Tmin - current (L), new (R)

[rod-glover]

Discrepancies in Gridded Hydrologic Model Output

EDIT: Not a problem.

EDIT: Current (prod) PDP doesn't specify STYLES (i.e., colourscale) for the map tiles, or its equivalent, PALETTE for the legend graphic (colour bar). By visual examination, I determined that the default colour scale (which is what presumably is happening in prod PDP) is x-Rainbow. This is specified explicitly in new PDP. I think new PDP is actually correct, and old PDP is visually wrong.

All else appears identical in the layer requests and information being processed by the frontend:

The scales are the same, the layer unique_id's are the same. Metadata is the same.

One difference is that ncWMS 2.x doesn't permit NUMCOLORBANDS > 249; ncWMS 1.x permitted (and we used) 254. Hard to imagine that would make any difference.

Thoughts?

A few examples:

ACCESS1-0 Baseflow

CanESM2 Baseflow

CanESM2 Precipitation

[rod-glover]

No more Modelled Streamflow Data portal?

EDIT: Clarified.

Prod currently gives 404: https://data.pacificclimate.org/portal/hydro_stn/map/

But same path on new PDP gives a map.

[jameshiebert]

Re: "No more Modelled Streamflow Data portal":

The links from our [website]() show: https://data.pacificclimate.org/portal/hydro_stn_cmip5/map/ for CMIP5 and https://data.pacificclimate.org/portal/hydro_stn_archive/map/ for CMIP3

[jameshiebert]

Re: "Discrepancies in Gridded Hydrologic Model Output"

I'm comfortable with nominal changes in the visualized representation of the datasets. The goal of the PDP is to deliver data, not necessarily to visualize data. The maps that are shown are essentially an arbitrary single day within a 55,000 time step simulation. So an "accurate" visual representation isn't really meaningful.

tl;dr it's understandable for the plots to change a bit. Just be prepared to explain it if questions come up.

[corviday]

Response to this comment.

Monthly Timeseries:

No discrepencies, this files are unchanged.

PR Climatologies:

Previously, these files contained twelve monthly values and a much larger annual value, with the annual value determining the top of the scale. Presumably the annual value reflected cumulative precipitation across all months. Here's a graph of one of the original, annual-is-cumulative datasets, note the much higher final (annual) value, the one that goes upwards and backwards across the graph:

It's the annual value that pegs the top end of the scale in the old dataset, and presumably the annual value that is displayed on the portal page (it says "June 1996" which could be either the annual value or the monthly value for June). Since the values in the new monthly-only dataset are much lower, I would expect to see something similar to what you show here - values clustered at the low end of the scale - if you use the same colour scale as the monthly-plus-annual dataset with the monthly-only dataset.

Alas, we probably need separate colour scales for the new monthly and annual datasets, ugh.

Temperature Climatologies I'm not sure what's going on here yet, I will examine the old files. I suspect it may be similar, like maybe the annual tmin and tmax value represent whole-year minimums and maximums and make the scales weird.

[corviday]

Temperature Climatologies

For these datasets, it looks like the annual value is the mean of the monthly values, not anything weirder like the minimum or maximum. Here's one of the old monthly+annual tmin datasets:

Here's one of the old monthly+annual tmax datasets:

My suspicion is that in these cases, the data map discrepancy we're seeing is just caused by switching the displayed timestamp from displaying June (annual timestamp - the mean values, centred nicely in the middle of the scale) to June (monthly timestamp - maximum values at top of scale). It's still pretty legible, so I don't know if we need to fix it.

[rod-glover]

@corviday , your deductions are completely confirmed by the observation that the annual datasets in the new app exactly match the "monthly" datasets in the old one. Thank you!

[rod-glover]

Verifying data downloads:

• [x] BC Station Data - PCDS: does not use ncWMS
  • looks plausible; file sizes identical; ncview does not like the files from prod or demo, however (NetCDF: Numeric conversion not representable)
  • ASCII download: files identical
• [x] High-Resolution PRISM Climatology
  • looks plausible; formerly merged datasets (monthly, annual) are split so difficult to make a direct comparison, but ncview of them is comparable and likely differ only because of different data ranges
• [x] Statistically Downscaled GCM Scenarios
  • downloaded file sizes identical; ncview of downloaded files identical on first timestep
• [x] Daily Gridded Meteorological Datasets
  • downloaded file sizes identical; ncview of downloaded files identical on first timestep
• [x] Gridded Hydrologic Model Output
  • downloaded file sizes identical; ncview of downloaded files identical on first timestep
• [x] Modelled Streamflow Data: does not use ncWMS
  • downloaded files identical

[rod-glover]

@jameshiebert , @corviday : All portals appear to be working correctly. I've verified file downloads from each portal, to the extent I can with a moderate effort. With the discussion and resulting modifications above, do you think that this PR is ready to merge?

[corviday]

I think so.

[rod-glover]

James also agrees that we are ready to merge.