COAWST Barnegat Bay Hindcast sci-wmi erroring

[srstsavage]

This may be mentioned in another issue, but tiles for COAWST Barnegat Bay Hindcast are erroring:

http://cmgdata.usgsportals.net/#module-metadata/7e090bc3-6fbc-313d-aa0b-26e49ec723be/2bcf181a-83bb-41e2-bdc3-c405ce3e5c58

Looks to be a problem with the sci-wms?

http://sci-wms.whoi.edu/wms/datasets/bbleh_spring2012_00_dir_roms/show

[ayan-usgs]

Hi @shane-axiom,

Thanks for bringing this problem up. After some preliminary sleuthing it looks to be a problem with styling. Specifically, sci-wms has a limited number of styling options; it doesn't know what to do with a boxfill style. Support for it can be added though... can you elaborate on what on effect boxfill should have?

[daf]

@ayan-usgs every GetCaps link on that server was 500'ing when @shane-axiom filed this issue, it looks to be resolved now. You're likely seeing something different.

[ayan-usgs]

@daf

Okay. Thanks for the info!

[kwilcox]

The issue here (now) is that the datasets were being served with ncWMS and they are now being served with sci-wms. The CSW harvester isn't smart enough to look at the GetCaps to see which server it actually was and is still defaulting to ncWMS (so you are seeing the client try to use the ncWMS styles). I'll update the CSW harvester with some logic to test which WMS server the datasets is being served from.

[rsignell-usgs]

@kwilcox , phew, I'm glad this isn't a sci-wms problem!

[kwilcox]

@rsignell-usgs this is actually a sci-wms issues now. It is erroring when using a TIME parameter. @ayan-usgs can you take a look at this?

Working

http://sci-wms.whoi.edu/wms/datasets/use_fmrc_coawst_4_use_best?LAYERS=Hwave&STYLES=filledcontours_jet&FORMAT=image%2Fpng&TRANSPARENT=true&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A3857&BBOX=-6887893.491875,2504688.5425,-6261721.35625,3130860.678125&WIDTH=256&HEIGHT=256

Not working

http://sci-wms.whoi.edu/wms/datasets/use_fmrc_coawst_4_use_best?LAYERS=Hwave&STYLES=filledcontours_jet&FORMAT=image%2Fpng&TRANSPARENT=true&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A3857&BBOX=-6887893.491875,2504688.5425,-6261721.35625,3130860.678125&WIDTH=256&HEIGHT=256&TIME=2015-08-27T16%3A00%3A00Z
http://sci-wms.whoi.edu/wms/datasets/use_fmrc_coawst_4_use_best?LAYERS=Hwave&STYLES=filledcontours_jet&FORMAT=image%2Fpng&TRANSPARENT=true&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A3857&BBOX=-6887893.491875,2504688.5425,-6261721.35625,3130860.678125&WIDTH=256&HEIGHT=256&TIME=2012-06-27T16%3A00%3A00Z

[ayan-usgs]

@kwilcox,

Yep, I can take a look at the TIME parameter issue this afternoon.

[rsignell-usgs]

I think the behavior should be to plot the closest in time to what is requested. So if data before the start is requested, you get the 1st point. If data past the end is requested, you get the last point.

But I don't understand why @shane-axiom 2nd wms request isn't working, since there is data at 2015-08-27 16:00:00 UTC. Here's the ncWMS/Godiva plot that shows this:

[kwilcox]

@rsignell-usgs its a bug in sci-wms...

[ayan-usgs]

@kwilcox,

I'm unable to reproduce the error on the current version of sci-wms on my dev environment.

@rsignell-usgs: Are are there logs from the server you can provide me with? If there aren't, we might be able to get more information by setting DEBUG=True -- doing that doesn't make me feel warm and cozy, but it'll tell us more than we know now.

[kwilcox]

:+1: to @ayan-usgs suggestion

@rsignell-usgs Please try the "Update" button on the sci-wms front page for this dataset... it could be the DAP server was down when you originally added the dataset so the times didn't get cached... other than that you'll have to post the logs.

[rsignell-usgs]

I'm getting errors like this:

[2015-08-27 14:41:04,750: ERROR/MainProcess] Internal Server Error: /wms/datasets/bbleh_spring2012_00_dir_roms
Traceback (most recent call last):
  File "/home/usgs/miniconda/envs/sci-wms/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/usgs/miniconda/envs/sci-wms/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/usgs/miniconda/envs/sci-wms/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/usgs/python/sci-wms/wms/views.py", line 263, in get
    request = enhance_getmap_request(dataset, layer, request)
  File "/home/usgs/python/sci-wms/wms/views.py", line 122, in enhance_getmap_request
    colormap=wms_handler.get_colormap(request),
  File "/home/usgs/python/sci-wms/wms/wms_handler.py", line 187, in get_colormap
    requested_cm = request.GET.get(parameter).split(',')[0].split('_')[1]
IndexError: list index out of range

[ayan-usgs]

@rsignell-usgs,

That looks like the problem I saw earlier about an unsupported style being requested for the bbleh_spring2012_00_dir_roms dataset. I don't think that would where we would be seeing problems with the TIME parameter.

[rsignell-usgs]

@ayan-usgs , oh, right. Here's the time problem:

[2015-08-27 14:49:42,884: ERROR/MainProcess] Internal Server Error: /wms/datasets/use_fmrc_coawst_4_use_best
Traceback (most recent call last):
  File "/home/usgs/miniconda/envs/sci-wms/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/usgs/miniconda/envs/sci-wms/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/usgs/miniconda/envs/sci-wms/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/usgs/python/sci-wms/wms/views.py", line 271, in get
    response = getattr(dataset, reqtype.lower())(layer, request)
  File "/home/usgs/python/sci-wms/wms/models/datasets/sgrid.py", line 216, in getmap
    time_index, time_value = self.nearest_time(layer, request.GET['time'])
  File "/home/usgs/python/sci-wms/wms/models/datasets/netcdf.py", line 243, in nearest_time
    num_date = round(nc4.date2num(time, units=units, calendar=calendar))
  File "netCDF4/_netCDF4.pyx", line 4570, in netCDF4._netCDF4.date2num (netCDF4/_netCDF4.c:49367)
TypeError: can't subtract offset-naive and offset-aware datetimes

Could this be the same problem described here: https://github.com/Unidata/netcdf4-python/issues/449 and that I've got a different version of netCDF4?
My version is netcdf4 1.1.9

[rsignell-usgs]

Here's the conda list -e from my sci-wms conda environment:

(sci-wms)usgs@gam:~$ conda list -e
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
argh=0.26.1=py27_0
basemap=1.0.7=np19py27_0
cairo=1.12.18=4
certifi=14.05.14=py27_0
curl=7.43.0=1
dateutil=2.4.1=py27_0
dj-static=0.0.6=py27_0
django=1.7.3=py27_0
django-autoslug=1.8.0=py27_0
django-debug-toolbar=1.3.0=py27_0
django-extensions=1.5.5=py27_0
django-grappelli=2.6.5=py27_0
django-typed-models=0.4.0=py27_0
djangorestframework=3.1.3=py27_0
fontconfig=2.11.1=4
freetype=2.5.2=2
geos=3.4.2=1
gunicorn=19.1.0=py27_0
h5py=2.5.0=np19py27_3
hdf5=1.8.15.1=1
isodate=0.5.1=py27_1
jsonfield=1.0.3=py27_1
krb5=1.13.2=0
libgfortran=1.0=0
libnetcdf=4.3.3.1=1
libpng=1.6.17=0
libspatialindex=1.8.5=1
libxml2=2.9.2=0
matplotlib=1.4.3=np19py27_2
nco=4.5.1=0
nco-bindings=0.0.2=nppy27_1
netcdf4=1.1.9=np19py27_0
numpy=1.9.2=py27_1
openblas=0.2.14=1
openssl=1.0.1k=1
pandas=0.16.2=np19py27_0
pastedeploy=1.5.2=py27_1
pathtools=0.1.2=py27_0
pip=7.1.2=py27_0
pixman=0.26.2=0
proj4=4.9.1=2
py2cairo=1.10.0=py27_2
pyaxiom=1.0.3=nppy27_0
pyncml=0.0.8=nppy27_0
pyparsing=2.0.3=py27_0
pyproj=1.9.4=np19py27_1
pyqt=4.11.3=py27_1
pysgrid=0.1.0=np19py27_0
python=2.7.10=0
python-dateutil=2.4.2=py27_1
pytz=2015.4=py27_0
pyugrid=0.1.5=np19py27_0
pyyaml=3.11=py27_1
qt=4.8.6=3
readline=6.2=2
rtree=0.7.0=py27_0
sci-wms=1.1.0=0
scipy=0.16.0=np19py27_1
setproctitle=1.1.8=py27_1
setuptools=18.1=py27_0
shapely=1.5.8=np19py27_0
sip=4.16.5=py27_0
six=1.9.0=py27_0
south=1.0.2=py27_1
sqlite=3.8.4.1=1
sqlparse=0.1.15=py27_0
ssl_match_hostname=3.4.0.2=py27_0
static3=0.6.1=py27_0
system=5.8=2
tk=8.5.18=0
tornado=4.2.1=py27_0
udunits2=2.2.20=0
watchdog=0.8.3=py27_0
werkzeug=0.10.4=py27_0
wheel=0.24.0=py27_1
yaml=0.1.6=0
zlib=1.2.8=0

[ayan-usgs]

@rsignell-usgs,

Well, upgrading to netCDF4 1.1.9 is giving me the same error you're seeing.

[kwilcox]

@ayan-usgs Most likely related to https://github.com/Unidata/netcdf4-python/issues/357 if you are coming from an old (<1.1.5) version of netcdf4-python.

[ayan-usgs]

@kwilcox You hit it right on the nose. :smile:

[rsignell-usgs]

@ayan-usgs, are you pursuing?

[ayan-usgs]

@rsignell-usgs: See https://github.com/sci-wms/sci-wms/issues/76

[rsignell-usgs]

Fixed in https://github.com/sci-wms/sci-wms/commit/7dfad42d0d788cf1ba131597d9c040792b465ca1