My WMS subdaily layer problem

[batsc]

I've got a problem in trying to set up a WMS subdaily layer. I'm experimenting doing this by altering one of the layers provided in the demo, namely MYR4ODLOLLDY_global_10km. I've built the code on a RHEL6 system.

First, the following works fine. This is taken from 'epsg4326.map':

LAYER
NAME "MYR4ODLOLLDY_global_10km"
TYPE RASTER
STATUS ON
VALIDATION
"default_MYR4ODLOLLDY_global_10km_TIME" "TTTTTTT"
"MYR4ODLOLLDY_global_10km_TIME" "^[0-9]{7}$"
"default_MYR4ODLOLLDY_global_10km_YEAR" "YYYY"
"MYR4ODLOLLDY_global_10km_YEAR" "^[0-9]{4}$"
END
METADATA
"wms_title" "MYR4ODLOLLDY_global_10km"
"wms_extent" "-180 -90 180 90"
"wms_timeextent" "2014-10-04/2014-10-04/P1D"
"wms_timedefault" "2014-10-04"
END
DATA "/data/local/batsc/Work/WebService/onearth_setup/data/EPSG4326/MYR4ODLOLLDY_global_10km//%MYR4ODLOLLDY_global_10km_YEAR%/MYR4ODLOLLDY%MYR4ODLOLLDY_global_10km_TIME%_.mrf"
PROJECTION
"init=epsg:4326"
END
END

and the following URL, with the TIME parameter specified, retrieves an image successfully:

http://localhost:7777/onearth/wms/wms.cgi?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=MYR4ODLOLLDY_global_10km&CRS=EPSG%3A4326&STYLES=&WIDTH=1536&HEIGHT=768&BBOX=-135%2C-270%2C135%2C270&TIME=2014-10-04

For information (which may or may not indicate if something is wrong with my setup), if I specify a TIME in the URL outside of the 'wms_timeextent' (e.g. 2014-10-03) I get the following error:

msWMSLoadGetMapParams(): WMS server error. Unable to access -- corrupt, empty or missing file

Now, I have copied the MYR4ODLOLLDY_global_10km data and given it a subdaily timestamp of 12UTC, i.e.:

batsc@localhost EPSG4326 $ ls -R MYR4ODLOLLDY_global_10km_test
MYR4ODLOLLDY_global_10km_test:
2014 YYYY

MYR4ODLOLLDY_global_10km_test/2014:
MYR4ODLOLLDY2014277120000_.idx MYR4ODLOLLDY2014277120000_.mrf MYR4ODLOLLDY2014277120000_.ppg

MYR4ODLOLLDY_global_10km_test/YYYY:
MYR4ODLOLLDYTTTTTTTTTTTTT_.idx MYR4ODLOLLDYTTTTTTTTTTTTT_.mrf MYR4ODLOLLDYTTTTTTTTTTTTT_.ppg

and I've specified this test layer in the map file as:

LAYER
NAME "MYR4ODLOLLDY_global_10km_test"
TYPE RASTER
STATUS ON
VALIDATION
"default_MYR4ODLOLLDY_global_10km_test_TIME" "TTTTTTTTTTTTT"
"MYR4ODLOLLDY_global_10km_test_TIME" "^[0-9]{13}$"
"default_MYR4ODLOLLDY_global_10km_test_YEAR" "YYYY"
"MYR4ODLOLLDY_global_10km_test_YEAR" "^[0-9]{4}$"
END
METADATA
"wms_title" "MYR4ODLOLLDY_global_10km_test"
"wms_extent" "-180 -90 180 90"
"wms_timeextent" "2014-10-04T12:00:00Z/2014-10-04T12:00:00Z/P1D"
"wms_timedefault" "2014-10-04T12:00:00Z"
END
DATA "/data/local/batsc/Work/WebService/onearth_setup/data/EPSG4326/MYR4ODLOLLDY_global_10km_test//%MYR4ODLOLLDY_global_10km_test_YEAR%/MYR4ODLOLLDY%MYR4ODLOLLDY_global_10km_test_TIME%_.mrf"
PROJECTION
"init=epsg:4326"
END
END

When I use the following URL with an appropriate TIME parameter:

http://localhost:7777/onearth/wms/wms.cgi?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=MYR4ODLOLLDY_global_10km_test&CRS=EPSG%3A4326&STYLES=&WIDTH=1536&HEIGHT=768&BBOX=-135%2C-270%2C135%2C270&TIME=2014-10-04T12%3A00%3A00Z

I get the following response:

Invalid TIME format, must be YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ

Specifying just TIME=2014-10-04 leads to the msWMSLoadGetMapParams(): WMS server error. Unable to access -- corrupt, empty or missing file error.

I've tried various things to see if I can get any clues as to what may be wrong but I'm out of ideas. I'm afraid I don't know C otherwise I'd play with the source code to output what it thinks is the time parameter specified and why it might be invalid.

Is there anything here that hints at what might be wrong? Any advice would be greatly appreciated.

[jtroberts]

It looks like you need to include an additional [LAYER]_SUBDAILY variable in the map file. Try the following map configuration:

LAYER
NAME "MYR4ODLOLLDY_global_10km_test"
TYPE RASTER
STATUS ON
VALIDATION
"default_MYR4ODLOLLDY_global_10km_test_TIME" "TTTTTTT"
"MYR4ODLOLLDY_global_10km_test_TIME" "^([0-9]|T){7}$"
"default_MYR4ODLOLLDY_global_10km_test_YEAR" "YYYY"
"MYR4ODLOLLDY_global_10km_test_YEAR" "^([0-9]|Y){4}$"
"default_MYR4ODLOLLDY_global_10km_test_SUBDAILY" "TTTTTT"
"MYR4ODLOLLDY_global_10km_test_SUBDAILY" "^([0-9]|T){6}$"
END
METADATA
"wms_title" "MYR4ODLOLLDY_global_10km_test"
"wms_extent" "-180 -90 180 90"
"wms_timeextent" "2014-10-04T12:00:00Z/2014-10-04T12:00:00Z/PT24H”
"wms_timedefault" "2014-10-04T12:00:00Z"
END
DATA "/data/local/batsc/Work/WebService/onearth_setup/data/EPSG4326/MYR4ODLOLLDY_global_10km_test//%MYR4ODLOLLDY_global_10km_test_YEAR%/MYR4ODLOLLDY%MYR4ODLOLLDY_global_10km_test_TIME%%MYR4ODLOLLDY_global_10km_test_SUBDAILY%_.mrf"
PROJECTION
"init=epsg:4326"
END
END

Are you creating the map files manually? If you use the oe_configure_layer.py tool and have a subdaily time period specified, then the map files should automatically be created with the correct configurations.

Best, Joe

[batsc]

Thanks, yes I was trying manually but have now tried with the configure layer script, which generates the map file with the settings you've shown above. Unfortunately I'm still seeing the same error.

I also noticed in the endpoint configuration documentation that the TWMS endpoints must exist for time snapping to work for WMS, and I had not set these up. Now that I have I get a more appropriate error for the non-subdaily layer when specifying a date outside of the extent (i.e. WMS server error. Unable to access -- invalid TIME for LAYER). For the test subdaily layer, not specifying a TIME parameter in the URL successfully leads to a retrieved image. Specifying TIME=2014-10-04 leads to this same (invalid time) error, whereas TIME=2014-10-04T12%3A00%3A00Z gives Invalid TIME format, must be YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ.

The WMTS endpoint works perfectly, and I can specify TIME=2014-10-04T12%3A00%3A00Z in a GETTILE URL with success. However I need the WMS endpoint to work.

Out of curiousity is the 'with_python' necessary for the mapserver build? I couldn't get this to work because I'm doing a local build, and make tried to install the mapscript file in a location I did not have permission for. It built fine using with_python=0.

What else might I try to find some clues about what is going wrong?

[jtroberts]

Does your request work if you replace the URL encoded "%3A" with the actual ":" character? I think the problem is that one of the modules is not handling the URL encoded values correctly. I tested a request successfully using ":" - but when I tested using "%3A" I got the same error that you reported. I'll look into a software fix for that. There might also be a workaround using mod_rewrite or something like that.

I don't think with_python is necessary for the MapServer build, unless you want to use MapScript. If everything built fine then you should be okay.

[batsc]

Yes it does, great! However I would really benefit from the potential fix you mention because I'm using an OpenLayers client and it looks like this converts the ':' to %3A in TIME updates (which as I mention work fine for WMTS, but not as it seems WMS).

Many thanks for your help.

[jtroberts]

I'll leave this ticket open until we get the fix in. A lot of people use OpenLayers, so it would definitely be good to get it working right. Thanks for helping us discover this issue.

[jtroberts]

I just pushed a fix for the TIME problem with URL encoded characters. It is in the OE 1.3.1 release. You should be able to make the requests successfully using OpenLayers now.

[batsc]

Thanks very much, that works perfectly.