search

USGS-CMG / usgs-cmg-portal

do_convert.sh
6 stars 13 forks source link

Add model output from Dave Ralston #212

Closed rsignell-usgs closed 7 years ago

rsignell-usgs commented 8 years ago

Dave Ralston is sharing some data that we want to show up on our Hurricane Sandy portal at http://cmgdata.usgsportals.net/ which is outside the WHOI/USGS domain.

Dave has his data shared on http://clancy.whoi.edu:8080/thredds/catalog/data1/dralston/hudson/sandy/catalog.html

Two requests:

1) We can't see clancy.whoi.edu from the outside world. Could port 8080 be opened so we can access data from the THREDDS Data Server there?

2) We need to be able to access the ISO metadata service on this machine, and it looks like it's not enabled. Can you enable it by editing the threddsConfig.xml and uncommenting the nciso section? After editing this should look like this:

https://github.com/rsignell-usgs/xml/blob/master/THREDDS/geoport/threddsConfig.xml#L201-L205

rsignell-usgs commented 8 years ago

@gregpike , okay, I can get to clancy, and ncISO is working -- I see the dataset showing up in cells [7] and [8] here: http://nbviewer.jupyter.org/gist/rsignell-usgs/0fc4f5cd1f85ec411458

The dataset is not showing up the in portal, however, and after investigation, I discovered that the WMS service is not working. If you click the WMS service at http://clancy.whoi.edu:8080/thredds/catalog/data1/dralston/hudson/sandy/sandy009/catalog.html?dataset=data1/dralston/hudson/sandy/sandy009/00_dir_roms.ncml you get a 403 error.

This usually means that WMS is not enabled on the Clancy TDS (it's disabled by default).

Can you or @dkralston edit the threddsConfig.xml, enable WMS, and then restart thredds or tomcat?

The lines in threddsConfig.xml should look like this:


  <!--
  The WMS Service is off by default.
  -->
  <WMS>
    <allow>true</allow>
    <allowRemote>false</allowRemote>
    <maxImageWidth>2048</maxImageWidth>
    <maxImageHeight>2048</maxImageHeight>
  </WMS>
gregpike commented 8 years ago

Hi Rich,

Give this another try, it should be working now.

Best regards, /Greg

On Mon, Jun 6, 2016 at 9:54 AM, Rich Signell notifications@github.com wrote:

@gregpike https://github.com/gregpike , okay, I can get to clancy, and ncISO is working -- I see the dataset showing up in cells [7] and [8] here: http://nbviewer.jupyter.org/gist/rsignell-usgs/0fc4f5cd1f85ec411458

The dataset is not showing up the in portal, however, and after investigation, I discovered that the WMS service is not working. If you click the WMS service at

http://clancy.whoi.edu:8080/thredds/catalog/data1/dralston/hudson/sandy/sandy009/catalog.html?dataset=data1/dralston/hudson/sandy/sandy009/00_dir_roms.ncml you get a 403 error.

This usually means that WMS is not enabled on the Clancy TDS (it's disabled by default).

Can you or @dkralston https://github.com/dkralston edit the threddsConfig.xml, enable WMS, and then restart thredds or tomcat?

The lines in threddsConfig.xml should look like this:

true false 2048 2048

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/212#issuecomment-223965765, or mute the thread https://github.com/notifications/unsubscribe/ALPX38YU4ZRo0zCDMTL_-JEiJimBwwe2ks5qJCaagaJpZM4Iu55J .

rsignell-usgs commented 8 years ago

@gregpike, the WMS is responding now, but the response is an error I haven't seen before.

The WMS getCapabilities request: http://clancy.whoi.edu:8080/thredds/wms/data1/dralston/hudson/sandy/sandy009/00_dir_roms.ncml?service=WMS&version=1.3.0&request=GetCapabilities

produces this error:

<ServiceExceptionReport xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.3.0" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wms/1.3.0/exceptions_1_3_0.xsd">
<ServiceException>
Unexpected error of type java.io.FileNotFoundException
</ServiceException>
<StackTrace>
<![CDATA[
java.io.FileOutputStream.open(Native Method)java.io.FileOutputStream.<init>(FileOutputStream.java:221)java.io.FileOutputStream.<init>(FileOutputStream.java:171)ucar.nc2.ncml.AggregationExisting.persistWrite(AggregationExisting.java:277)ucar.nc2.dataset.NetcdfDataset.close(NetcdfDataset.java:1276)uca
]]>
<![CDATA[
r.nc2.dataset.NetcdfDataset.close(NetcdfDataset.java:1286)ucar.nc2.dt.grid.GridDataset.close(GridDataset.java:511)thredds.server.wms.ThreddsWmsController.dispatchWmsRequest(ThreddsWmsController.java:255)uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.handleRequestInternal(AbstractWmsController.java:200)org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:146)org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)javax.servlet.http.HttpServlet.service(HttpServlet.java:624)org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)javax.servlet.http.HttpServlet.service(HttpServlet.java:731)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)thredds.servlet.filter.RequestQueryFilter.doFilter(RequestQueryFilter.java:118)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)thredds.servlet.filter.RequestCORSFilter.doFilterInternal(RequestCORSFilter.java:49)org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)thredds.servlet.filter.RequestPathFilter.doFilter(RequestPathFilter.java:94)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)thredds.server.RequestBracketingLogMessageFilter.doFilter(RequestBracketingLogMessageFilter.java:81)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)java.lang.Thread.run(Thread.java:745)
]]>
</StackTrace>
</ServiceExceptionReport>

Any ideas? If not we can ask Unidata.

gregpike commented 8 years ago

Hi Rich,

The error is a "file not found" error, so it's trying to open a file that doesn't exist (or maybe it's just a permissions problem?). I don't see the specific file name embedded in the output anywhere, so I don't know what it's trying to open. Does that help you chase it down any further?

Best regards, /Greg

On Mon, Jun 6, 2016 at 12:57 PM, Rich Signell notifications@github.com wrote:

@gregpike https://github.com/gregpike, the WMS is responding now, but the response is one I haven't seen before:

The WMS getCapabilities request:

http://clancy.whoi.edu:8080/thredds/wms/data1/dralston/hudson/sandy/sandy009/00_dir_roms.ncml?service=WMS&version=1.3.0&request=GetCapabilities

produces this error:

Unexpected error of type java.io.FileNotFoundException (FileOutputStream.java:221)java.io.FileOutputStream.(FileOutputStream.java:171)ucar.nc2.ncml.AggregationExisting.persistWrite(AggregationExisting.java:277)ucar.nc2.dataset.NetcdfDataset.close(NetcdfDataset.java:1276)uca ]]>

Any ideas? If not we can ask Unidata.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/212#issuecomment-224019644, or mute the thread https://github.com/notifications/unsubscribe/ALPX35f1UerlNzlz3SuS5-7PbBA28xjcks5qJFFugaJpZM4Iu55J .

rsignell-usgs commented 8 years ago

@gregpike, I can read data from this dataset just fine: http://nbviewer.jupyter.org/gist/rsignell-usgs/b674dbb82da733ea649184343adf61f3 so I think the "file not found" is actually referring to some java file not found, which to me indicates a problem with the THREDDS Data Server set up on this machine?

kwilcox commented 8 years ago

This could be permission errors when trying to write a cache file? It is not a THREDDS issue, but something in the installation

gregpike commented 8 years ago

Here's the relevant header from the error stack in the logs. I'm sure it's something simple, but we probably need someone more versed in Tomcat to point out the obvious. Any ideas, anyone?

Jun 07, 2016 7:50:00 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [root] in context with path [/thredds] threw exception dap4.core.util.DapException: dap4.core.util.DapException: Resource root not found

Best regards, /Greg

On Tue, Jun 7, 2016 at 8:06 AM, Kyle Wilcox notifications@github.com wrote:

This could be permission errors when trying to write a cache file? It is not a THREDDS issue, but something in the installation

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/212#issuecomment-224260493, or mute the thread https://github.com/notifications/unsubscribe/ALPX31E-xavw3yMCqlOQGfnFixcA2UBvks5qJV7LgaJpZM4Iu55J .

rsignell-usgs commented 8 years ago

Turns out changing the permissions on all tomcat files to user tomcat and restarting did the trick to get rid of the WMS getcapabilities problem, so now the dataset is being ingested by the portal, but it's being rejected because CORS is not enabled on Clancy.

We can test a WMS getMap request URL into the right side of this page

http://www.test-cors.org

and if it comes up with XHR status: 200 then CORS is enabled.

but if it comes up with XHR status: 0 then CORS is NOT enabled

Try dropping these URLs into http://www.test-cors.org

http://geoport-dev.whoi.edu/thredds/wms/coawst_4/use/fmrc/coawst_4_use_best.ncd?LAYERS=temp&ELEVATION=-0.03125&TIME=2016-05-02T00%3A00%3A00.000Z&TRANSPARENT=true&STYLES=areafill%2Frainbow&COLORSCALERANGE=-5.15%2C34.85&NUMCOLORBANDS=20&LOGSCALE=false&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&FORMAT=image%2Fpng&SRS=EPSG%3A4326&BBOX=-101.74989476701,11.888467352229,-53.253084362467,48.463925392208&WIDTH=256&HEIGHT=256

http://clancy.whoi.edu:8080/thredds/wms/data1/dralston/hudson/sandy/sandy009/00_dir_roms.ncml?LAYERS=salt&ELEVATION=-0.96875&TIME=2012-11-05T00%3A00%3A00.000Z&TRANSPARENT=true&STYLES=boxfill%2Frainbow&COLORSCALERANGE=1.996%2C30&NUMCOLORBANDS=20&LOGSCALE=false&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&FORMAT=image%2Fpng&SRS=EPSG%3A4326&BBOX=-74.430244254568,40.329155997624,-73.620737941393,42.747200795&WIDTH=256&HEIGHT=256

rsignell-usgs commented 8 years ago

@gregpike chatted: CORS is available on clancy at the Apache level... at least for a few more hours before I upgrade it. Not sure if that will flow down to THREDDS, though.

Looks like not, as the CORS test below still produces XHR status: 0

http://www.test-cors.org/#?client_method=GET&client_credentials=false&server_url=http%3A%2F%2Fclancy.whoi.edu%3A8080%2Fthredds%2Fwms%2Fdata1%2Fdralston%2Fhudson%2Fsandy%2Fsandy009%2F00_dir_roms.ncml%3FLAYERS%3Dsalt%26ELEVATION%3D-0.96875%26TIME%3D2012-11-05T00%253A00%253A00.000Z%26TRANSPARENT%3Dtrue%26STYLES%3Dboxfill%252Frainbow%26COLORSCALERANGE%3D1.996%252C30%26NUMCOLORBANDS%3D20%26LOGSCALE%3Dfalse%26SERVICE%3DWMS%26VERSION%3D1.1.1%26REQUEST%3DGetMap%26EXCEPTIONS%3Dapplication%252Fvnd.ogc.se_inimage%26FORMAT%3Dimage%252Fpng%26SRS%3DEPSG%253A4326%26BBOX%3D-74.430244254568%2C40.329155997624%2C-73.620737941393%2C42.747200795%26WIDTH%3D256%26HEIGHT%3D256&server_enable=true&server_status=200&server_credentials=false&server_tabs=remote

kwilcox commented 8 years ago

Usually apache would run on port 80 but THREDDS are running on port 8080. Can you proxy to tomcat through apache?

gregpike commented 8 years ago

Hmmm, no reason why we couldn't. I'll take a whack at it tonight and see what blows up.

Best regards, /Greg

On Fri, Jun 10, 2016 at 5:39 PM, Kyle Wilcox notifications@github.com wrote:

Usually apache would run on port 80 but THREDDS are running on port 8080. Can you proxy to tomcat through apache?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/212#issuecomment-225302529, or mute the thread https://github.com/notifications/unsubscribe/ALPX3-o-96LqBEOPB4XtsH0NT8HRvV1Oks5qKdmGgaJpZM4Iu55J .

rsignell-usgs commented 8 years ago

@gregpike , the CORS test above is still failing, so we can't see Dave's model output on the portal.

Is the latest plan to ask @dkralston when would be a good time to upgrade the OS on clancy so that we can install the latest TDS using Docker?

dkralston commented 8 years ago

Greg, let me know when you can work on it and I will clear off any running jobs.

Dave.

On Wed, Jun 22, 2016 at 7:46 AM, Rich Signell notifications@github.com wrote:

@gregpike https://github.com/gregpike , the CORS test above is still failing, so we can't see Dave's model output on the portal.

Is the latest plan to ask @dkralston https://github.com/dkralston when would be a good time to upgrade the OS on clancy so that we can install the latest TDS using Docker?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/212#issuecomment-227719545, or mute the thread https://github.com/notifications/unsubscribe/AIky9RKvlb2YYw6aM35ZoBlXt-o-Utdiks5qOSCGgaJpZM4Iu55J .

rsignell-usgs commented 8 years ago

Okay, we finally have an updated TDS on Ralston's clancy machine(using Docker), and CORS is enabled via NGINX (also using Docker) instead of using Apache.

@dkralston , can you please edit the NcML and add these attributes?

units of "deg_C" for temperature units of "0.001" for salinity

rsignell-usgs commented 8 years ago

BTW, the NcML in question is on clancy at: /data1/dralston/hudson/sandy/sandy009/00_dir_roms.ncml

dkralston commented 8 years ago

just did it -- let me know if it worked. I prefer "psu", but whatever.

On Wed, Jul 6, 2016 at 10:31 AM, Rich Signell notifications@github.com wrote:

BTW, the NcML in question is on clancy at: /data1/dralston/hudson/sandy/sandy009/00_dir_roms.ncml

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/212#issuecomment-230790032, or mute the thread https://github.com/notifications/unsubscribe/AIky9cQ8wqJWgi06vOcfXS2gQeXS93Ewks5qS7xJgaJpZM4Iu55J .

rsignell-usgs commented 7 years ago

It's working, so closing