opendap/hyrax_ncwms:1.16.2 wms broken

gajowi commented 4 years ago

Previously wms and godiva3 worked for/with /opendap/data/nc/coads_climatology.nc.html

such as at our production service: https://data-cbr.csiro.au/ncWMS2/Godiva3.html?server=https://data-cbr.csiro.au/ncWMS2/wms/lds/testdata/nc/coads_climatology.nc (also fine without tls...).

With 1.16.2 I think the related wms GetCapabilities response is OK but when getting data there is an error. 400 Bad Request

Could not read underlying data

(for url /ncWMS2/wms/lds/data/nc/coads_climatology.nc?FORMAT=image%2Fpng&TRANSPARENT=TRUE&STYLES=default-scalar%2Fdefault&LAYERS=lds%2Fdata%2Fnc%2Fcoads_climatology.nc%2FSST&TIME=0000-12-16T01%3A20%3A06.000Z&COLORSCALERANGE=%2C&NUMCOLORBANDS=250&ABOVEMAXCOLOR=extend&BELOWMINCOLOR=extend&BGCOLOR=extend&LOGSCALE=false&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG%3A4326&BBOX=-57.12,32.88,65.76,155.76&WIDTH=256&HEIGHT=256)

I haven't fully understood the associated godiva3 set of requests. The ones that return empty tiles seem to be OK. The error only happens for tiles with data.

Can you see the problem?

regards, Gareth

ndp-opendap commented 4 years ago

I think issue lies in the netcdf-java library used by NcWMS 4.2.2. That (older) library fails to properly URL encode the constraint expressions sent to the DAP service. Uncoded URLs that contain characters that should be encoded (like "[" and "]") have been rejected by Tomcat since roughly June 2018 (Tomcat-8.0.53). Certainly the version of Tomcat in the "latest" docker container will reject them, returning only an HTTP status of 400 and no response body.

See this issue for more: https://github.com/OPENDAP/hyrax-docker/issues/29

The NcWMS-2.2.2 bundle contains WEB-INF/lib/netcdf4-5.0.0-20160523.203326-5.jar which is definitely older than the patch for this issue.

I think I need to reach out to our friends at NcWMS and see if they have a newer version with an updated netcdf-java library.

Sincerely,

Nathan

gajowi commented 4 years ago

Thanks Nathan. It had crossed my mind that might be related. FWIW with TDS I've been working around this with relaxedQueryChars as documented in https://tomcat.apache.org/tomcat-8.5-doc/config/http.html (and more specifically described in many posts/questions/answers). I can probably moderate the issue by crafting a custom server.xml for tomcat until the upstream problem can be solved - or I keep holding off on upgrading hyrax.

If I try tweaking tomcat, I'll first try: relaxedQueryChars="[:]"

Gareth

BTW. Our data providers and users place high value on WMS (via ncWMS) and also godiva3, particularly to get a quick overview of data. I'm considering running ncWMS2 in a separate container and figuring out how to integrate that service in to hyrax. I see there is a container referenced in ncWMS2 docs: https://reading-escience-centre.gitbooks.io/ncwms-user-guide/content/02-installation.html#docker

ndp-opendap commented 4 years ago

Hi Gareth, I have been trying to avoid delivering containers with relaxedQueryChars because, well, we should fix the clients. I am wondering if the updated netcdf-java.jar file could just be injected into the docker image using a volume.

I reached out to the ncWMS folks and maybe they'll supply a new WAR file with an update netcdf-java...

I'm considering running ncWMS2 in a separate container and figuring out how to integrate that service in to hyrax.

This section of our documentation: https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#WMS_Service describes what to do, except for the docker container business.

ndp-opendap commented 4 years ago

Hi Gareth,

I heard back from Guy Griffiths - looks like we need to update the ncWMS distribution in the server. I need to get together with my team to decide how to proceed, but I suspect it means a new numbered release, as:

A new ncWMS deployment is a big change and needs configuration and integration testing.
We have a fair bit of new code in the Hyrax project so a 1.16.3 release probably makes sense at this point.

What is your level of urgency? (I suspect fairly high based on your comments about how popular this featured is)

Sincerely,

Nathan

gajowi commented 4 years ago

Thanks Nathan. Sounds good. My level of urgency is low. I'm just sticking with 1.16.1 for the moment and don't think there are any critical updates I need. It will be good to have this resolved in future releases along with general developments/improvements. Gareth

gajowi commented 4 years ago

Any news?

gajowi commented 4 years ago

bump

ndp-opendap commented 4 years ago

Thanks Gareth - We will be planning the next block of work during the next ten days and I will get this on the schedule.

gajowi commented 3 years ago

any news?

gajowi commented 3 years ago

While working on #36 I built a new container that should be equivalent to the one in dockerhub (and his issue). Something is different and I get the following stacktrace info. Possibly I should make a new issue. Ultimately I'd like to help with upstream fixes and wait. for 1.16.3 :-)

HTTP Status 500 - ncWMS configuration object is null
type Exception report

message ncWMS configuration object is null

description The server encountered an internal error that prevented it from fulfilling this request.

exception

javax.servlet.ServletException: ncWMS configuration object is null
    uk.ac.rdg.resc.edal.ncwms.NcwmsServlet.init(NcwmsServlet.java:92)
    org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:498)
    org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
    org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
    org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
    org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    java.lang.Thread.run(Thread.java:748)
note The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.

ndp-opendap commented 3 years ago

Rebuilt and pushed hyrax_ncwms:1.16.2 with updated ncWMS-2.4.2 and of course patched OS.

ndp-opendap commented 3 years ago

@gajowi Please let me know if this works for you when you have a chance.

Thanks,

N

gajowi commented 3 years ago

Will do.

On Sat, 19 Dec 2020, 6:33 am Nathan Potter, notifications@github.com wrote:

@gajowi https://github.com/gajowi Please let me know if this works for you when you have a chance.

Thanks,

N

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OPENDAP/hyrax-docker/issues/30#issuecomment-748281211, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUSKFIT2PQWYW3BHK2TH4LSVOVCJANCNFSM4OYBAOBQ .

gajowi commented 3 years ago

Hi Nathan. I have has partial success testing. Actually that is optimistic... I have a new error: javax.servlet.ServletException: ncWMS configuration object is null (when getting /ncWMS2/wms/lds/testdata/nc/coads_climatology.nc?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.3.0) Maybe I need to explicitly configure ncWMS2 with the new code. I'll get back to this later in the month. BTW I tested the new 1.16.2 build. I'd happily move straight to testing the new 1.16.3 build if that has the same changes.

steingod commented 3 years ago

I am trying to run the Docker setup provided herein for 1.16.3 (using docker-compose). Hyrax works fine, but ncWMS does not and I get messages like Could not initialize class com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl. Is this connected to this issue? I can see that 1.16.3 still runs netcdf4-5.0.0-beta6.jar?

ndp-opendap commented 3 years ago

Hi Nathan. I have has partial success testing. Actually that is optimistic... I have a new error: javax.servlet.ServletException: ncWMS configuration object is null (when getting /ncWMS2/wms/lds/testdata/nc/coads_climatology.nc?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.3.0) Maybe I need to explicitly configure ncWMS2 with the new code. I'll get back to this later in the month. BTW I tested the new 1.16.2 build. I'd happily move straight to testing the new 1.16.3 build if that has the same changes.

Use 1.16.3

ndp-opendap commented 3 years ago

I am trying to run the Docker setup provided herein for 1.16.3 (using docker-compose). Hyrax works fine, but ncWMS does not and I get messages like Could not initialize class com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl.

Something has broken that I do not understand, and unfortunately other folks have been contributing and I am afraid a change has slipped through that broke this.

Is this connected to this issue?

Well, yes and no. But insofar as ncWMS is still broken then yes.

I can see that 1.16.3 still runs netcdf4-5.0.0-beta6.jar?

No part of Hyrax utilizes that jar, it is part of the ncWMS2-2.4.2 war file so I have no control over its inclusion. The latest ncWMS (2.5.0) has moved too far into the future to be compatible with Hyrax, so for now we are stuck with 2.4.2

ndp-opendap commented 3 years ago

I will try to revisit the issue this week and see what I can learn

ndp-opendap commented 3 years ago

@gajowi @steingod I am curious to know:

Do you get the same error if you run ncWMS-2.4.2 standalone?

steingod commented 3 years ago

I haven't tried that yet, but will check (unfortunately it may take some days before I have a time slot to push on that again).

ndp-opendap commented 3 years ago

It works for me when bundled with Hyrax. I did the following:

cd hyrax-docker/hyrax-1.16.3;
docker build -t hyrax --build-arg USE_NCWMS=true hyrax
docker run  -h hyrax -p 8080:8080 --name=hyrax_container hyrax_image -n http://localhost:8080

And the server built and started correctly.

I was able to access:

http://localhost:8080/ncWMS2/Godiva3.html?server=http://localhost:8080/ncWMS2/wms/lds/data/nc/coads_climatology.nc

With one a small issue:

The background maps failed to load because there's an issue with the source. My map came up with "?" marks instead of images. I tried to open one of the image tiles in a new browser tab and got:

If that outage persists I'll reach out to Guy and see if he can help with the failed map loads.

ndp-opendap commented 3 years ago

I have also tried building and running just the ncWMS docker container:

docker build -t ncwms ncWMS --build-arg DEVELOPER_MODE=true
docker run --name ncwms -p 8080:8080 ncwms

And there I did encounter the error that @gajowi and @steingod encountered when I tried to go the to top of the service: http://localhost:8080/ncWMS2/

java.lang.NoClassDefFoundError: Could not initialize class com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl

This sure seems to be an issue with ncWMS because that docker image contains only tomcat-8 and ncWMS-2.4.2 and not Hyrax.

Interestingly, my browser remembered things and it coughed up this URL:

http://localhost:8080/ncWMS2/Godiva3.html?server=http://localhost:8080/ncWMS2/wms/lds/data/nc/coads_climatology.nc

Which fails in two ways, both expected at this point:

ncWMS is not configured with a Dynamic Service called "lds"
It didn't load the map tiles (see previous)

But, interestingly, the page manages to load and does not generate a java.lang.NoClassDefFoundError

Hey @guygriffiths, do you have any insights regarding these two problems??

The two problems:

Dereferencing http://localhost:8080/ncWMS2/ encounters java.lang.NoClassDefFoundError: Could not initialize class com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl
The background maps are failing to load because there's an issue with the source. My map came up with "?" marks instead of images. I tried to open one of the image tiles in a new browser tab and got:

steingod commented 3 years ago

@ndp-opendap thanks for helping out with this. I can confirm that i was able to make the system work when I built the bundled container. I have the same issues as you though concerning background maps. I can however now, continue testing the service configuration set up with THREDDS Catalogues which was what I intended to do. I will follow any progress here on whether the docker-compose setup will start working again. Thanks.

guygriffiths commented 3 years ago

@ndp-opendap - The background map issue is due to an issue with our hosting. The university is going through some infrastructure changes and the background maps at godiva.reading.ac.uk are not always available. Hopefully this will become more stable soon (although I just checked and it appears to be working right now). If the problem persists, you should be able to select the Demis or the GEBCO background maps which are hosted elsewhere.

The XML class not found issue I'm not sure about, but I believe may be related to Java version changes - we certainly had issues with moving towards Java 11 being the default version. ncWMS 2.5.0 fixes these issues, so if you're able to update to that it might help things. Note that v2.5.0 requires Java 11 as a minimum.

ndp-opendap commented 3 years ago

Hi @guygriffiths,

We are encountering the java.lang.NoClassDefFoundError problem on servers that deploy ncWMS-2.4.2 on CentOS-7 with Java 8:

[centos@ip-172-31-14-207 ~]$ more /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
[centos@ip-172-31-14-207 ~]$  java -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)

Or on a docker image based on:

FROM tomcat:8-jre8

Migrating these deployments to Java 11 will take us some time.

It would be great to find a work around in the meantime.

Do you have any ideas?

Thanks

Nathan

guygriffiths commented 3 years ago

I'm not sure, I'd probably try rolling back a couple of versions, maybe to 2.3.1

mylesmc123 commented 2 weeks ago

ncWMS Godiva3 viewer is not working with the current DockerHub Image: https://hub.docker.com/r/opendap/hyrax_ncwms Tag: latest which is version : 1.16.8

However, Godiva is working with version: 1.17.0-202

Tested using the following URI: http://localhost:8080/ncWMS2/Godiva3.html?server=http://localhost:8080/ncWMS2/wms/lds/data/nc/coads_climatology.nc

After loading the container via: docker run -d -h hyrax -p 8080:8080 --name=hyrax opendap/hyrax_ncwms:1.17.0-202 -n http://localhost:8080

I have not investigated the differences that may have caused this issue, but I thought it could use an update after I found myself here with a similar problem 3 years later with a workaround.

Mikejmnez commented 2 weeks ago

Thanks for the heads up @mylesmc123 ! Glad you found a workaround for this annoying issue.

On docker hub, It looks like latest was pushed manually two years ago (release of 1.16.8), whereas all newer tags are being pushed by travis ci, for example snapshot, which indeed 1.17-202.

@jgallagher59701 @ndp-opendap any thoughts on how to correct this?

OPENDAP / hyrax-docker

opendap/hyrax_ncwms:1.16.2 wms broken #30