WMS Dataset Links Lead to 404 Error

[julienchastang]

I have a feeling this has something to do with the fairly restrictive permission settings in the parent Tomcat container. This may explain the java.lang.IllegalStateException: No output folder we are seeing in the logs. Will create a testing environment that will gradually lesson the Tomcat container permission settings and see where that leads me.

@lesserwhirls @cwardgar Does the TDS WMS service write temporary files or need write access somewhere on the file system? Thanks.

[lesserwhirls]

What does the full message look like?

[rsignell-usgs]

 418  java.io.FileNotFoundException: /usr/local/tomcat/work/Catalina/localhost/thredds/org/apache/jsp/tag/web/wms/Folder_tag.java (Permission denied)

localhost.2017-11-29.log

[lesserwhirls]

Interesting. Does the container not have permission to write to Tomcat's work directory?

[rsignell-usgs]

root@c12cb20c84db:/usr/local/tomcat# ls -sailrt
total 144
     96  4 drwxr-x---  2 tomcat tomcat  4096 May  5  2017 work
     99 16 -rw-r-----  1 tomcat tomcat 15946 May  5  2017 RUNNING.txt
     93  8 -rw-r-----  1 tomcat tomcat  7064 May  5  2017 RELEASE-NOTES
     98  4 -rw-r-----  1 tomcat tomcat  1723 May  5  2017 NOTICE
     94 56 -rw-r-----  1 tomcat tomcat 57092 May  5  2017 LICENSE
     29  4 drwxr-sr-x  4 tomcat tomcat  4096 Jul  3 05:50 content
     90  4 drwxr-sr-x  4 tomcat tomcat  4096 Nov 29 16:00 javaUtilPrefs
     83  4 drwxr-x---  2 tomcat tomcat  4096 Nov 29 16:00 bin
     12  4 drwxrwsr-x 48 root   staff   4096 Nov 29 16:00 ..
     91  4 drwxr-x--- 10 tomcat tomcat  4096 Nov 29 16:00 webapps
     27  4 drwx--S---  2 tomcat tomcat  4096 Nov 29 16:00 conf
     95  4 drwxr-x---  2 tomcat tomcat  4096 Nov 29 16:00 temp
     92  4 drwxr-x---  4 tomcat tomcat  4096 Nov 29 16:00 lib
     97  4 drwxr-sr-x  4 tomcat tomcat  4096 Nov 29 16:00 include
     13  4 drwxr-sr-x 40 tomcat tomcat  4096 Nov 29 16:00 .
     40  4 drwxr-sr-x  4 tomcat tomcat  4096 Nov 29 16:00 native-jni-lib
1544895 12 drwxr-xr-x  2 tomcat tomcat 12288 Dec  1 01:18 logs

[lesserwhirls]

It's not so much that the TDS or ncwms would be explicitly writing temporary files somewhere, but Tomcat (like compiled JSP files).

[lesserwhirls]

What does it look like under the work directory?

[lesserwhirls]

Or, even better, the permissions for /usr/local/tomcat/work/Catalina/localhost/thredds/org/apache/jsp/tag/web/wms/Folder_tag.java

[rsignell-usgs]

Note the date of the work directory: May 5! Nothing in there:

root@c12cb20c84db:/usr/local/tomcat/work# ls -sailrt
total 8
96 4 drwxr-x---  2 tomcat tomcat 4096 May  5  2017 .
13 4 drwxr-sr-x 40 tomcat tomcat 4096 Nov 29 16:00 ..

[lesserwhirls]

What user is running tomcat? I would assume user tomcat, but this makes me wonder. This directory shouldn't be empty.

[lesserwhirls]

Well..shouldn't be empty for a TDS that's been running for a bit. A 404 should cause Tomcat to write out something to that directory (for example, trying to hit /thredds/catalogNO.html)

[lesserwhirls]

Or, hitting serverInfo.html.

[rsignell-usgs]

Nothing is being written to that directory. I've been hitting catalogs and it stays empty. Email me your public key and you can play around in here. Seems silly me sending you snippets back and forth.

root@c12cb20c84db:/usr/local/tomcat/work# ps aux | grep tomcat
tomcat       1  0.1  9.9 9016952 1632628 ?     Ssl  Nov29   3:24 /docker-java-home/jre/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dtds.content.root.path=/usr/local/tomcat/content/ -Djava.util.prefs.systemRoot=/usr/local/tomcat/javaUtilPrefs -Djava.util.prefs.userRoot=/usr/local/tomcat/javaUtilPrefs -server -d64 -Xms4G -Xmx4G -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Djna.tmpdir=/tmp/ -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start

[rsignell-usgs]

You mean like: http://jetstream.signell.us:8080/thredds/serverInfo.html

Doesn't work.

[lesserwhirls]

Yep http://thredds-jetstream.unidata.ucar.edu/thredds/serverInfo.html 404s...not just WMS, but anything using JSP.

[lesserwhirls]

It looks like the processing running tomcat does not have permission to write to the work directory. No idea why, but that would explain why ncwms and the serverInfo pages are not working. The use JSP and Tomcat needs to be able to write to that directory.

[lesserwhirls]

As a test, you could bounce the TDS, chmod 777 the work directory before making any TDS requests, and try to hit the serverInfo page.

[lesserwhirls]

@julienchastang - maybe this again?

[julienchastang]

Hey Sean. Thanks for all your hard work here. This is a problem on the Docker side and not a TDS issue as I think we are narrowing in on. I had a similar issue with #164. Yes the permissions in the container are restrictive and Tomcat does not get permission to write wherever it wants per Tomcat security recommendations. See the parent Tomcat container for details. However, this can be modified or perhaps the WMS service provides a configuration option in the XML to change where it needs to write to. I now have something concrete to go on.

[lesserwhirls]

The work/ directory acts like a scratch directory for Tomcat - the TDS and ncWMS are not setting that location. Since it's a Tomcat thing, I'm surprised the recommendations do not explicitly state that tomcat should be able to write to it.

If you want to change that location, you can edit the $TOMCAT_HOME/conf/server.xml file for Tomcat and edit the <Host> element to include the attribute workDir, which defaults to $TOMCAT_HOME/work. See here for more info.

[lesserwhirls]

I'm guessing it would go here: https://github.com/Unidata/tomcat-docker/blob/master/server.xml#L123

[julienchastang]

Ahh excellent. OK that's useful info I did not know before (or forgot). I'll research best practices for the work directory with respect to permissions, location, etc. Thanks again.

[julienchastang]

Ugh! I did not have time to scrutinize this in more detail until now. I incorrectly assumed this had something do with overly restrictive permissions in the tomcat container. But no! And no, it is not similar #164. Sean is right!

root@507fb1bf2906:/usr/local/tomcat/work# ls -arlt
total 8
drwxr-x---  2 tomcat tomcat 4096 May  5  2017 .
drwxr-sr-x 40 tomcat tomcat 4096 Nov 30 23:25 ..
root@507fb1bf2906:/usr/local/tomcat/work# gosu tomcat touch foo
touch: cannot touch ‘foo’: Permission denied

See here for more details.

[julienchastang]

Yep. You need to switch from aufs to devicemapper. See #171.

Add

"storage-driver": "devicemapper"

to

/etc/docker/key.json

See Switching Docker from aufs to devicemapper.

WMS now works here.

[rsignell-usgs]

So thredds docker on xsede now has the WMS services working. Hurray!

[rsignell-usgs]

Uh, but I'm having trouble getting this going on my own machine. I probably didn't understand what I'm supposed to do.

On the host machine, I did:

sudo vi /etc/docker/key.json
ubuntu@tg-atm160027-rsignell:/opt/docker$ sudo more /etc/docker/key.json
{"crv":"P-256","d":"0rQrXYElfDSCcqYxiIA2CmD3q3ikhRfwgd0qFxHXTVQ","kid":"NSGR:S2J
V:C6PJ:UFR4:JNVW:QDLJ:A4MJ:7AB3:BE4I:IS2V:Q6IQ:ORNH","kty":"EC","x":"8ZUlN5h7CW2
d912OyzfqFflNvIIUF1KogsYqeOyQU1g","y":"2AogglFXjtE2xyqP50Eh3mnu6IJwjZiDwLTi4WHvd
Oo","storage-driver":"devicemapper"}
docker-compose down
docker-compose up -d

But still getting same "error getting data from server" WMS error on: http://jetstream.signell.us:8080/thredds/godiva2/godiva2.html?server=http://jetstream.signell.us:8080/thredds/wms/local/Sandy_ocean_his.nc

[rsignell-usgs]

@julienchastang , can you please provide a recipe for what I'm supposed to do to fix this problem? I received your offline question:

Did you try the docker save and load part? That should be documented somewhere in what I described.

but I don't know what that means.

[julienchastang]

I would:

Stop the container via docker or docker-compose.

docker save unidata/thredds-docker:latest > /tmp/thredds.tar.gz

With sudo, append

"storage-driver": "devicemapper"

to

/etc/docker/key.json

finally

docker load < /tmp/thredds.tar.gz

And start the container via docker or docker-compose.

Let me know if that works.

[rsignell-usgs]

No, that didn't work. Would you be willing to take a look?

ssh ubuntu@149.165.157.40`
cd /opt/docker
docker-compose down 

[rsignell-usgs]

@julienchastang , I don't know what you did, but it worked!

[julienchastang]

In short, beyond doing what I suggested earlier, I updated your VM according with this script.