search

OPENDAP / hyrax-docker

GNU Lesser General Public License v3.0
22 stars 12 forks source link

Provide a stable location for tomcat configuration #59

Closed MattF-NSIDC closed 1 year ago

MattF-NSIDC commented 1 year ago

I'm working on migrating an application at my org to use these official images instead of our own custom ones, and I'd like to be able to have a stable location to volume mount our own tomcat configuration. Currently I volume mount to /home/tomcat/apache-tomcat-9.0.65/server.xml, but I expect this path to change in a future upgrade, and it would require execing in to the docker image to discover the new path. What about /etc/tomcat?

Thanks all for providing these images :heart:

ndp-opendap commented 1 year ago

I think if you look at the image you'll discover that /usr/share/tomcat is a softlink pointing to /home/tomcat/..., just like in the (now old) rpm installations of Tomcat from yum. See lines 158-187 of the hyrax Dockerfile, and in particular this:

    && ln -s /home/tomcat/${TOMCAT_DISTRO} /usr/share/tomcat >&2

Please let me know if that works (And then I'll close this)

If a persistent OLFS configuration is also desired:

MattF-NSIDC commented 1 year ago

Ah brilliant, thanks, I do see that now. I used a find / -name server.xml to locate the config and didn't consider that find wouldn't traverse symlinks!

MattF-NSIDC commented 1 year ago

If a persistent OLFS configuration is also desired:

Any reason not to volume mount to /usr/share/tomcat/webapps/opendap/WEB-INF/conf/olfs.xml?

ndp-opendap commented 1 year ago

I think find will traverse soft links of you add -L to the parameter list.

MattF-NSIDC commented 1 year ago

I think find will traverse soft links of you add -L to the parameter list.

Yes, I found it's also important to ... 2>/dev/null for that one :)

ndp-opendap commented 1 year ago

Any reason not to volume mount to /usr/share/tomcat/webapps/opendap/WEB-INF/conf/olfs.xml?

It depends:

But if injecting the olfs.xml really covers your needs then you just can do that. (Although I have a concern that the volume mount will get in trouble with the symbolic links in the path)

MattF-NSIDC commented 1 year ago

Although I have a concern that the volume mount will get in trouble with the symbolic links in the path

I thought the same, but this seems to be working exactly as I needed it to :) 

    volumes:                                  
      # Config                       
      - ./olfs/tomcat/server.xml:/usr/share/tomcat/server.xml:ro
      - ./olfs/tomcat/olfs.xml:/usr/share/tomcat/webapps/opendap/WEB-INF/conf/olfs.xml:ro

Thanks Nathan!

ndp-opendap commented 1 year ago

YW!

MattF-NSIDC commented 1 year ago

I found logs were not present at the WEB-INF location, but was able to persist them from /etc/olfs/logs! Thanks again :) 

[root@5c7815569f0a /]# ls -la /etc/olfs/logs/
total 16
drwxrwxr-x 2 tomcat tomcat 4096 Sep  6 20:44 .
drwxr-xr-x 6 root   root   4096 Sep  6 20:44 ..
-rw-r----- 1 root   root      0 Sep  6 20:44 AnonymousAccess.log
-rw-r----- 1 root   root      0 Sep  6 20:44 BESCommands.log
-rw-r----- 1 root   root    195 Sep  6 20:46 HyraxAccess.log
-rw-r----- 1 root   root   3753 Sep  6 20:46 HyraxErrors.log
[root@5c7815569f0a /]# ls -la /usr/share/tomcat/webapps/opendap/WEB-INF/conf/logs
total 8
drwx------ 1 tomcat tomcat 4096 Jul 22  2022 .
drwxr-xr-x 1 root   root   4096 Sep  6 20:44 ..
ndp-opendap commented 1 year ago

[root@5c7815569f0a /]# ls -la /usr/share/tomcat/webapps/opendap/WEB-INF/conf/logs

That's very odd.

When I do the same on one of our NASA deployments (which is deployed as the ROOT/default webapp):

[root@3cd2ffe31246 /]# ls -la  /usr/share/tomcat/webapps/ROOT/WEB-INF/conf/logs
total 4308
drwx------ 1 tomcat tomcat     232 Sep  7 00:02 .
drwxr-xr-x 1 root   root        76 Sep  6 17:52 ..
-rw-r----- 1 root   root      3455 Sep  7 00:02 BESCommands.log
-rw-r----- 1 root   root     70506 Sep  6 23:52 EDLTimer.2023-09-06.log
-rw-r----- 1 root   root       192 Sep  7 00:02 EDLTimer.log
-rw-r----- 1 root   root   1893042 Sep  6 23:59 HyraxAccess.2023-09-06.log
-rw-r----- 1 root   root     23929 Sep  7 00:05 HyraxAccess.log
-rw-r----- 1 root   root   2384811 Sep  6 23:52 HyraxBESCommands.2023-09-06.log
-rw-r----- 1 root   root     18631 Sep  6 19:02 HyraxErrors.log
-rw-r----- 1 root   root         0 Sep  6 17:52 auth.log
-rw-r----- 1 root   root         0 Sep  6 17:52 debug.log
ndp-opendap commented 1 year ago

Also @MattF-NSIDC - We should look at your configuration - There is a combined log feature that when enabled will cause the OLFS to tell the BES most of it's log info and then the BES will include that in its log entry for the request. Then you can hoover up that stuff by:

decker run ... -v /my/local/bes.log:/var/log/bes/bes.log ...

Whatever works for you...

MattF-NSIDC commented 1 year ago

Ooh, I really like that! This deployment actually has two BES backends each configured a bit differently (this is supporting an AppEEARS feature), would that still work? I.e. the OLFS logs for a request to BES A would correctly end up in the logs for BES A?

There's one problem I noticed with the BES logs though. When running the olfs and besd containers separately as we do (I'm not sure if the hyrax container is impacted), all the log files except bes.log are written out as root so letting the mount directories get autocreated (they'll always be owned by root) works fine. bes.log is written out as user 998 (IIRC), and so if I want to persist the logs on the host, I need to pre-create the mounted directory and set its permissions o+rw, or set its ownership to match the user that writes the log file.

MattF-NSIDC commented 1 year ago

That's very odd.

When I do the same on one of our NASA deployments (which is deployed as the ROOT/default webapp):

Perhaps this is because we're using opendap/olfs:1.18.13 (the original release), and not any of the newer snapshot images. As far as I could tell, there was no tag for the latest stable patch release. Is it possible for that tag to be auto-updated to always point to the latest stable patch release?

ndp-opendap commented 1 year ago

The tag latest has a special meaning. It's not the tag, it's OPeNDAP that's at fault. If we were able to make more frequent official releases that would resolve this.

But I have an idea. In our NGAP wiki - which I think you have access - we keep a page that details the running changes in the images that get deployed to our PROD (production) environment. So these images are vetted, all tests pass, security checks passed, etc. (or were vetted, older ones will have issues fixed in later versions)

Would that be good enough?

ndp-opendap commented 1 year ago

Ooh, I really like that! This deployment actually has two BES backends each configured a bit differently (this is supporting an AppEEARS feature), would that still work? I.e. the OLFS logs for a request to BES A would correctly end up in the logs for BES A?

I have not tested this but I think that is the case based on how the software does the things.

There's one problem I noticed with the BES logs though. When running the olfs and besd containers separately as we do (I'm not sure if the hyrax container is impacted), all the log files except bes.log are written out as root so letting the mount directories get autocreated (they'll always be owned by root) works fine. bes.log is written out as user 998 (IIRC), and so if I want to persist the logs on the host, I need to pre-create the mounted directory and set its permissions o+rw, or set its ownership to match the user that writes the log file.

Yeah the besd and beslistener processes should be running as the user bes, and your solution is pretty much what I worked up. I think if you execute a docker run on the image but do not use the -d switch you should see in the output (from entrypoint.sh) a line reporting the bes user and group :

bes_uid=$(id -u bes)
bes_gid=$(id -g bes)
echo "Launching besd [uid: ${bes_uid} gid: ${bes_gid}]" >&2
MattF-NSIDC commented 1 year ago

The tag latest has a special meaning. It's not the tag, it's OPeNDAP that's at fault. If we were able to make more frequent official releases that would resolve this.

Sorry, didn't meant to suggest we want to use latest; I prefer a tag like X.Y which is updated with the latest patch version, so may actually refer to X.Y.9, not X.Y.0. For our Hyrax images, we'd like to be able to specify opendap/olfs:1.18.13 and get the latest stable builds instead of having to pin to an exact build number.

But I have an idea. In our NGAP wiki - which I think you have access - we keep a page that details the running changes in the images that get deployed to our PROD (production) environment. So these images are vetted, all tests pass, security checks passed, etc. (or were vetted, older ones will have issues fixed in later versions)

Oh cool, thanks! I can access this. Since we deploy the olfs and bes images instead of the hyrax image, how can we figure out, for example, what olfs and bes build goes in to opendap/hyrax:ngap-1.16.8-292?

MattF-NSIDC commented 1 year ago

I have not tested this but I think that is the case based on how the software does the things.

We may not have time to test this out in the near-term, working on very limited maintenance funding at the moment :) But we'll try and get back to you on what we find when we do!

ndp-opendap commented 1 year ago

Oh cool, thanks! I can access this. Since we deploy the olfs and bes images instead of the hyrax image, how can we figure out, for example, what olfs and bes build goes in to opendap/hyrax:ngap-1.16.8-292?

That should have been on each of those deployment diffs pages, my bad. It's fixed now: https://wiki.earthdata.nasa.gov/pages/viewpage.action?pageId=316089101

MattF-NSIDC commented 1 year ago

Amazing, thanks so much!

ndp-opendap commented 1 year ago

Oh @MattF-NSIDC , one last thing:

An excellent source of truth for each builds inventory can be found in the git commit log for the hyrax-docker project

Clicking the ellipsis by the commit title will show the build inventory for the build associated with the commit.

MattF-NSIDC commented 1 year ago

Ah, very cool, thanks again Nathan!