Feedback and questions on Docker setup for Metacat

[amoeba]

Hey @mbjones I needed to set up a fresh Metacat instance so I ran through the Docker setup you've pushed to https://github.com/NCEAS/metacat/compare/feature-docker. It looks great and I appreciated the readme. I got to a running Metacat stack in pretty short order. I had some issues and some questions though:

General stuff

• What do you think about also providing MetacatUI as part of the default stack?

metacat.env

This is really clean which is nice. Though I noticed that the Postgres password and Metacat admin password are pre-filled which makes me worried users won't edit this file and there will be Metacat stacks running default passwords. What do you think about not providing these and having the build/startup fail when they aren't set? Also, maybe change the EMAIL and ADMIN defaults to not include jones and use something generic like admin or metacat.

Building

I've always had trouble building Metacat and I'm not sure quite why. AFAICT it looks like it might be a Java version thing but the errors are unclear. The things I run into when I run ant distbin are:

• build.properties hardcodes the maven.home property so I always have to change this to something like maven.home=/usr/local/Cellar/maven/3.5.2/libexec. Could this require less configuration?

• I get an eventual build failure:

  BUILD FAILED /Users/bryce/src/metacat/build.xml:786: The following error occurred while executing this line: /Users/bryce/src/metacat/build/seek/projects/ecogrid/buildfiles/metacatImpl.xml:87: The following error occurred while executing this line: /Users/bryce/src/metacat/build/seek/projects/ecogrid/buildfiles/metacatAuth.xml:71: Compile failed; see the compiler error output for details.

any insights?

Using it

I got the stack up and running with docker-compose without issue so I went to submit a Data Package to it using R. It wasn't clear to me which token signing environment my Metacat was set up with so I assumed test. After some debugging, I figured out I was in the PROD signing environment, but that R thinks it's in the test environment:

> mn@env
[1] "test"

which means, in R, that I have to set a production token as my test token to start hitting authenticated endpoints. I'm not sure what R is using to determine this nor was I sure what I might change to change it. Which token signing environment does it make the most sense to start Metacat off in, production or some test environment? I see your MODE env var so it seems like that would be a logical place to control this all from.

[mbjones]

Awesome. Thanks!

0. Glad it worked for you. I agree more thought needs to happen on how to set up the default environment vars, especially passwords.
1. MetacatUI:
   • I've been debating this. I have it running in my docker setup, using the default MetacatUI that is bundled with Metacat. You can access it at https://$HOST/metacatui. Given that some people may not want to run it, I've considered removing it from the example compose file. Maybe add it to the MetacatUI repository? Discussion on this would be hepful.
2. Build failures.
   • I get similarly annoyed by the maven config. I have no doubt that could be streamlined. At a minimum, mvn -v produces a "Maven home" directory, so we should be able to find it out programmatically.
   • Your other build failures look to be related to the ECOGRID download. Have you tried a build from a completely fresh checkout? The build takes literally forever, and it actually rebuilds everything not twice but three whole times, but it eventually works for me.
3. R package
   • The MODE environment variable in metacat.env is for Let's Encrypt, and not for us. That will clearly require better documentation. Right now I am not configuring the DataONE properties at all, so they are left at defaults. So, my guess is that it wants production certs because metacat.properties is pointing to cn.dataone.org by default. The R client probably looks in the CN's node service to see if a node is registered in production, and because we are not, it wants a test cert. I think all of this stuff will need to be tweaked to support easy switching from test to production status. I've never liked the R code using different env vars for test and production environments.

[amoeba]

Thanks. Look like I missed that MetacatUI gets set up by default: That's nice. Re: my build failures, yes this is a fresh checkout. I have everything I can think of installed via homebrew if that helps diagnose the build failures.

[gothub]

The R client calls CN listNodes to determine if a member node is in the production environment and if it is not known to the CN then the node is designated as belonging to a dev env. The DataONE MN API doesn't provide a call for discovery of the D1 env it is in, AFAIK. If there are any ideas on changing or improving this then now seems like a good time, to help streamline the docker metacat setup.

The motivation to use the R options dataone_token and dataone_test_token was so that users could easily switch between test and prod environments, as discussed here. If this has caused and problems for users then we should revisit this.

[amoeba]

The R client calls CN listNodes to determine if a member node is in the production environment and if it is not known to the CN then the node is designated as belonging to a dev env.

Ah, that makes sense, and is reasonable. Thanks!

[csjx]

I understand the reasoning for setting D1Client.CN_URL to the production CN, but I do wonder if we set it to https://cn-stage-2.test.dataone.org/cn, because that's the "stable" DataONE test environment, and to be registered in the production DataONE, you still have to go through a testing phase of some sort. Switching that property is inevitable at some point for all Metacat operators, and you still get a valid token from STAGE2.

[mbjones]

@csjx I think this discussion of how the R client works with CNs and auth tokens is a little off topic for this ticket on the metacat docker deploy. Do we want to move that to a R client ticket, or see if one already exists, or just stick with what we have which works for most people using production machines (only devs tend to use non-registered machines a lot)?

[amoeba]

On-topic or off, I think it'd be great if this Docker deployment allowed me to specify which environment my Metacat stack would point at, rather than just default to the production CN which is almost never what someone wants. An option for an additional Issue might be to change this behavior at the Metacat level but, since there are multiple DataONE testing environments available, allowing this to be configured for the Docker deployment would be handy.

[mbjones]

@amoeba I totally agree on being able to choose an environment for the docker deploy. At this point I hadn't started on DataONE configuration in docker at all. There are several other pieces of info we will need to get dataone connectivity set up, not the least of which is a cert. so I bypassed that config for now.

One thing you might not have noticed is that you can set any metacat.properties key and value in the docker folder app.properties file and it will override the defaults. So you could set the cn property there.

[artntek]

Closing this, since it is now superseded by the work done for the Epic #1623