remkop commented 4 years ago

Currently, ami --version does not show any useful output. There is a need for better output to facilitate support and troubleshooting.

From the OpenVirus Slack:

Peter Murray-Rust 6:05 PM @Remko Popma I need to make ami --version give a meaningful response. I don't use major/minor releases, though perhaps we should have a release strategy. Are there any tools to help automate this. (Currently I refer to commits by their git-hash and jar files by their date).

Remko Popma 6:12 PM We can include version information as part of the build. But the version number/string needs to come from somewhere (perhaps a version.properties file next to pom.xml ) and that is usually manually maintained - meaning the version number is bumped up when some meaningful increment in functionality was added. (edited) 6:13 In our case, since our users build from source, we may need to think about some variation of that... We don't want to automatically change this version with every build, because then ami users would see a different version with each rebuild... (edited) 6:15 One simple solution for now is to modify AMI.java and have @Command(name = "ami", version = "SOME NUMBER/STRING" - and update this as we add functionality. (NUMBER/STRING could be a date/timestamp). We can migrate to a version provider that reads from a version.properties file later. (edited)

Peter Murray-Rust 6:46 PM Thanks @Remko Popma. The immediate reason was so that we could ask users what version they were running. So we can diagnose the --validate problems as only occurring before a certain version.

remkop commented 4 years ago

I have pushed this change to AMI.java:

@Command(name = "ami", version = AMI.VERSION,
...
public class AMI implements Runnable {
...
    public static final String VERSION = "2020-07-09"; // update this with each change

Please update this VERSION string with each meaningful change; if multiple changes per day, consider adding a time. This is a simple strategy to get us going. We can refine as we go.

remkop commented 4 years ago

We can enhance this in several ways:

add version strings of other components/libraries used by ami (pdfbox, lucene, picocli), as well as printing version info of the Java runtime and operating system; may be useful when troubleshooting. The "problem" is which components to include (would including the full classpath (minus shared prefixes) be excessive? I count 267 entries - works out to 2 or 3 screens full with just the comma-separated jar names, or 8 screens with newline-separated jar names. Full class path maybe too much.)
externalize the ami version string into a separate file, maybe version.properties, that may be easier to update then the AMI.java source file. Note that increasing the version is a manual action by design. We don't want to change the version with every build, since our users build from source.
we could "ease the pain" by having a script/program that we would invoke to update the version info. This could be a Java program that prints the current UTC date/time in iso-8601 format to the version.properties file, for example.

petermr commented 4 years ago

Thanks. I'll try to make this work. Happy to give version.properties a try. I suppose it could be altered from outside (including a Github edit). If anything changes the interface (e.g. adding --validate I'll definitely upgrade to version. If it fixes a bug we'll upgrade. At the moment, I generally only push once a day and not all those would merit an upgrade.

remkop commented 4 years ago

The properties-maven-plugin may be useful (see also usage).

If it does what I think it does, this will allow us to have a version.properties file with

# our project version
ami.version=2020-07-09T17:15:00+0900

# dependencies versions
# ...
log4j.version=2.13.3
pdfbox2.version=2.0.19

Then, in our POM, we can reference these properties to define:

our AMI version:

    <groupId>org.contentmine</groupId>
    <artifactId>ami3</artifactId>
    <!-- to sync with new cproject-norma-ami versions -->
    <version>0.1-SNAPSHOT</version>

becomes

    <groupId>org.contentmine</groupId>
    <artifactId>ami3</artifactId>
    <version>${ami.version}</version>

versions of all dependencies used in the pom.xml. I think we will be able to remove our dependency versions like <log4j.version>2.13.3</log4j.version>, <pdfbox2.version>2.0.19</pdfbox2.version> etc from the <properties> section of our pom. Dependency versions will then live in the version.properties file going forward.

Once this is done, we need to ensure that the version.properties file is included in the ami.jar, so we can read it at runtime. We can then create a version provider that reads this properties file:

@Command(name = "ami", versionProvider = PropertiesVersionProvider.class)
class AMI { ... }

and

    /**
     * {@link IVersionProvider} implementation that returns version information from a {@code /version.properties} file in the classpath.
     */
    static class PropertiesVersionProvider implements IVersionProvider {
        public String[] getVersion() throws Exception {
            URL url = getClass().getResource("/version.properties");
            if (url == null) {
                return new String[] {"No version.properties file found in the classpath."};
            }
            Properties properties = new Properties();
            properties.load(url.openStream());
            TreeMap<String, String> map = new TreeMap<String, String>((Map) properties);
            List<String> list = map.entrySet().stream()
        .map(e -> e.getKey() + "=" + e.getValue()).collect(Collectors.toList()); 
            return list.toArray(new String[0]);
        }
    }

remkop commented 4 years ago

Update: it turns out that the properties-maven-plugin is not useful. Maven won't let us externalize the versions of dependencies. (Various StackOverflow questions: 1, 2, and 3.)

We can still have a version.properties, but it will likely only contain the ami version. This version is still duplicated in the pom.xml, which is not great. Still thinking...

petermr commented 4 years ago

Thanks.

On Tue, Jul 14, 2020 at 12:07 PM Remko Popma notifications@github.com wrote:

Update: it turns out that the properties-maven-plugin is not useful. Maven won't let us externalize the versions of dependencies. (Various StackOverflow questions: 1 https://stackoverflow.com/questions/56626789/reading-dependency-versions-in-pom-xml-from-external-properties-file, 2 https://stackoverflow.com/questions/14725197/reading-properties-file-from-pom-file-in-maven/14727072#14727072, and 3 https://stackoverflow.com/questions/40114869/maven-pom-file-externalizing-dependency-versions-to-a-properties-file .)

We can still have a version.properties, but it will likely only contain the ami version. This version is still duplicated in the pom.xml, which is not great. Still thinking...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/petermr/ami3/issues/51#issuecomment-658118878, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFTCS3LDMYM2YRCDLWG4OTR3Q37RANCNFSM4OVA3KBA .

-- Peter Murray-Rust Founder ContentMine.org and Reader Emeritus in Molecular Informatics Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK

remkop commented 4 years ago

There are two other maven plugins related to version management:

TL;DR - Neither meets our needs exactly. Still may be useful.

versions-maven-plugin

Updates project version in the pom.xml, with commands like mvn versions:set -DnewVersion=1.0.1-SNAPSHOT and mvn versions:commit . Seems mostly useful for multi-module projects to avoid manually updating versions in many pom.xml files.

May be useful if it allows version information being externalized in a separate properties file, but this does not seem to be the case. (Need to double check to make sure.)

Maven Release Plugin

This does a lot. Bumps version numbers, creates an SCM tag, and invokes mvn deploy to publish the release. If this publishing bit meets our needs this may be a good choice.

There is a way to skip running the tests during the release process, which we need, otherwise this plugin would not be usable for us.

The batch mode is interesting, because it allows the release version number and some other properties to be defined in a release.properties file. Unfortunately, this file needs to exists in the same location as the pom.xml, so it is not automatically included in the ami.jar (which we would like so we can read it at runtime when ami --version is invoked).

Still need to look at the publishing bit.

remkop commented 4 years ago

Axel Fontaine wrote a series of articles about how inefficient and fragile the Maven Release Plugin is, requiring 3 full builds with multiple checkouts to do a release. He proposes to replace this with maven-resources-plugin, maven-install-plugin and maven-deploy-plugin configuration, plus maven-scm-plugin. This allows him to

publish new snapshots using mvn clean deploy
release and tag new versions using mvn clean deploy scm:tag -DVERSION_NUMBER=1.2.3

Additionally, he wrote a follow-up article "the Final Nail" in 2013 that replaces the maven-resources-plugin, maven-install-plugin and maven-deploy-plugin configuration with versions-maven-plugin. This is much simpler. It does require running Maven twice: once for updating the version and the again with the updated version.

The final installment Maven Release Plugin: Dead and Buried, describes how to go down to just running maven once, without the versions-maven-plugin.

petermr commented 4 years ago

This sounds good. Can you go ahead and give a recipe?

On Mon, 20 Jul 2020, 01:43 Remko Popma, notifications@github.com wrote:

Related:

Axel Fontaine wrote a series of articles about how inefficient and fragile the Maven Release Plugin is, requiring 3 full builds with multiple checkouts to do a release. He proposes to replace this with maven-resources-plugin, maven-install-plugin and maven-deploy-plugin configuration, plus maven-scm-plugin. This allows him to

publish new snapshots using mvn clean deploy

release and tag new versions using mvn clean deploy scm:tag -DVERSION_NUMBER=1.2.3

https://axelfontaine.com/blog/maven-releases-steroids.html

https://axelfontaine.com/blog/maven-releases-steroids-2.html

https://axelfontaine.com/blog/maven-releases-steroids-3.html

Additionally, he wrote a follow-up article "the Final Nail" https://axelfontaine.com/blog/final-nail.html in 2013 that replaces the maven-resources-plugin, maven-install-plugin and maven-deploy-plugin configuration with versions-maven-plugin. This is much simpler. It does require running Maven twice: once for updating the version and the again with the updated version.

The final installment Maven Release Plugin: Dead and Buried https://axelfontaine.com/blog/dead-burried.html, describes how to go down to just running maven once, without the versions-maven-plugin.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/petermr/ami3/issues/51#issuecomment-660738726, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFTCS3ELEKOG6NUYAAQJPTR4OHJVANCNFSM4OVA3KBA .

remkop commented 4 years ago

Yes, please consider all these comments “notes to self” more than instructions.

This is very much work in progress.

One I have a process that works and gives us versions as well as a place to host the artifacts created by a release (so our users can just download the binaries instead of building from source), I will document the steps for versioning and publishing a release (our role) and the steps for downloading the latest binary release (our user’s role).

petermr commented 4 years ago

That's excellent. I suspect that one or two reports of errors were because the correct version of the software was not used after it had been patched. Giving a version number would have avoided some of this. The current user community is now well versed in git. At present we use a single master branch and this works well with our current discipline (we have only had one broken build). This is partly because we build different components on a single-dev basis. So I think we should continue for the next 1-2 months, after which activity may ramp down a bit.

On Mon, Jul 20, 2020 at 9:34 AM Remko Popma notifications@github.com wrote:

Yes, please consider all these comments “notes to self” more than instructions.

This is very much work in progress.

One I have a process that works and gives us versions as well as a place to host the artifacts created by a release (so our users can just download the binaries instead of building from source), I will document the steps for versioning and publishing a release (our role) and the steps for downloading the latest binary release (our user’s role).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/petermr/ami3/issues/51#issuecomment-660886575, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFTCS4F6GJKL6FAN6DHXETR4P6QTANCNFSM4OVA3KBA .

-- Peter Murray-Rust Founder ContentMine.org and Reader Emeritus in Molecular Informatics Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK

remkop commented 4 years ago

Closing, the work on this ticket means we now have a versioning strategy, as well as a packaging (#17) strategy and a publishing (#24) strategy.

I believe all that remains is documentation. I created a separate ticket https://github.com/petermr/ami3/issues/53 for that.

petermr / ami3

Manage ami version #51

versions-maven-plugin

Maven Release Plugin