Deliver ROBOT plugins in ODK - whats the best way?

[matentzn]

We need to decide on a way to make ROBOT plugins easily configurable with ODK.

There are basically two completely different approaches (at least)

1. We pass in a plugins dir to run.sh (as a volume) and require all custom plugins to be present
2. We add custom plugins into the build process. Basically, we depend on them the same way we depend on data artefacts, download them into a local plugin directory in the project, and point ROBOT_PLUGINS_DIRECTORY at that. For example:

Makefile contents:

export ROBOT_PLUGINS_DIRECTORY=$(PWD)/tmp/plugins

$(ROBOT_PLUGINS_DIRECTORY)/monarch.jar:
    mkdir -p $(ROBOT_PLUGINS_DIRECTORY)
    wget ... -O $@

hp-eq.owl: | $(ROBOT_PLUGINS_DIRECTORY)/monarch.jar
    $(ROBOT) monarch:upheno-augment -i hp-edit.owl --relation has_phenotype_affecting  -o $@

While I appreciate the cleanliness of approach 1, it puts too much of a burden on most of the people I work with - I would just like to fold it into the general make process somehow, but I am open to add both alternatives.

Of course you could sort of do a hybrid: pass in the ~/.robot/plugins dir as a volume and download dependencies as necessary. But I would like to keep dependencies on local file structures as minimal as possible.

[gouttegd]

A related question is: Should the ODK come with some bundled “default” plugins directly within the image, with ROBOT pre-configured to use them?

I don’t have a strong opinion on that question (I’ll just note that if we think a plugin is useful enough to be provided by the ODK, that’s probably a good indication that such a plugin should really be a ROBOT built-in command instead), but if do want to do that, it will have an impact on the issue you raise, because ROBOT will need to be configured to look for plugins in (at least) two places: one location within the image for bundled plugins, and one location outside of the image (e.g. $(TMPDIR)/tmp/plugins) for custom plugins provided by the user.

[gouttegd]

I think we should support both the possibility of downloading the plugins as a pre-requisite step of a custom workflow (as shown in the ticket), and the possibility of the plugins being committed directly in the repository.

For that, I envision reserving two distinct locations in a ODK repository:

• plugins (at the top level), intended for plugins one wants to commit to the repository (so that they would always be there);
• src/ontology/tmp/plugins, intended for plugins that are downloaded as part of a custom Makefile step.

Early in the workflow, we check for the presence of plugins in the top-level directory, and if there are any, we copy (or symlink) them to the src/ontology/tmp/plugins directory. Then we can set the ROBOT_PLUGINS_DIRECTORY variable to that directory only.

[gouttegd]

That is, in the standard Makefile we would have the following additions:

# Make sure ROBOT knows where to find plugins
export ROBOT_PLUGINS_DIRECTORY=$(TMPDIR)/plugins

# List of "repository plugins" (plugins permanently installed in top-level plugins directory, entirely managed by users)
ROBOT_REPOSITORY_PLUGINS=$(wildcard ../../plugins/*.jar)

# Standard rule to create symlinks in $(TMPDIR)/plugins, so that the repository plugins
# are visible to ROBOT
repository_plugins:
        for plugin in $(ROBOT_REPOSITORY_PLUGINS) ; do ln $plugin $(TMPDIR)/plugins ; done

Then in the custom Makefile, the repository’s workflow engineers would have to either:

• if they prefer to download whatever plugins they need as part of their workflows, create custom rules to download those plugins and put them in $(TMPDIR)/plugins (as is done in the example in the ticket), then make sure that any custom rule that is needing those plugins depend on the plugin-downloading rule;
• if instead they have already put the plugins “manually” into the top-level plugins directory, then they just need to make sure that any rule that is needing those plugins depend on the standard repository_plugins rule.

[gouttegd]

(The following is a much longer-term putative new feature, may be ignored for now – and maybe forever…)

We could also have a src/plugins directory intended to contain plugins in source form (i.e. Java source files rather than JAR archives).

This would be intended only for small, simple plugins (that can fit entirely within a single Java file) that are highly specific to one repository only.

Then we would have a standard rule (compile_local_plugins or something like that) that would detect the presence of Java files in src/plugins and, if there are some, automatically compile them and place the resulting JAR archive into the “effective” plugins directory ($(TMPDIR)/plugins).

(The odkfull image comes with a full JDK and Maven, so we should have everything we need to compile simple plugins.)

[matentzn]

Lets go exactly with your suggestion above:

https://github.com/INCATools/ontology-development-kit/issues/909#issuecomment-1668233368

I will work on a test for this.

[matentzn]

@gouttegd testing this a bit in HPO: https://github.com/obophenotype/human-phenotype-ontology/pull/9952/files so far so good.

If you have time to look this over at some point..