Please add a maven cache

[KonradHoeffner]

Compiling the different OLS components takes several minutes, mostly due to downloading maven dependencies. This could be drastically sped up by caching maven dependencies, for example by mounting a local ~/.m2 directory or some cache volume. Right now, building everything on the unified docker compose branch, https://github.com/annosaxfdm/ols/commit/d3bb1d8c0bb498f99228c97329d13a706f56367a docker-compose --no-cache takes more than 400s.

[KonradHoeffner]

The ols-indexer alone took 461s:

 => [build 5/5] RUN cd /opt/ols && ls && mvn clean package -DskipTests                              461.1s

[KonradHoeffner]

I don't know why but adding the cache causes an error:

#11 9.386 [WARNING] Failure to transfer com.google.code.findbugs:jsr305/maven-metadata.xml from https://m2.neo4j.org/content/repositories/releases/ was cached in the local repository, resolution will not be reattempted until the update interval of maven-neo4j has elapsed or updates are forced. Original error: Could not transfer metadata com.google.code.findbugs:jsr305/maven-metadata.xml from/to maven-neo4j (https://m2.neo4j.org/content/repositories/releases/): Transfer failed for https://m2.neo4j.org/content/repositories/releases/com/google/code/findbugs/jsr305/maven-metadata.xml
#11 9.386 [WARNING] Failure to transfer com.google.code.findbugs:jsr305/maven-metadata.xml from https://repo.spring.io/libs-release was cached in the local repository, resolution will not be reattempted until the update interval of spring-releases has elapsed or updates are forced. Original error: Could not transfer metadata com.google.code.findbugs:jsr305/maven-metadata.xml from/to spring-releases (https://repo.spring.io/libs-release): Authentication failed for https://repo.spring.io/libs-release/com/google/code/findbugs/jsr305/maven-metadata.xml 401 Unauthorized
#11 9.387 [WARNING] Failure to transfer com.google.code.findbugs:jsr305/maven-metadata.xml from http://repo.spring.io/libs-milestone-local was cached in the local repository, resolution will not be reattempted until the update interval of repo.spring.io has elapsed or updates are forced. Original error: Could not transfer metadata com.google.code.findbugs:jsr305/maven-metadata.xml from/to repo.spring.io (http://repo.spring.io/libs-milestone-local): Transfer failed for http://repo.spring.io/libs-milestone-local/com/google/code/findbugs/jsr305/maven-metadata.xml 308 Permanent Redirect
#11 9.387 [WARNING] Failure to transfer com.google.code.findbugs:jsr305/maven-metadata.xml from http://m2.neo4j.org/content/repositories/releases/ was cached in the local repository, resolution will not be reattempted until the update interval of neo4j has elapsed or updates are forced. Original error: Could not transfer metadata com.google.code.findbugs:jsr305/maven-metadata.xml from/to neo4j (http://m2.neo4j.org/content/repositories/releases/): Transfer failed for http://m2.neo4j.org/content/repositories/releases/com/google/code/findbugs/jsr305/maven-metadata.xml
#11 9.396 [INFO] ------------------------------------------------------------------------
#11 9.397 [INFO] Reactor Summary for Ontology Lookup Service 3.2.1-SNAPSHOT:
#11 9.397 [INFO] 
#11 9.397 [INFO] ols-dependencies ................................... SUCCESS [  0.057 s]
#11 9.397 [INFO] Ontology Lookup Service ............................ SUCCESS [  0.002 s]
#11 9.397 [INFO] resource-usage ..................................... SUCCESS [  0.669 s]
#11 9.397 [INFO] ontology-tools ..................................... SUCCESS [  6.111 s]
#11 9.397 [INFO] ols-core ........................................... SUCCESS [  0.100 s]
#11 9.397 [INFO] ols-solr ........................................... SUCCESS [  0.380 s]
#11 9.398 [INFO] ols-mongo .......................................... SUCCESS [  0.097 s]
#11 9.398 [INFO] ols-neo4j .......................................... FAILURE [  0.786 s]
#11 9.398 [INFO] ols-web ............................................ SKIPPED
#11 9.398 [INFO] ols-solr-app ....................................... SKIPPED
#11 9.398 [INFO] ols-indexer ........................................ SKIPPED
#11 9.398 [INFO] ols-config-importer ................................ SKIPPED
#11 9.398 [INFO] ols-neo4j-app ...................................... SKIPPED
#11 9.398 [INFO] ols-apps ........................................... SKIPPED
#11 9.398 [INFO] Ontology Lookup Service ............................ SKIPPED
#11 9.398 [INFO] ------------------------------------------------------------------------
#11 9.398 [INFO] BUILD FAILURE
#11 9.398 [INFO] ------------------------------------------------------------------------
#11 9.398 [INFO] Total time:  8.342 s
#11 9.399 [INFO] Finished at: 2022-04-22T09:44:28Z
#11 9.399 [INFO] ------------------------------------------------------------------------
#11 9.399 [ERROR] Failed to execute goal on project ols-neo4j: Could not resolve dependencies for project uk.ac.ebi.spot:ols-neo4j:jar:3.2.1-SNAPSHOT: Failed to collect dependencies at org.springframework.data:spring-data-neo4j:jar:3.4.5.RELEASE -> org.neo4j:neo4j-cypher-dsl:jar:2.0.1: Failed to read artifact descriptor for org.neo4j:neo4j-cypher-dsl:jar:2.0.1: Could not transfer artifact org.neo4j:neo4j-cypher-dsl:pom:2.0.1 from/to maven-neo4j (https://m2.neo4j.org/content/repositories/releases/): Transfer failed for https://m2.neo4j.org/content/repositories/releases/org/neo4j/neo4j-cypher-dsl/2.0.1/neo4j-cypher-dsl-2.0.1.pom: Unknown host m2.neo4j.org -> [Help 1]
#11 9.400 [ERROR] 
#11 9.400 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
#11 9.400 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
#11 9.400 [ERROR] 
#11 9.400 [ERROR] For more information about the errors and possible solutions, please read the following articles:
#11 9.400 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
#11 9.400 [ERROR] 
#11 9.400 [ERROR] After correcting the problems, you can resume the build with the command
#11 9.400 [ERROR]   mvn <args> -rf :ols-neo4j

Maybe this is an unrelated problem that didn't show up before? However I don't understand why it doesn't occur when not using the cache. P.S.: Ah it seems that this is the problem that the "build-fix" fixes, apparently the following interferes with the build-fix:

COPY build-fix/. /root/.m2/repository/
RUN --mount=type=cache,target=/root/.m2 cd /opt/ols && ls && mvn clean package -DskipTests

I guess that mounting the cache will overwrite the build fix, however we can't reorder those two statements because then the build fix would be applied to late.

[KonradHoeffner]

Trying this:

COPY build-fix/. /root/build-fix
RUN --mount=type=cache,target=/root/.m2 cp -r /root/build-fix/* /root/.m2/repository/ && cd /opt/ols && ls && mvn clean package -DskipTests

This works and gets the build of ols-indexer with existing maven cache down to 24.6 s on an Intel Core i7-9700 :

 => [build 5/5] RUN --mount=type=cache,target=/root/.m2 cp -r /root/build-fix/* /root/.m2/repository/ && cd /opt/ols && ls && mvn clean package -Dskip  24.6s

Parallelizing Maven with 1 Thread per CPU (8 cores no hyperthreading on Intel Core i7-9700) via mvn clean package -T 1C does not help, in fact the time increases to 36.5s, so leave that out. Now it's up to 42.4, still much faster than before but I don't know why it varies so much.

However with ols-web there is an error:

#0 226.2 Downloading from central: https://repo
#0 226.2 [output clipped, log limit 1MiB reached]
------
failed to solve: executor failed running [/bin/sh -c cp -r /root/build-fix/* /root/.m2/repository/ && cd /opt/ols && ls && mvn clean package -DskipTests]: exit code: 1

But due to the clipped output I don't know the reason.

On the next try it worked in a complete docker compose setup so I enabled it again for ols-we, which brings the compile time of that component down to around 30-40s. However I still recommend someone else to test it before merging the pull request.

[KonradHoeffner]

Update: This version was unnecessarily building components separately and thus building dependencies multiple times. A more efficient approach is to just build everything at once in a separate container and put all the jars in a volume.