Builds pushed to build-locker without dependency jars included

[henrychao-rcsb]

The current step when archiving a .jar build for BuildLocker only includes the jar file: https://github.com/rcsb/devops-cicd-github-actions/blob/master/.github/workflows/deploy-to-buildloker.yaml#L27

In comparison, the old GitLab CI/CD pipeline will create an archive file that also includes the dependency-jars/ directory: https://github.com/rcsb/rcsb-everglades/blob/master/.gitlab-ci.yml#L22

Without that dependency directory, the weekly update workflow tasks which call the everglades Java library will fail due to missing dependencies:

A task failed when running. Most likely run() raised an exception.
name    etl.create_indexes.CreateTermsAutocompleteIndex
contentType computational
Command line
/usr/local/bin/luigi --module weekly_update.etl.end RunETLWorkflow --ETLWorkflowConfig-appendCoastAndChannelToConfigLoc --MasterConfig-whatCoast east --MasterConfig-appendCoastToStateFilesBasePath --MasterConfig-appendCoastToFtpFileBasePath --scheduler-host etl-east.rcsb.org --workers 1
Traceback
Runtime error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/luigi/worker.py", line 191, in run
    new_deps = self._run_get_new_deps()
  File "/usr/local/lib/python3.8/dist-packages/luigi/worker.py", line 133, in _run_get_new_deps
    task_gen = self.task.run()
  File "/usr/local/lib/python3.8/dist-packages/weekly_update/etl/create_indexes.py", line 141, in run
    run_everglades_java_command("org.rcsb.everglades.elasticsearch.CreateTermsAutocomplete", self.contentType)
  File "/usr/local/lib/python3.8/dist-packages/weekly_update/etl/redwood_functions.py", line 102, in run_everglades_java_command
    return run_loader_java_command(ETLWorkflowConfig().indexerProjectName,
  File "/usr/local/lib/python3.8/dist-packages/weekly_update/etl/redwood_functions.py", line 83, in run_loader_java_command
    run_command(cmd)
  File "/usr/local/lib/python3.8/dist-packages/weekly_update/utils/general_utils.py", line 309, in run_command
    raise ValueError("Command '%s' failed with return %d. The stderr follows:\n%s"
ValueError: Command '/usr/bin/java -Xmx32g -DrcsbConfigProfile=https://cdn.rcsb.org/config/c5aecb46a9fe2698498ed4d1ab1925e9a8fef7f5/profiles/parks/prod/east/A/ -Dlog4j.configurationFile=https://cdn.rcsb.org/config/c5aecb46a9fe2698498ed4d1ab1925e9a8fef7f5/profiles/parks/prod/east/A/log4j2.everglades.xml -DlogFileName=/var/log/rcsb/everglades.CreateTermsAutocomplete-computational.log -cp /tmp/wuw_CreateTermsAutocomplete_n8pdddqh/rcsb-everglades-1.11.10-SNAPSHOT.jar org.rcsb.everglades.elasticsearch.CreateTermsAutocomplete -ct=computational' failed with return 1. The stderr follows:
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
    at org.rcsb.everglades.elasticsearch.CreateTermsAutocomplete.(CreateTermsAutocomplete.java:28)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    ... 1 more

The solution should be to include that directory as part of the deploy-build-locker-jar job step, but I'm not sure if this would impact other repos that call this workflow or not. FYI @Ingvord

[Ingvord]

I would suggest the other way around - to create a single fat jar in everglades, see this PR: rcsb-everglades/#12

That makes all the projects consistent.

A side note:

Command

/usr/bin/java -Xmx32g -DrcsbConfigProfile=https://cdn.rcsb.org/config/c5aecb46a9fe2698498ed4d1ab1925e9a8fef7f5/profiles/parks/prod/east/A/ -Dlog4j.configurationFile=https://cdn.rcsb.org/config/c5aecb46a9fe2698498ed4d1ab1925e9a8fef7f5/profiles/parks/prod/east/A/log4j2.everglades.xml -DlogFileName=/var/log/rcsb/everglades.CreateTermsAutocomplete-computational.log -cp /tmp/wuw_CreateTermsAutocomplete_n8pdddqh/rcsb-everglades-1.11.10-SNAPSHOT.jar org.rcsb.everglades.elasticsearch.CreateTermsAutocomplete -ct=computational

does not contain a classpath ref to dependency-jars/

[josemduarte]

I do like the single jar file solution, it is more standard.

However, the issue is that we use the tgz bundle solution currently in other projects.

Group 1. These 2 work similarly (the tgz bundle contains the main jar and a dependency-jars dir):

• everglades
• redwood

Group 2. These bunch seem to be doing tgz but containing a single jar file (@JonStargaryen could you confirm?):

• seqmotif
• strucmotif
• teilen
• download
• pecos

Group 3. And for completeness, then we have war packaging for these others:

• yosemite
• arches
• borrego
• shape

So I would agree to go to the single jar solution, but it requires:

• fix redwood with a similar PR to https://github.com/rcsb/rcsb-everglades/pull/12
• make sure the projects in Group 2 are being built with a single jar in github workflows
• a few changes in SysAdmin and weekly-update-workflow to adapt from tgz to jar

[Ingvord]

In Group 2 the current workflow is archiving a single jar into tgz, so there is no need to adjust SysAdmin in that regard

Group 3 has been already taken care of, see #17

[Ingvord]

A side note:

Command

/usr/bin/java -Xmx32g -DrcsbConfigProfile=https://cdn.rcsb.org/config/c5aecb46a9fe2698498ed4d1ab1925e9a8fef7f5/profiles/parks/prod/east/A/ -Dlog4j.configurationFile=https://cdn.rcsb.org/config/c5aecb46a9fe2698498ed4d1ab1925e9a8fef7f5/profiles/parks/prod/east/A/log4j2.everglades.xml -DlogFileName=/var/log/rcsb/everglades.CreateTermsAutocomplete-computational.log -cp /tmp/wuw_CreateTermsAutocomplete_n8pdddqh/rcsb-everglades-1.11.10-SNAPSHOT.jar org.rcsb.everglades.elasticsearch.CreateTermsAutocomplete -ct=computational

does not contain a classpath ref to dependency-jars/

The mystery has been solved! Actually, it was the jar plugin that put a ref into the jar's manifest, effectively hiding it...

[JonStargaryen]

Group 2. These bunch seem to be doing tgz but containing a single jar file (@JonStargaryen could you confirm?):

• seqmotif
• strucmotif
• teilen
• download
• pecos

Yes, all of these build a single .tgz with a single fat executable .jar in them.

Just a heads-up: https://github.com/rcsb/rcsb-download-app and https://github.com/rcsb/rcsb-pecos use dist/ as target folder, but that can be easily changed as needed.

[josemduarte]

In Group 2 the current workflow is archiving a single jar into tgz, so there is no need to adjust SysAdmin in that regard

Oh that's a good point... true. Then, also for Group 1 we should be safe, because the weekly-update-workflow function will still still work in the same way: https://github.com/rcsb/weekly-update-workflow/blob/master/weekly_update/etl/redwood_functions.py#L57-L62

[Ingvord]

@JonStargaryen thanks for mentioning more dist/ projects! Surprisingly, rcsb-pecos looks ok, e.g. this build

[JonStargaryen]

Surprisingly, rcsb-pecos looks ok, e.g. this build

Perfect. Could be because it uses the shade-plugin, which might behave differently/better.

[josemduarte]

Perfect. Could be because it uses the shade-plugin, which might behave differently/better.

Mmmh that would be another issue. Shouldn't we settle on assembly plugin everywhere, now that we are at cleaning up?

[Ingvord]

Surprisingly, rcsb-pecos looks ok, e.g. this build

Perfect. Could be because it uses the shade-plugin, which might behave differently/better.

Well, I may be mistaken, mvn could simply build a thin jar in the target folder and that's the one that was uploaded... We need to double check. Will update our Miro board in the mean time

[josemduarte]

Just a heads-up: https://github.com/rcsb/rcsb-download-app and https://github.com/rcsb/rcsb-pecos use dist/ as target folder, but that can be easily changed as needed.

@JonStargaryen could you take care of those 2 and standardise them in same way as we did for others (using target)? I think that would be the last TODO for java CI that I have in my list.

[JonStargaryen]

@JonStargaryen could you take care of those 2 and standardise them in same way as we did for others (using target)? I think that would be the last TODO for java CI that I have in my list.

Raised a PR for each project to output the artifact to target/. Download will then also use jib + GitHub workflows.

[JonStargaryen]

Kinda unrelated: Is it possible to fix the typo in deploy-to-buildloker.yaml.

[Ingvord]

Kinda unrelated: Is it possible to fix the typo in deploy-to-buildloker.yaml.

Totally. Done in 58fd894

[henrychao-rcsb]

Closing out this old issue, as the issue has been addressed and jar files/builds are working as expected now.