Build rpm for Java-1.8.eb fails

[AnibalMG]

I am getting an error when I try to generate the rpm for Java-1.8.eb. FPM could not fine the lua files because the modulerc easyblock is only genereting the .modulerc file.

== 2019-01-30 08:48:59,358 build_log.py:251 INFO packaging...
== 2019-01-30 08:48:59,358 easyblock.py:2610 INFO Starting package step
== 2019-01-30 08:48:59,359 easyblock.py:2616 INFO Running method package_step part of step package
== 2019-01-30 08:48:59,359 easyblock.py:2031 INFO Generating rpm package in /tmp
== 2019-01-30 08:48:59,359 utilities.py:84 INFO Will be creating rpm package(s) in /tmp/eb-SbnuSD/eb-pkgs-97NtDB
== 2019-01-30 08:48:59,361 run.py:192 INFO running cmd: ['/usr/bin/env', 'fpm', '--workdir', '/tmp/eb-SbnuSD/eb-pkgs-97NtDB', '--name', 'Java-1.8', '--provides', 'Java-1.8', '-t', 'rpm', '-s', 'dir', '--version', 'eb-3.8.0', '--iteration', '1', '--description', 'Java Platform, Standard Edition (Java SE) lets you develop and deploy\n Java applications on desktops and servers.', '--url', 'http://java.com/', '--depends', 'Java-1.8.0_201', '--exclude', '.local/easybuild/software/Java/1.8/easybuild/*.log', '--exclude', '.local/easybuild/software/Java/1.8/easybuild/*.md', '.local/easybuild/software/Java/1.8', .local/easybuild/modules/all/Java/1.8.lua'] 
== 2019-01-30 08:49:00,089 build_log.py:162 ERROR EasyBuild crashed with an error (at easybuild/software/EasyBuild/3.8.0/lib/python2.6/site-packages/vsc_base-2.8.3-py2.6.egg/vsc/utils/exceptions.py:124 in __init__): cmd "['/usr/bin/env', 'fpm', '--workdir', '/tmp/eb-SbnuSD/eb-pkgs-97NtDB', '--name', 'Java-1.8', '--provides', 'Java-1.8', '-t', 'rpm', '-s', 'dir', '--version', 'eb-3.8.0', '--iteration', '1', '--description', 'Java Platform, Standard Edition (Java SE) lets you develop and deploy\n Java applications on desktops and servers.', '--url', 'http://java.com/', '--depends', 'Java-1.8.0_201', '--exclude', '/.local/easybuild/software/Java/1.8/easybuild/*.log', '--exclude', '.local/easybuild/software/Java/1.8/easybuild/*.md', '.local/easybuild/software/Java/1.8', '.local/easybuild/modules/all/Java/1.8.lua']" exited with exit code 1 and output:
{:timestamp=>"2019-01-30T08:49:00.019756+0000", :message=>"Invalid package configuration: Cannot package the path '/.local/easybuild/modules/all/Java/1.8.lua', does it exist?", :level=>:error}
 (at easybuild/software/EasyBuild/3.8.0/lib/python2.6/site-packages/easybuild_framework-3.8.0-py2.6.egg/easybuild/tools/run.py:501 in parse_cmd_output)
== 2019-01-30 08:49:00,089 easyblock.py:2864 WARNING build failed (first 300 chars): cmd "['/usr/bin/env', 'fpm', '--workdir', '/tmp/eb-SbnuSD/eb-pkgs-97NtDB', '--name', 'Java-1.8', '--provides', 'Java-1.8', '-t', 'rpm', '-s', 'dir', '--version', 'eb-3.8.0', '--iteration', '1', '--description', 'Java Platform, Standard Edition (Java SE) lets you develop and deploy\n Java application
== 2019-01-30 08:49:00,089 easyblock.py:286 INFO Closing log for application name Java version 1.8

I saw the issue on EB 3.7.1 and 3.8.0.

[boegel]

So the problem is basically that the packaging logic expects to find a module file Java/1.8.lua, but there is none?

This is a tricky one to solve, since the .modulerc.lua is not owned only by Java/1.8, you could have another module wrapper for Java/1.9 as well for example (which would also be registered in .modulerc.lua)...

[Micket]

Embed some sort of install-script that performs this step (in an otherwise empty RPM)?

[boegel]

It may make sense to consult @rtmclay on this one...

How does TACC handle installations that involve touching Java/.modulerc.lua to register a short-hand version like 1.8 for 1.8.0_212?

[rtmclay]

Well, we only install java occasionally and only the system one. But we do have a similar problem with our .version files for all packages.

What we do is have each rpm package that has a module file also create a .version.X.Y.Z file where X.Y.Z is the version (so a .version.1.8.0_212 in this case). Then a separate tool creates a symlink between .version and .version.X.Y.Z.

Normally, we can use the highest version to be the default but sometimes version 1.8 is preferred over 1.9 say. So in those cases, we create a symlink between the .version.X.Y.Z and .version

[boegel]

@rtmclay That approach won't work here though, since the Java/.modulerc.lua may have to contain multiple module_version statements, each of which tied to a different software version (and hence separate package)...

Can we avoid that somehow?

[rtmclay]

You can't. You can't have one file shared by multiple rpm. The solution is to not put .modulerc.lua in the rpm.

[boegel]

@rtmclay Is there a way to have multiple .modulerc files in a single directory?

For the Java/1.8 and Java/11 wrappers, literally the only thing to install/change is the .modulerc.lua, so not including it in the RPM is not really a good option ;)

[boegel]

Idea pitched by @Micket: how about letting Lmod support a .modulerc/ directory?

That way each Java wrapper could own a separate file in that directory...

[rtmclay]

I don't understand what you are trying to do. Say we are creating an rpm for Java 1.8 we would create a file that we placed in the rpm called:

.modulerc-1.8.lua

For the rpm for Java 1.9 we would create:

 .modulerc-1.9.lua

Then we use a separate tool to make a symbolic link. Say both java 1.8 and java 1.9 are installed. Then we have a tool which creates a symbolic link between .modulerc.lua and .modulerc-1.8.lua.

What is wrong with that? You only get one default. I don't know what a .modulerc.d directory would mean. You couldn't call it .modulerc because that would interfere with the old stuff. Sure you could set aliases. But what if you tried to set a default in each. What takes priority?

[boegel]

@rtmclay This is not related to defining a default version for Java.

We use the .modulerc to define aliases or wrappers, here are the contents of Java/.modulerc.lua when both the Java/1.8 and Java/11 wrappers are installed:

module_version("Java/11.0.2", "11")
module_version("Java/1.8.0_212", "1.8")

The suggestion is to add support to let Lmod (also) pick up on something like .modulerc.d/Java-1.8.lua:

module_version("Java/1.8.0_212", "1.8")

and .modulerc.d/Java-11.lua:

module_version("Java/11.0.2", "11")

That way, the files are separate and can be included in the respective RPMs without a problem. The actual names of the files in .modulerc.d are mostly irrelevant (unless perhaps w.r.t. order for letting one overrule another, but that's not relevant from the EasyBuild side of it).

[rtmclay]

So why make Lmod do the work?. RPM's support post install scripts. Have the post install script take all the files in .modulerc.d and combine them into .modulerc.lua with "cat".

[rtmclay]

I forgot to mention that you'll need to also create a cleanup script which rebuilds the .modulerc.lua file after the old rpm is removed.

[boegel]

@rtmclay Well, ok, but that only works when EB's packaging feature is being used to create packages for software installations...

If EasyBuild is used to directly install software, then we'd need to keep doing what we do now (generating .modulerc.lua directly). Or I guess in that case eb could take the entries from .modulerc.d and combine them into .modulerc.lua, while taking some care not to kick out any entries from .modulerc.lua...

Anyway, I need to think about this some more, your suggestion could work, but makes things quite complicated imho...

[Mycron]

Hello, guys! I've made some changes to avoid this kind of issue in my fork. Can you look at it and review it? It's maybe a potential solution for such cases https://github.com/Mycron/easybuild-framework/blob/85bd95d974e59fb87e381ea71cd85c0b46201ec5/easybuild/tools/package/utilities.py#L139

[AnibalMG]

@Mycron Great! It works for me!

[alirezaghavaminia]

Just a thought, Why we don't create a Lua module for Java-11 instead of modifying the modulerc file?

[Micket]

Very late reply, but since this issue isn't resolved, i'll just shed some light on the issue:

A Java/11 module would conflict with the actual versions e.g. Java/11.x.x.x.

Unfortunately, the workaround @Mycron made also completely breaks if you

1. Need to install mutiple Java version, e.g . Java-11.eb + Java-13.eb as both would try to modify the same file
2. The Java-XX.eb modules are expected to be rebuilt frequently, making them just a bad fit for RPM installs. I suppose a rpm reinstall would work.

Currently I'm thinking, assuming a modern Lmod is in use, one can just have the RPM completely empty. Shortened module names like "Python/3" already works.

Example, with .modulerc:

$ module load Groovy/2.5.9-Java-11
$ module list
Currently Loaded Modules:
  1) Java/11 -> Java/11.0.20   2) Groovy/2.5.9-Java-11

without any .modulerc:

$ module purge
$ mv .modulerc.lua .modulerc.lua_bak
$ module load Groovy/2.5.9-Java-11
$ module list
Currently Loaded Modules:
  1) Java/11.0.20   2) Groovy/2.5.9-Java-11

It will still work for these simple cases for anything that uses depends_on("Java/11")

For cases like STAR-CCM+ where we have

module_version("STAR-CCM+/18.06.006-r8", "2310-r8")

it's trickier. Making the fpm add a post script that modified the .modulerc should be possible though. But, if we just want to support Java and some simpler cases like that, then just making the java wrapper libraries do literally nothing will solve this issue.