Open brailsmt opened 3 years ago
@brailsmt I'm having a hard time getting this to work on my end. Can you provide some exact code that can replicate this problem?
I would venture to guess this is the same problem as #599
/usr/lib/jvm/java-8-adoptopenjdk-hotspot
, if that directory exists the cookbook will refuse to extract the java tarball into the directory. The code that introduces the issue is https://github.com/sous-chefs/java/blob/master/resources/adoptopenjdk_linux_install.rb#L53, which is used in archive_file on line 73. The java_home property defaults to /usr/lib/jvm/java-8-adoptopenjdk-hotspot/jdk8u232-b09
(for a java 8, adoptopenjdk, hotspot jvm). The extract_dir
is then /usr/lib/jvm/java-8-adoptopenjdk-hotspot/
. The archive_file resource will then refuse to extract the tarball to the extract directory if it already exists, regardless of whether /usr/lib/jvm/java-8-adoptopenjdk-hotspot/jdk8u232-b09
exists.In the above scenario, I could recreate the issue following these steps:
$ mkdir /usr/lib/jvm/java-8-adoptopenjdk-hotspot/
$ chef-client -o 'recipe[java]'
Where the java recipe just contains:
adoptopenjdk_install '8' do
variant adoptopenjdk
url 'https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries/releases/download/jdk8u272-b10/OpenJDK8U-jdk_x64_linux_8u272b10.tar.gz'
checksum '654a0b082be0c6830821f22a9e1de5f9e2feb9705db79e3bb0d8c203d1b12c6a'
end
EDIT: I apologize if there are errors in how to recreate the issue, I will work up a complete example this afternoon or this evening when I have time to more properly test it, instead of trying to recreate it from memory as I did here.
I think I am hitting this too. Recently upgraded our wrapper from using ~> 8.3.1
to ~> 8.5.0
and I was not seeing the correct (new) tarball be extracted until I removed the /usr/lib/jvm/java-8-adoptopenjdk-hotspot
directory. I believe the issue is here. Maybe the overwrite
property of archive_file
needs to be set to true
?
UPDATE: Confirmed that is the issue. See debug output below:
* adoptopenjdk_linux_install[8] action install[2021-03-25T17:15:23-04:00] INFO: Processing adoptopenjdk_linux_install[8] action install (cg_java::adoptopenjdk line 17)
* directory[/usr/lib/jvm] action create[2021-03-25T17:15:23-04:00] INFO: Processing directory[/usr/lib/jvm] action create (/var/chef/cache/cookbooks/java/resources/adoptopenjdk_linux_install.rb line 57)
(up to date)
* remote_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u282b08.tar.gz] action create[2021-03-25T17:15:23-04:00] INFO: Processing remote_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u282b08.tar.gz] action create (/var/chef/cache/cookbooks/java/resources/adoptopenjdk_linux_install.rb line 64)
(up to date)
* archive_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u282b08.tar.gz] action extract[2021-03-25T17:15:23-04:00] INFO: Processing archive_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u282b08.tar.gz] action extract (/var/chef/cache/cookbooks/java/resources/adoptopenjdk_linux_install.rb line 72)
[2021-03-25T17:15:23-04:00] DEBUG: Not extracting archive as /usr/lib/jvm/java-8-adoptopenjdk-hotspot exists and resource not set to overwrite.
(up to date)
Directory contents:
$ ls -l /usr/lib/jvm/java-8-adoptopenjdk-hotspot/
total 0
drwxr-xr-x 8 root root 176 Jul 28 2020 jdk8u265-b01
@bkonick We worked around this issue by modifying java_home
in our recipe prior to calling the java resource. We changed it include the java version like this:
jdk_install_base_dir = "/usr/lib/jvm/java-#{java_version}-#{java_flavor}-#{java_variant}"
jdk_install_dir = File.join(jdk_install_base_dir, jdk_version, jdk_version)
node.default['java']['java_home'] = jdk_install_dir
This awkwardly doubles up the version in the directory like this /usr/lib/jvm/java-8-adoptopenjdk-hotspot/jdk8u232-b0/jdk8u232-b0
, but it works. I'm not sure that just setting overwrite
is correct because the java resource also sets JAVA_HOME for everyone on the system. That will break other applications unless they are all stopped and restarted with the correct, new JAVA_HOME for the java version. On long running servers with multiple applications, that isn't desirable.
@bkonick We worked around this issue by modifying
java_home
in our recipe prior to calling the java resource. We changed it include the java version like this:jdk_install_base_dir = "/usr/lib/jvm/java-#{java_version}-#{java_flavor}-#{java_variant}" jdk_install_dir = File.join(jdk_install_base_dir, jdk_version, jdk_version) node.default['java']['java_home'] = jdk_install_dir
This awkwardly doubles up the version in the directory like this
/usr/lib/jvm/java-8-adoptopenjdk-hotspot/jdk8u232-b0/jdk8u232-b0
, but it works. I'm not sure that just settingoverwrite
is correct because the java resource also sets JAVA_HOME for everyone on the system. That will break other applications unless they are all stopped and restarted with the correct, new JAVA_HOME for the java version. On long running servers with multiple applications, that isn't desirable.
I am going to do some local testing today to see how the overwrite property behaves. I think in this case the JDK is always unpacked to a subdirectory, so I am hoping that it doesn't actually clear out the directory, but just permits it to have contents extracted to it.
Interestingly, the overwrite property causes the extract to happen during every run which breaks enforce_idempotency
in test-kitchen. I may have to go with that workaround after all @brailsmt
This is a limitation of the archive_file
resource, and is a regression from previous versions of the java cookbook.
@bkonick We worked around this issue by modifying
java_home
in our recipe prior to calling the java resource. We changed it include the java version like this:jdk_install_base_dir = "/usr/lib/jvm/java-#{java_version}-#{java_flavor}-#{java_variant}" jdk_install_dir = File.join(jdk_install_base_dir, jdk_version, jdk_version) node.default['java']['java_home'] = jdk_install_dir
This awkwardly doubles up the version in the directory like this
/usr/lib/jvm/java-8-adoptopenjdk-hotspot/jdk8u232-b0/jdk8u232-b0
, but it works. I'm not sure that just settingoverwrite
is correct because the java resource also sets JAVA_HOME for everyone on the system. That will break other applications unless they are all stopped and restarted with the correct, new JAVA_HOME for the java version. On long running servers with multiple applications, that isn't desirable.
@brailsmt do you have a more specific example of how you are implementing this workaround? I was trying to re-create it in our wrapper but was having some issues. What is jdk_version
defined as?
For me I added the following to the archive_file for tarball with adoptopenjdk. This would make it extract contents.
overwrite true
not_if { ::Dir.exist?(new_resource.java_home) }
This stops it from extract again if the java_home exists. It was skipping if the partial path was there.
I added the not_if for archive_file as well to keep it from downloading since I don't backup the file.
@bkonick The jdk_version
can be seen in the path that I mentioned. It is jdk8u232-b09
and is derived from the URL where the jdk is downloaded. In our case, it is a mirror of https://github.com/AdoptOpenJDK/openjdk8-binaries/releases that we host internally. In this specific example that URL is, and the jdk_version is the path directly preceding the tar.gz:
https://internalrepo.net/github-releases/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u232-b09/OpenJDK8U-jdk_x64_linux_hotspot_8u232b09.tar.gz
Instead of overwriting and possibly halting running programs we probably need to place AdoptOpenJDK in sub-folders and have alternatives take care of it for us. That way if files are removed during a new patch they're not left behind.
:speaking_head: Foreword
While working with the latest version of the java cookbook and chef 16, we are unable to upgrade to a later java version. We often need to uplift to a newer java version due to security and bug fixes. We are not yet running in a containerized environment so we need the ability to upgrade java versions even if previous versions were installed. We are also unable to safely remove all old installations of java prior to upgrading.
:ghost: Brief Description
If java has already been installed on the node in question, then subsequent runs of the sous-chef/java recipe will silently fail to upgrade the java version. We have created a thin wrapper around the java resource to install java. The recipe is essentially this (actual values changed, obviously):
If we run
chef-client -o "recipe[company_java]"
, the chef-client run succeeds. If we then override the config and install a new version of java, it fails to upgrade the java version, but the cookbook still succeeds.:pancakes: Cookbook version
We are experiencing this with 8.4.0.
:woman_cook: Chef-Infra Version
16.6.14
:tophat: Platform details
Oracle Linux 7.9
Steps To Reproduce
Steps to reproduce the behavior:
/usr/lib/jvm/java-8-adoptopenjdk-hotspot will contain the jdk8u272-b10 directory, and there will not be a jdk8u275-b01 directory.
:police_car: Expected behavior
The expected behavior is that version of java that was installed in step 2 would be the java version installed and linked through /etc/alternatives.
:heavy_plus_sign: Additional context
Here is a snippet from a chef-client run that was intended to install jdk8u232-b09:
Instead of the application being run with jdk8u232-b09, it was running with the version of java that was previously installed which happened to be jdk8u265-b01. While the version that was previously installed was newer is beside the point and is due to a separate issue in our own cookbooks.