`archive_file` within `corretto_install` will not extract if directory already exists

jakauppila commented 4 years ago

:speaking_head: Foreword

Thank for taking the time to fill this bug report fully. Without it we may not be able to fix the bug, and the issue may be closed without resolution.

:ghost: Brief Description

archive_file within corretto_install will not extract the archive if the parent directory already exists.

https://github.com/chef/chef/blob/master/lib/chef/resource/archive_file.rb#L97

This is problematic if I want to install a different version within a major family.

:pancakes: Cookbook version

8.3.0

:woman_cook: Chef-Infra Version

Cinc 16.2.50

:tophat: Platform details

bento/centos-7

Steps To Reproduce

Steps to reproduce the behavior:

Execute the following recipe:

directory '/usr/lib/jvm/java-11-corretto' do
  recursive true
end

corretto_install '11' do
  checksum 'dbbf98ca93b44a0c81df5a3a4f2cebf467ec5c30e28c359e26615ffbed0b454f'
  action :install
end

:police_car: Expected behavior

Corretto 11 is successfully installed/extracted to /usr/lib/jvm/java-11-corretto/amazon-corretto-11.0.7.10.1-linux-x64

:heavy_plus_sign: Additional context

This is more a limitation around how archive_file is designed. From a corretto_install perspective, I would prefer to have more control around where exactly the Java installation is going rather than being in a sub-folder as generated by the tar.gz archive.

To that end, I guess I would rather just utilize tar (like in the tomcat cookbook) instead of archive_file and instead extract the archive directory into the java_home specified (ideally that has the full version specified).

tar -xzf #{Chef::Config[:file_cache_path]}/#{tarball_name} -C #{new_resource.java_home} --strip-components=1

So in my ideal scenario:

corretto_install '11.0.7.10.1' do
  java_home '/app/java/amazon-corretto-11.0.7.10.1'
  action :install
end

Would result in everything one folder deep of the archive to be extracted into the java_home path specified.

jakauppila commented 3 years ago

Confirmed that this behavior is happening inside of archive_file at https://github.com/chef/chef/blob/master/lib/chef/resource/archive_file.rb#L97

       Recipe: test::test
         * directory[/usr/lib/jvm/java-11-corretto] action create[2020-09-09T20:13:38+00:00] INFO: Processing directory[/usr/lib/jvm/java-11-corretto] action create (test::test line 1)
       [2020-09-09T20:13:38+00:00] INFO: directory[/usr/lib/jvm/java-11-corretto] created directory /usr/lib/jvm/java-11-corretto

           - create new directory /usr/lib/jvm/java-11-corretto
           - restore selinux security context
         * corretto_install[11] action install[2020-09-09T20:13:38+00:00] INFO: Processing corretto_install[11] action install (test::test line 5)

           * directory[/usr/lib/jvm] action create[2020-09-09T20:13:38+00:00] INFO: Processing directory[/usr/lib/jvm] action create (/tmp/kitchen/cache/cookbooks/java/resources/corretto_install.rb line 56)
        (up to date)
           * remote_file[/tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz] action create[2020-09-09T20:13:38+00:00] INFO: Processing remote_file[/tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz] action create (/tmp/kitchen/cache/cookbooks/java/resources/corretto_install.rb line 63)
       [2020-09-09T20:16:31+00:00] INFO: remote_file[/tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz] created file /tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz

             - create new file /tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz[2020-09-09T20:16:31+00:00] INFO: remote_file[/tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz] updated file contents /tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz

             - update content in file /tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz from none to dbbf98
             (file sizes exceed 10000000 bytes, diff output suppressed)[2020-09-09T20:16:32+00:00] INFO: remote_file[/tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz] mode changed to 644

             - change mode from '' to '0644'
             - restore selinux security context
           * archive_file[/tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz] action extract[2020-09-09T20:16:32+00:00] INFO: Processing archive_file[/tmp/kitchen/cache/amazon-corretto-11-x64-linux-jdk.tar.gz] action extract (/tmp/kitchen/cache/cookbooks/java/resources/corretto_install.rb line 71)
       [2020-09-09T20:16:32+00:00] DEBUG: Not extracting archive as /usr/lib/jvm/java-11-corretto exists and resource not set to overwrite.
        (up to date)

We can't just add overwrite true to archive_file since we would lose idempotency on subsequent executions.

ramereth commented 3 years ago

@jakauppila do you think you could make a PR to chef/chef to maybe fix this in archive_file?

jakauppila commented 3 years ago

@ramereth I don't think the behavior of archive_file is inherently wrong, but moreso how it's used here and the tars we're dealing with.

To walk through what's done in the following corretto_install resource:

corretto_install '11' do
  action :install
end

Create /usr/lib/jvm
Download Corretto "latest" version 11 (https://corretto.aws/downloads/latest/amazon-corretto-11-x64-linux-jdk.tar.gz)
Extract amazon-corretto-11-x64-linux-jdk.tar.gz to /usr/lib/jvm/java-11-corretto
1. It first attempts to create this target directory and goes no further if it already exists
  - You can force it with overwrite, but then we'd be executing every time
2. The resulting extracted path ends up being /usr/lib/jvm/java-11-corretto/amazon-corretto-11.0.8.10.1-linux-x64

Ultimately I think there's some fundamental changes that need to be made to the corretto_install resource that I touched on in #628:

The logic of taking a major version 8 or 11 is somewhat flawed since it basically assumes the current full version and checksum as https://corretto.aws/downloads/latest/amazon-corretto-11-x64-linux-jdk.tar.gz is constantly changing.
- As soon as the version behind that latest changes, the resource still start to (and currently) fails due to the checksum not being up-to-date.

I have opened https://github.com/chef/chef/issues/10229 for being able to support something similar to tar's --strip-components as that could allow us to extract the Corretto tar to a specific location as defined by the user.

jakauppila commented 3 years ago

Not sure how I missed it, but this is just a duplicate of #599

sous-chefs / java