[BUG] draft release workflow fails with an invalid certificate

[dblock]

Describe the bug

https://github.com/opensearch-project/logstash-output-opensearch/actions/runs/5954433808/job/16150938024

Run gem build logstash-output-opensearch.gemspec
  gem build logstash-output-opensearch.gemspec
  mkdir dist && mv logstash-output-opensearch-*.gem dist/
  tar -cvf artifacts.tar.gz dist
  shell: /usr/bin/bash -e {0}
  env:
    AWS_DEFAULT_REGION: us-east-1
    AWS_REGION: us-east-1
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
    AWS_SESSION_TOKEN: ***
WARNING:  open-ended dependency on logstash-codec-plain (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  open-ended dependency on logstash-devutils (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  open-ended dependency on flores (>= 0, development) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  See https://guides.rubygems.org/specification-reference/ for help
INFO:  Your certificate has expired, trying to re-sign it...
ERROR:  While executing gem ... (Gem::Security::Exception)
    certificate /CN=opensearch/DC=amazon/DC=com not valid after [2](https://github.com/opensearch-project/logstash-output-opensearch/actions/runs/5954433808/job/16150938024#step:6:2)02[3](https://github.com/opensearch-project/logstash-output-opensearch/actions/runs/5954433808/job/16150938024#step:6:3)-08-17 17:52:3[4](https://github.com/opensearch-project/logstash-output-opensearch/actions/runs/5954433808/job/16150938024#step:6:4) UTC
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/security/policy.rb:97:in `check_cert'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/security/policy.rb:243:in `verify'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/security/signer.rb:1[5](https://github.com/opensearch-project/logstash-output-opensearch/actions/runs/5954433808/job/16150938024#step:6:5)4:in `sign'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/package/tar_writer.rb:190:in `add_file_signed'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/package.rb:280:in `add_metadata'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/package.rb:308:in `block in build'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/package/tar_writer.rb:78:in `new'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/package.rb:307:in `block in build'
    org/jruby/RubyIO.java:1178:in `open'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/package/file_source.rb:25:in `with_write_io'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/package.rb:30[6](https://github.com/opensearch-project/logstash-output-opensearch/actions/runs/5954433808/job/16150938024#step:6:6):in `build'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/package.rb:136:in `build'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/commands/build_command.rb:99:in `build_package'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/commands/build_command.rb:[8](https://github.com/opensearch-project/logstash-output-opensearch/actions/runs/5954433808/job/16150938024#step:6:8)[9](https://github.com/opensearch-project/logstash-output-opensearch/actions/runs/5954433808/job/16150938024#step:6:9):in `build_gem'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/commands/build_command.rb:69:in `execute'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/command.rb:323:in `invoke_with_build_args'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/command_manager.rb:185:in `process_args'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/command_manager.rb:149:in `run'
    /home/runner/.rubies/jruby-9.4.3.0/lib/ruby/stdlib/rubygems/gem_runner.rb:51:in `run'
    /home/runner/.rubies/jruby-9.4.3.0/bin/jgem:21:in `<main>'
    org/jruby/RubyKernel.java:1[10](https://github.com/opensearch-project/logstash-output-opensearch/actions/runs/5954433808/job/16150938024#step:6:10)5:in `load'
    /home/runner/.rubies/jruby-9.4.3.0/bin/gem:4:in `<main>'

[dblock]

Closed via #220

[peterzhuamazon]

The rootcause is still unclear but the reason of error has been found. See log: dataprepper-2.0.2-install.txt

When install without any security settings, aws-sdk-s3 as a transient dependency to aws-sdk is being installed as 1.134.0. https://github.com/opensearch-project/logstash-output-opensearch/blob/main/logstash-output-opensearch.gemspec#L48

When switching to any of the security settings, it will then fall back to 1.131.0, as shown in the log above.

Since aws-sdk-s3 never being signed, when running with -P HighSecurity, it will fail the run due to high security requires all packages to be signed, no matter having signature or not.

[opensearch@fa202d0638a5 logstash-output-opensearch]$ gem install logstash-output-opensearch-2.0.2-java.gem -P HighSecurity
Fetching aws-sdk-s3-1.131.0.gem
ERROR:  While executing gem ... (Gem::Security::Exception)
    unsigned gems are not allowed by the High Security policy

No Sig Files in the aws-sdk-s3 gem

[opensearch@fa202d0638a5 logstash-output-opensearch]$ ll testsdk/
total 676
-rw-rw-r-- 1 opensearch opensearch 344064 Aug 24 23:09 aws-sdk-s3-1.134.0.gem
-r--r--r-- 1 opensearch opensearch    300 Aug 24 18:10 checksums.yaml.gz
-r--r--r-- 1 opensearch opensearch 339253 Aug 24 18:10 data.tar.gz
-r--r--r-- 1 opensearch opensearch   1358 Aug 24 18:10 metadata.gz

The solution is to follow this order:

1. Install without security
2. Uninstall logstash
3. Install with MediumSecurity
4. Uninstall logstash
5. Install with HighSecurity
6. Successfully installed gem

It is possible to directly install with MediumSecurity to start with tho.

Thanks.

[peterzhuamazon]

PRs:

• 220

• 223

• https://github.com/opensearch-project/opensearch-build-libraries/pull/298

• 224