Breaking Change from Minor Release in `aws-sdk-core` from 3.198.0 to latest

dinukarajapaksha commented 3 months ago

Describe the bug

We are using inline bundler with the following gems

gemfile do
  source "https://rubygems.org"

  gem "nokogiri"
  gem "aws-sdk-core"
  gem "aws-sdk-ssm"
  gem "thor"
end

The aws-sdk-core gem version later than 3.198.0 there is a CBOR library compatibility error.

Expected Behavior

There shouldn't be any breaking changes from 3.198.0 to later minor releases

Current Behavior

When the inline script is executed it generates the following error

Traceback (most recent call last):
    19: from ../../../../../scripts/service_list:6:in `<main>'
    18: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/inline.rb:54:in `gemfile'
    17: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/settings.rb:124:in `temporary'
    16: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/inline.rb:70:in `block in gemfile'
    15: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `require'
    14: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `each'
    13: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:69:in `block in require'
    12: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:69:in `each'
    11: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:74:in `block (2 levels) in require'
    10: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:74:in `require'
     9: from /usr/local/bundle/gems/aws-sdk-ssm-1.173.0/lib/aws-sdk-ssm.rb:11:in `<top (required)>'
     8: from /usr/local/bundle/gems/aws-sdk-ssm-1.173.0/lib/aws-sdk-ssm.rb:11:in `require'
     7: from /usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core.rb:82:in `<top (required)>'
     6: from /usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core.rb:82:in `require_relative'
     5: from /usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/rpc_v2.rb:1:in `<top (required)>'
     4: from /usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/rpc_v2.rb:1:in `require_relative'
     3: from /usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/cbor.rb:3:in `<top (required)>'
     2: from /usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/cbor.rb:5:in `<module:Aws>'
     1: from /usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/cbor.rb:104:in `<module:Cbor>'
/usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/cbor.rb:85:in `set_default_engine': Unable to find a compatible cbor library. (RuntimeError)
    9: from ../../../../../scripts/service_list:6:in `<main>'
    8: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/inline.rb:54:in `gemfile'
    7: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/settings.rb:124:in `temporary'
    6: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/inline.rb:70:in `block in gemfile'
    5: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `require'
    4: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `each'
    3: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:69:in `block in require'
    2: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:69:in `each'
    1: from /usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:73:in `block (2 levels) in require'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:77:in `rescue in block (2 levels) in require': There was an error while trying to load the gem 'aws-sdk-ssm'. (Bundler::GemRequireError)
Gem Load Error is: Unable to find a compatible cbor library.
Backtrace for gem load error is:
/usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/cbor.rb:85:in `set_default_engine'
/usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/cbor.rb:104:in `<module:Cbor>'
/usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/cbor.rb:5:in `<module:Aws>'
/usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/cbor.rb:3:in `<top (required)>'
/usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/rpc_v2.rb:1:in `require_relative'
/usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core/rpc_v2.rb:1:in `<top (required)>'
/usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core.rb:82:in `require_relative'
/usr/local/bundle/gems/aws-sdk-core-3.201.0/lib/aws-sdk-core.rb:82:in `<top (required)>'
/usr/local/bundle/gems/aws-sdk-ssm-1.173.0/lib/aws-sdk-ssm.rb:11:in `require'
/usr/local/bundle/gems/aws-sdk-ssm-1.173.0/lib/aws-sdk-ssm.rb:11:in `<top (required)>'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:74:in `require'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:74:in `block (2 levels) in require'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:69:in `each'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:69:in `block in require'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `each'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `require'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/inline.rb:70:in `block in gemfile'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/settings.rb:124:in `temporary'
/usr/local/bundle/gems/bundler-2.1.4/lib/bundler/inline.rb:54:in `gemfile'

Reproduction Steps

Run a inline script with both aws-sdk-core aws-sdk-ssm in bundler 2.1.4

require "bundler/inline"
require "json"

gemfile do
  source "https://rubygems.org"

  gem "nokogiri"
  gem "aws-sdk-core"
  gem "aws-sdk-ssm"
  gem "thor"
end

Possible Solution

We temporarily fixed this by locking the version on the script. But this is causing some other BAU to not function as expected.

require "bundler/inline"
require "json"

gemfile do
  source "https://rubygems.org"

  gem "nokogiri"
  gem "aws-sdk-ssm", "1.171.0"
  gem "aws-sdk-core", "3.198.0"
  gem "thor"
end

Additional Information/Context

We expect to have a minor release compatibility over the versions

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-core

Environment details (Version of Ruby, OS environment)

ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-linux-musl]

mullermp commented 3 months ago

I'm not able to reproduce this. My script output is:

core version: 3.201.0
ssm version: 1.173.0
ruby version: 2.7.8

(minor ruby version shouldn't matter here) and my script is:

require "bundler/inline"
require "json"

gemfile do
  source "https://rubygems.org"

  gem "nokogiri"
  gem "aws-sdk-core"
  gem "aws-sdk-ssm"
  gem "thor"
end

require 'aws-sdk-ssm'

client = Aws::SSM::Client.new
commands = client.list_commands

puts "core version: #{Aws::CORE_GEM_VERSION}"
puts "ssm version: #{Aws::SSM::GEM_VERSION}"
puts "ruby version: #{RUBY_VERSION}"

Based on a warning I see running this in Ruby 3+, is it possible that you don't have bigdecimal available? Try adding bigdecimal to your gemfile?

dinukarajapaksha commented 3 months ago

I'm not able to reproduce this. My script output is:
core version: 3.201.0
ssm version: 1.173.0
ruby version: 2.7.8
(minor ruby version shouldn't matter here) and my script is:
require "bundler/inline"
require "json"

gemfile do
  source "https://rubygems.org"

  gem "nokogiri"
  gem "aws-sdk-core"
  gem "aws-sdk-ssm"
  gem "thor"
end

require 'aws-sdk-ssm'

client = Aws::SSM::Client.new
commands = client.list_commands

puts "core version: #{Aws::CORE_GEM_VERSION}"
puts "ssm version: #{Aws::SSM::GEM_VERSION}"
puts "ruby version: #{RUBY_VERSION}"
Based on a warning I see running this in Ruby 3+, is it possible that you don't have bigdecimal available? Try adding bigdecimal to your gemfile?

Tried this but still fails

mullermp commented 3 months ago

Can you try upgrading your ruby version or cleaning your gem environment?

mullermp commented 3 months ago

The code it's ultimately failing on is this:

      def load_engine(name)
        require "aws-sdk-core/cbor/#{name}_engine"
        const_name = name[0].upcase + name[1..-1] + 'Engine'
        const_get(const_name)
      end

This file should exist in the latest core - it should look for "cbor_engine". My best guess is that maybe you are monkey patching or messing with load paths and it can't find this? Is service_list your script?

Another thing to try is change the gem source code locally to do a print statement or byebug statement here and check the load paths and presence of this file. Since you are the only one to report this and I can't reproduce it, it's likely something environment related.

dinukarajapaksha commented 3 months ago

The code it's ultimately failing on is this:
      def load_engine(name)
        require "aws-sdk-core/cbor/#{name}_engine"
        const_name = name[0].upcase + name[1..-1] + 'Engine'
        const_get(const_name)
      end
This file should exist in the latest core - it should look for "cbor_engine". My best guess is that maybe you are monkey patching or messing with load paths and it can't find this? Is service_list your script?

Another thing to try is change the gem source code locally to do a print statement or byebug statement here and check the load paths and presence of this file. Since you are the only one to report this and I can't reproduce it, it's likely something environment related.

I tried executing the same script with the official ruby 2.7 Docker image and it showed no errors. I guess it has to be something to do with the packages in our image but I'm not exactly sure why it is failing with the SDK version.

we are building the Docker file similar to this

FROM hashicorp/terraform:0.14.11

ENV GEM_HOME="/usr/local/bundle"
ENV PATH $GEM_HOME/bin:$GEM_HOME/gems/bin:$PATH

RUN apk --no-progress --quiet add --update --no-cache ca-certificates jq ruby ruby-etc ruby-json build-base ruby-dev py3-pip bash

Then at the runtime run gem install bundler -v '~> 2.1.4' --no-doc

In this environment the SDK fails with the cbor lib location error. Appreciate if you can point out anything we are doing wrong here @mullermp. But I'll be closing this issue for now and thank you for the support given.

github-actions[bot] commented 3 months ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

aws / aws-sdk-ruby