how to force a stub to raise an exception?

[glasswalk3r]

I'm trying to fix an issue with Awspec::Type::SnsTopic: when there is no SNS Topic with the given ARN, an exception Aws::SNS::Errors::NotFound is raised by the AWS::SNS::Client.

In some tests this generates some inconsistences: in some cases the exception Awspec::NoExistingResource is raised, sometimes the Aws::SNS::Errors::NotFound.

I already have #444 to fix that (evaluated through a real connection with AWS), but now I'm struggling to make the stub generate an exception. The fact is, different from other classes that may return an empty response, AWS::SNS::Client raises the exception, so I can't just add an empty response.

I was able to find this official documentation but I wasn't able to far to figure out how to manipulate the SNS client creation and provide the parameters as described by the documentation.

[glasswalk3r]

Created the PR #454 to mark the testing of those features as pending.

[k1LoW]

but now I'm struggling to make the stub generate an exception.

I think that this problem is related to all resource types.

I almost give up on this problem.

[glasswalk3r]

I think that this problem is related to all resource types.

Surely it is. Maybe not all classes will raise a exception if the resource cannot be located by it's ID (whatever that ID means), but that doesn't mean they won't generate exceptions given other circumstances.

For example, Aws::SNS::Client can return those exceptions when invoking get_topic_attributes:

• Aws::SNS::Errors::NotFound
• Aws::SNS::Errors::InvalidClientTokenId
• Aws::SNS::Errors::InvalidParameter

I almost give up on this problem.

I strongly suggest for not doing it.

Besides the obvious use case with SNS, not testing for exceptions when they are expected leaves the project specs only covering the "happy path".

It is not clear to me yet how the client loads the stub, it's a bit too "magical" for me yet. Meanwhile I'm trying to figure it out, I've being doing some experiments with promising results, even though I don't quite understand them.

I'll keep you posted. Keep the faith. :smiley:

[glasswalk3r]

My first attempt to use stubs with exceptions:

Aws.config[:sns] = {
  stub_responses: {
    get_topic_attributes: Aws::SNS::Errors::NotFound,
    list_subscriptions_by_topic: Aws::SNS::Errors::NotFound
  }
}

And the related spec:

require 'spec_helper'
Awspec::Stub.load 'sns_topic_exception'

invalid_topic_arn = 'arn:aws:sns:us-east-1:123456789:invalid'

describe sns_topic(invalid_topic_arn) do
  context 'Issue https://github.com/k1LoW/awspec/issues/445 is still open' do
    it { should_not exist }
  end
end

That didn't work out very well:

$ bundle exec rspec -f d spec/type/sns_topic_spec_exception.rb 

sns_topic 'arn:aws:sns:us-east-1:123456789:invalid'
  Issue https://github.com/k1LoW/awspec/issues/445 is still open
    should not exist (FAILED - 1)

Failures:

  1) sns_topic 'arn:aws:sns:us-east-1:123456789:invalid' Issue https://github.com/k1LoW/awspec/issues/445 is still open should not exist
     Failure/Error: results = client.send(m, *args, &block)

     ArgumentError:
       wrong number of arguments (given 0, expected 2)
     # ./lib/awspec/helper/client_wrap.rb:26:in `method_missing'
     # ./lib/awspec/helper/finder/sns_topic.rb:46:in `find_sns_topic'
     # ./lib/awspec/type/sns_topic.rb:16:in `resource_via_client'
     # ./lib/awspec/type/base.rb:26:in `respond_to_missing?'
     # ./spec/type/sns_topic_spec_exception.rb:8:in `block (3 levels) in <top (required)>'

Finished in 0.01059 seconds (files took 3.8 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/type/sns_topic_spec_exception.rb:8 # sns_topic 'arn:aws:sns:us-east-1:123456789:invalid' Issue https://github.com/k1LoW/awspec/issues/445 is still open should not exist

Did some more testing, but this time with the doc example using pry:

[1] pry(main)> require 'aws-sdk-sns'
=> true
[2] pry(main)> 
[3] pry(main)> topic_arn = 'arn:aws:sns:us-east-1:123456789:invalid'
=> "arn:aws:sns:us-east-1:123456789:invalid"
[4] pry(main)> client = Aws::SNS::Client.new(stub_responses: true)
=> #<Aws::SNS::Client>
[5] pry(main)> client.stub_responses(:get_topic_attributes, Aws::SNS::Errors::NotFound)
=> [{:error=>Aws::SNS::Errors::NotFound}]
[6] pry(main)> 
[7] pry(main)> begin
[7] pry(main)*   client.get_topic_attributes({topic_arn: topic_arn})  
[7] pry(main)* rescue Exception => ex  
[7] pry(main)*   puts "Caught #{ex.class} error calling 'get_topic_attributes' on #{topic_arn}"  
[7] pry(main)* end  
Caught ArgumentError error calling 'get_topic_attributes' on arn:aws:sns:us-east-1:123456789:invalid
=> nil
[8] pry(main)> topic_arn = 'arn:aws:sns:us-east-1:123456789:invalid'
=> "arn:aws:sns:us-east-1:123456789:invalid"
[9] pry(main)> client = Aws::SNS::Client.new(stub_responses: true)
=> #<Aws::SNS::Client>
[10] pry(main)> client.stub_responses(:get_topic_attributes, Timeout::Error)
=> [{:error=>Timeout::Error}]
[11] pry(main)> 
[12] pry(main)> begin
[12] pry(main)*   client.get_topic_attributes({topic_arn: topic_arn})  
[12] pry(main)* rescue Exception => ex  
[12] pry(main)*   puts "Caught #{ex.class} error calling 'get_topic_attributes' on #{topic_arn}"  
[12] pry(main)* end  
Caught Timeout::Error error calling 'get_topic_attributes' on arn:aws:sns:us-east-1:123456789:invalid
=> nil
[13] pry(main)> raise Aws::SNS::Errors::NotFound
ArgumentError: wrong number of arguments (given 0, expected 2)
from /home/cin_afreitas/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/aws-sdk-core-3.46.2/lib/aws-sdk-core/errors.rb:16:in `initialize'
[14] pry(main)> 
$ bundle exec ri Aws::SNS::Errors::NotFound
Nothing known about Aws::SNS::Errors::NotFound

And the issue is with the exception itself, not the way we are declaring it on the stub. Badly documented, I went through the code on gems/2.4.0/gems/aws-sdk-core-3.46.2/lib/aws-sdk-core/errors.rb.

require 'thread'

module Aws
  module Errors

    class NonSupportedRubyVersionError < RuntimeError; end

    # The base class for all errors returned by an Amazon Web Service.
    # All ~400 level client errors and ~500 level server errors are raised
    # as service errors.  This indicates it was an error returned from the
    # service and not one generated by the client.
    class ServiceError < RuntimeError

      # @param [Seahorse::Client::RequestContext] context
      # @param [String] message
      def initialize(context, message)
        @code = self.class.code
        @context = context
        super(message)
      end

Change the stub to:

Aws.config[:sns] = {
  stub_responses: {
    get_topic_attributes: Aws::SNS::Errors::NotFound.new('foobar', 'no such topic'),
    list_subscriptions_by_topic: Aws::SNS::Errors::NotFound.new('foobar', 'no such topic')
  }
}

Makes the spec finally to pass the test. Passing as the first argument to Aws::SNS::Errors::NotFound.initialize an instance of Seahorse::Client::RequestContext with default arguments values also works.

Now, if this is a hack or the expected way to make it work, that I'm not able to tell. 😄

[glasswalk3r]

@k1LoW , I just created the PR #547 , which includes the stubs I mentioned previously and that should work when validating exceptions. I did some real integration tests with AWS, trying to fetch invalid SNS topic names, and the behavior is exactly the same with the stubs. Although I'm not sure if it is the best to way to generate exceptions from stubs, it's clear working as expected. If you agree with the PR #547 , I guess we can close this issue as well.

[k1LoW]

👍