[Feature Request] Ability to run sippy_cup from minitest

[leifmadsen]

I started looking into this a bit (only a few minutes), and the documentation examples show how to embed this into another Ruby process, but it looks like it is mostly just for creating the scenario file.

It would be amazing if I could embed this into minitest on my Chef runs (via minitest-chef-handler). Each time I ran my call server cookbooks, I could perform REGISTER tests, load testing, pretty much anything I would want.

[leifmadsen]

FYI: https://github.com/calavera/minitest-chef-handler

[benlangfeld]

There's a bunch of changes to this stuff coming up in 0.3.0 to make it easier to do exactly this kind of thing. It'll come with documentation and won't be too far off. Stay tuned.

[leifmadsen]

Fantastic news!

[leifmadsen]

As @bklang is at AstriCon this week with me, I'm going to bug him to show me some of the things in 0.3 that might make this more possible :)

[leifmadsen]

Some basic documentation that I figured out this week. Likely needs some cleanup. Also, anyone who is better with Ruby than I am can likely clean this up a bit too. Just here for future reference when someone (or I) comes back to clean this up and get a patch set.

This is done via minitest-chef-handler and is in the /files/tests/default_test.rb

require 'minitest/spec' 
require 'sippy_cup'

register_test = SippyCup::Scenario.new 'register', source: 'localhost:10001', destination: 'localhost:5060', filename: '/tmp/register' do |s|
  s.register 'sipp' 
  s.receive_answer
end

register_test.compile!

describe_recipe "tlc-call-server::default" do
  describe "automated call tests" do
    include MiniTest::Chef::Assertions
    include MiniTest::Chef::Context
    include MiniTest::Chef::Resources

    def sipp(options = {})
      options[:destination]     ||= "127.0.0.1:5060"
      options[:max_concurrent]  ||= 1
      options[:cps]             ||= 1
      options[:num_calls]       ||= 1
      options[:listen_port]     ||= 10001

      return "sipp #{options[:destination]} -sf /tmp/#{options[:scenario]}.xml -p #{options[:listen_port]} -m #{options[:num_calls]} -l #{options[:max_concurrent]} -r #{options[:cps]} -nostdin >& /dev/null"
    end

    it "can register a device" do
      result = assert_sh sipp(:scenario => 'register')
    end
  end
end

[benlangfeld]

So part of the point of SippyCup's Ruby API is that you could do this without knowing about SIPp's command line API and without explicitly shelling out, simply using SippyCup::Runner. You would also not need to dump to the filesystem in the way you are doing; passing a Scenario to Runner stores it temporarily on the filesystem and ensures cleanup, as well as returning some basic statistics (which I hope to expand upon next time @bklang gives me some time to work on SippyCup).

One day I'll get around to demonstrating that, but glad you have something working :)

[leifmadsen]

@benlangfeld oooooh! well that makes perfect sense.

Any examples of how to do that would be epic, and would make this whole thing a LOT cleaner. I knew it was dirty when I was doing it, but I wanted to mostly get something work on Friday while all the 'murcans were out to Turkey Day :)

[leifmadsen]

Well crap. I figured out how to do this with SippyCup::Runner, however, it has the same issue I had to work around with the shell command: it freezes up on minitest execution.

recipe::tlc-call-server::default::automated call tests#test_0001_can register a device = I, [2013-12-02T10:50:02.924546 #18995]  INFO -- : Preparing to run SIPp command: sudo sipp -i localhost:10001 -p 8836 -sf /tmp/scenario20131202-18995-122jofq -l 1 -m 1 -r 1 -s 1 localhost:5060
Warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE = 1024Resolving remote host 'localhost'...Done.

It is technically working, but the way in which the test is executed within the shell is causing errors. This is why I had to wrap the call in the shell the way I did. Luckily the work I did in my previous comment could be applied to the Runner functionality to cause this to work.

My new test would look like this:

require 'minitest/spec'
require 'sippy_cup'

register_test = SippyCup::Scenario.new 'register', source: 'localhost:10001', destination: 'localhost:5060', max_concurrent: 1, number_of_calls: 1, calls_per_second: 1, filename: '/tmp/register' do |s| 
  s.register 'sipp'
  s.receive_answer
end

register_test.compile!

describe_recipe "tlc-call-server::default" do
  describe "automated call tests" do
    include MiniTest::Chef::Assertions
    include MiniTest::Chef::Context
    include MiniTest::Chef::Resources

    it "can register a device" do
      runner = SippyCup::Runner.new register_test
      runner.run
    end 
  end 
end

However, the runner.rb file needs to be slightly modified in these scenarios to not output the text to the console (which I believe is what is causing the real issue; some pseudo shell that hates the actual output from SIPp).

-nostdin >& /dev/null

The workaround I tried was the following:

destination: 'localhost:5060 -nostdin >& /dev/null'

With output example of:

recipe::tlc-call-server::default::automated call tests#test_0001_can register a device = I, [2013-12-02T11:06:17.861175 #28457]  INFO -- : Preparing to run SIPp command: sudo sipp -i localhost:10001 -p 8836 -sf /tmp/scenario20131202-28457-tucj3d -l 1 -m 1 -r 1 -s 1 localhost:5060 -nostdin >& /dev/null
I, [2013-12-02T11:06:18.891127 #28457]  INFO -- : Test completed successfully!
1.03 s = .

[leifmadsen]

FYI, this is my latest working example that will be going into our repo:

require 'minitest/spec'
require 'sippy_cup'

register_test = SippyCup::Scenario.new 'register', source: 'localhost:10001', destination: 'localhost:5060 -nostdin >& /dev/null', max_concurrent: 1, number_of_calls: 1, calls_per_second: 1, filename: '/tmp/register' do |s| 
  s.register 'sipp'
  s.receive_answer
end

register_test.compile!

describe_recipe "tlc-call-server::default" do
  describe "automated call tests" do
    include MiniTest::Chef::Assertions
    include MiniTest::Chef::Context
    include MiniTest::Chef::Resources

    it "can register a device" do
      runner = SippyCup::Runner.new register_test
      assert_sh runner.run
    end 
  end 
end

[benlangfeld]

Can you try this without the file name option and without calling #compile! ?

Em 2 Dec 2013, às 13:10, Leif Madsen notifications@github.com escreveu:

FYI, this is my latest working example that will be going into our repo:

require 'minitest/spec' require 'sippy_cup'

register_test = SippyCup::Scenario.new 'register', source: 'localhost:10001', destination: 'localhost:5060 -nostdin >& /dev/null', max_concurrent: 1, number_of_calls: 1, calls_per_second: 1, filename: '/tmp/register' do |s| s.register 'sipp' s.receive_answer end

register_test.compile!

describe_recipe "tlc-call-server::default" do describe "automated call tests" do include MiniTest::Chef::Assertions include MiniTest::Chef::Context include MiniTest::Chef::Resources

it "can register a device" do
  runner = SippyCup::Runner.new register_test
  assert_sh runner.run
end 

end end — Reply to this email directly or view it on GitHub.

[benlangfeld]

Also @leifmadsen, your last example shouldn't work at all. Are you sure it does?

[leifmadsen]

@benlangfeld I tested it in Vagrant and it passed when a peer existed to register against, and failed when it didn't exist. Can you point out what part you don't think should be working?

[benlangfeld]

assert_sh takes an argument which should be a shell command to execute. SippyCup::Runner#run returns true or false depending on wether execution was successful or not.

[leifmadsen]

I think in the end assert_sh is just curious about a true/false return value, because normally I just pipe things into {{test}} which returns true/false.

[benlangfeld]

If that works without any of the ulimit problems from earlier, you should be able to do runner.run.must_be_true and skip the complexity of assert_sh :)