melezhik / swat

Simple Web Application Test
48 stars 12 forks source link
api smoke-test tdd testing web

NAME

Swat

SYNOPSIS

Web testinfg framework consuming Outthentic::DSL.

What does swat stand for?

[S]imple
[W]eb
[A]application
[T]est

Description

Ok, now I hope you are ready to dive into swat tutorial! :)

Install

$ sudo apt-get install curl
$ sudo cpanm swat

Or install from source:

# could be useful for contributors and developers

$ perl Makefile.PL
$ make
$ make test
$ make install

Write your swat story

Swat test stories always answers on 2 type of questions:

As swat is a web test oriented tool it deals with some http related stuff as:

Swat leverages Unix file system to build an analogy for these things:

HTTP resources

HTTP resource is just a directory. You have to create a directory to define a http resource:

$ mkdir foo/
$ mkdir -p bar/baz

This code defines two http resources for your application - '/foo/' and '/bar/baz'

HTTP methods

HTTP method is just a file. You have to create a file to define a http method.

$ touch foo/get.txt
$ touch foo/put.txt
$ touch bar/baz/post.txt

Obviously `http methods' files should be located at `http resource' directories.

The list below describes two http resources ( /foo, /bar/baz ) and three http methods for these resources ( GET, PUT, DELETE ):

* GET  /foo
* PUT  /foo
* POST /bar/baz

Here is the list of predefined file names for a http methods files:

get.txt      --> GET      method
post.txt     --> POST     method
put.txt      --> PUT      method
delete.txt   --> DELETE   method

There is also special predefined file name `meta.txt', see meta stories.

Hostname / IP Address

You need to define hostname or ip address to send request to.

Just write it up to a special file called `host' and swat will use it.

$ echo 'app.local' > host

As swat makes http requests with the help of curl, the host name should be compliant with curl requirements, this for example means you may define a http schema or port here:

$ echo 'https://app.local' > host

$ echo 'app.local:8080' > host

HTTP Response

Swat makes request to a given http resources with a given http methods and then validates a response. Swat does this with the help so called check lists, check lists are defined at `http methods' files.

Check list is just a list of expressions a response should match. It might be a set of plain strings or regular expressions:

$ echo 200 OK > foo/get.txt
$ echo 'Hello I am foo' >> foo/get.txt

The code above defines two checks for response from `GET /foo':

You may add some regular expressions checks as well:

# for example check if we got something like 'date':
$ echo 'regexp: \d\d\d\d-\d\d-\d\d' >> foo/get.txt

Bringing all together

All these things like http method, http resource and check list build up an essential swat entity called a swat story.

Swat story is a very simple test plan, which could be expressed in a cucumber language as follows:

Given I have web application 'http://my.cool.app:80'
And I have http method 'GET'
And make http request 'GET /foo'
Then I should have response matches '200 OK'
And I should have response matches 'Hello I am foo'
And I should have response matches '\d\d\d\d-\d\d-\d\d'

From the file system point of view swat story is a:

Swat Project

Swat project is a bunch of a related swat stories kept under a single directory. This directory is called project root directory. The project root contains swat scenarios or stories to be parsed and executed by swat utility. See swat runner section for full explanation of this process.

This is an example swat project layout:

$ tree my/swat/project
  my/swat/project
  |--- host
  |--- get.txt
  |----foo
  |-----|----bar
  |           |---- post.txt
  |--- baz
        |--- get.txt

3 directories, 3 files

This project contains 3 swat stories:

Default swat story is story executed swat by default, this is $project_root/GET:

$ swat # will execute GET / story

You may set default story using -t option:

$ swat -t foo/bar/POST

Or by setting path variable inside swat.ini file:

$ echo path=baz/POST >> swat.ini

You define hostname to run request against by command line argument:

$ swat $project_root $hostname

For example:

$ swat . 127.0.0.1

Or by setting via host file:

$ echo 'app.local:8080' > host

If you don't defined project root explicitly, swat assume this as current working directory.

$ swat # run `GET /' story inside current working directory

Follow swat client section for full explanation of swat client command line API.

Swat check lists

Swat check lists complies Outthentic DSL format.

There are lot of possibilities here!

( For full explanation of outthentic DSL please follow documentation. )

A few examples:

Often all you need is to ensure that http response has some strings in:

# http response
200 OK
HELLO
HELLO WORLD

# check list
200 OK
HELLO

# swat output
OK - output matches '200 OK'
OK - output matches 'HELLO'

You may use regular expressions as well:

# http response
My birth day is: 1977-04-16

# check list
regexp: \d\d\d\d-\d\d-\d\d

# swat output
OK - output matches /\d\d\d\d-\d\d-\d\d/

Follow https://github.com/melezhik/outthentic-dsl#check-expressions to know more.

Yes you may generate new check list on run time:

# original check list

Say
HELLO

# this generator creates 3 new check expressions:

generator: [ qw{ say hello again } ]

# final check list:

Say
HELLO
say
hello
again

Follow https://github.com/melezhik/outthentic-dsl#generators to know more.

What about inline arbitrary perl code? Well, it's easy!

# check list
regexp: number: (\d+)
validator: [ ( capture()->[0] '>=' 0 ), 'got none zero number') ];

Follow https://github.com/melezhik/outthentic-dsl#validators to know more.

Need to validate that some lines goes in response successively ?

    # http response

    this string followed by
    that string followed by
    another one string
    with that string
    at the very end.

    # check list
    # this text block
    # consists of 5 strings
    # goes consequentially
    # line by line:

    begin:
        # plain strings
        this string followed by
        that string followed by
        another one string
        # regexps patterns:
    regexp: with (this|that) \S+
        # and the last one in a block
        at the very end
    end:

Follow https://github.com/melezhik/outthentic-dsl#comments-blank-lines-and-text-blocks to know more.

Swat ini files

Every swat story comes with some settings you may define to adjust swat behavior. These type of settings could be defined at swat ini files.

Swat ini files are file called "swat.ini" and located at `resources' directory:

 foo/bar/get.txt
 foo/bar/swat.ini

The content of swat ini file is the list of variables definitions in bash format:

$name=value

As swat ini files is bash scripts you may use bash expressions here:

if [ some condition ]; then
    $name=value
fi

Following is the list of swat variables you may define at swat ini files, it could be divided on two groups:

swat variables

Swat variables define swat basic configuration, like default story, debug mode, etc. Here is the list:

Example:

# define default story as GET => foo/
path=foo/GET    

For example:

# swat.ini
# assume that we set profile variable somewhere else
# 

if test "${profile}" = 'production'; then
    skip_story=1 # we don't want this one for production
fi

curl parameters

Curl parameters relates to curl client. Here is the list:

Here are some examples:

# -d curl parameter
curl_params='-d name=daniel -d skill=lousy' # post data sending via form submit.

# --data-binary curl parameter
curl_params=`echo -E "--data-binary '{\"name\":\"alex\",\"last_name\":\"melezhik\"}'"`

# set http header
curl_params="-H 'Content-Type: application/json'"

Follow curl documentation to get more examples.

other variables

This is the list of helpful variables you may use in swat ini files:

Alternative swat ini files locations

Swat try to find swat ini files at these locations ( listed in order )

Settings priority table

This table describes all possible locations for swat ini files. Swat try to find swat ini files in order:

| location                                  | order N     |
| --------------------------------------------------------|
| ~/swat.ini                                | 1           |
| `project_root_directory'/swat.ini         | 2           |
| `http resource' directory/swat.ini file   | 3           |
| environment variables                     | 4           |

In case the same variable is defined more than once at swat ini files with different locations, the file loaded last win:

curl_params="-H 'Foo: Bar'" # in a ~/swat.ini
curl_params="-H 'Bar: Baz'" # in a project_root_directory/swat.ini

# actual curl_params value:
"-H 'Bar: Baz'"

If you want concatenation mode use name="$name value" expression:

curl_params="-H 'Foo: Bar'" # in a ~/swat.ini
curl_params="$curl_params -H 'Bar: Baz'" # in a project_root_directory/swat.ini

# actual curl_params value:
"-H 'Foo: Bar' -H 'Bar: Baz'"

In case you need provide default value for some variable use name=${name default_value} expression:

# port will be set 80 unless it's not set somewhere else
port=${port:=80} # in a ~/swat.ini

Hooks

Hooks are extension points to hack into swat runtime phase. It's just files with perl code gets executed in the beginning of swat story. You should named your hook file as `hook.pm' and place it into `resource' directory:

# foo/hook.pm
diag "hello, I am swat hook";
sub red_green_blue_generator { [ qw /red green blue/ ] }

# foo/get.txt
generator: red_green_blue_generator()

There are lot of reasons why you might need a hooks. To say a few:

Hooks API

Swat hooks API provides several functions to change swat story at runtime

Redefine http responses

set_response(STRING)

Using set_response means that you never make a real request to a web application, but instead set response on your own side.

This feature is helpful when you need to mock up http responses instead of having them requested from a real web application. For example in absence of an access to a tested application or if response is too slow or it involves too much data which make it hard to execute a swat stories often.

This is an example of setting server response inside swat hook:

# hook.pm
set_response("THIS IS I FAKE RESPONSE\n HELLO WORLD");

# get.txt
THIS IS A FAKE RESPONSE
HELLO WORLD

You may call `set_response' more then once:

set_response("HELLO WORLD");
set_response("HELLO WORLD2");

A final response will be:

HELLO WORLD
HELLO WORLD2

Another interesting idea about set_response feature is a conditional http requests.

Let say we have `POST /login' request for user authentication, this is a simple swat story for it:

# login/post.txt
200 OK

Good. But what if you need to skip authentication under some conditions, like if you are already logged in before? We could write such a code:

# login/post.txt
generator:
$logged_in ? [ 'I am already logged in' : '200 OK' ]

# login/hook.pm
if ( ... check if user is logged in .... ){
    set_response('I am already logged in');
}

Process http responses

set_response_processor(CODEREF)

Response processors are custom perl function to modify content returned from server before invoking a validation process. Processor code should be defined by calling a process_response function with parameter as reference to processor function:

For example:

   set_response_processor( sub { 
      my $headers   = shift; # original response, http headers, String
      my $body      = shift; # original response, body, String
      $body=~s/hello/swat/;
      return $body;          # modified response will be return value of sub {} 
    });

What you should know about processor functions:

This is a rough schema of full process:

 # without processor function
 curl -i some-http-URL | validation-process

 # with processor script exists:
 curl -i some-http-URL | processor-function| validation-process

The possible usage of processor functions:

For example:

  # server response in json format
  {
      "Foo": {
          "Bar" : "Baz"
       }
  }

  # processor function
  my $headers   = shift;
  my $body      = shift;
  use JSON;
  $hash = decode_json($body);
  return 'Foo.Bar.Baz :'.( $hash->{Foo}->{Bar}->{Baz} )."\n";

  # server response in xml format
  <foo>
    <bar>
      <baz>aaa</baz>
    </bar>  
  </foo>

  # processor function
  my $headers   = shift;
  my $body      = shift;
  use XML:LibXML;
  my $doc = XML::LibXML->parse_string($body);
  return 'Foo.Bar.Baz :'.( $doc->find("string(/foo/bar/baz)")->string_value )."\n";

Redefine http resources

modify_resource(CODEREF)

To modify existed resource use modify_resource function:

# foo/bar/baz/ - resource

# hook.pm
modify_resource( sub { my $resource = shift; s/bar/bbaarr/, s/baz/bbaazz/ for $resource; $resource  } );

# modified resource
foo/bbaarr/bbaazz

Upstream and downstream stories

Swat allow you to call one story from another, using notion of swat modules.

Swat modules are reusable swat stories. Swat never executes swat modules directly, instead you have to call swat module from your swat story. Story calling another story is named a upstream story, story is being called is named a downstream story. ( This kind of analogy is taken from Jenkins CI )

Let show how this work on a previous `login' example. We need to ensure that user is logged in before doing some other action, like checking email list:

# email/list/get.txt
200 OK
email list

# email/list/hook.pm
run_swat_module( POST => '/login', { user => 'alex', password => 'swat' } )  

# and finally this is
# login/post.txt
200 OK

# login/swat.ini
swat_module=1 # this story is a swat module
curl_params="-d 'user=%user%' -d 'password=%password%'"

Here are the brief comments to the example above:

Here is an example code snippet:

# hook.pm
run_swat_module( GET => '/foo/' )
run_swat_module( POST => '/foo/bar' )
run_swat_module( GET => '/foo/' )

Use hash passed as third parameter of run_swat_module function:

run_swat_module( GET => '/foo', { var1 => 'value1', var2 => 'value2', var3=>'value3'   }  )

Swat interpolates module variables into `curl_params' variable in swat module story:

# swat module
# swat.ini
swat_module=1
# initial value of curl_params variable:
curl_params='-d var1=%var1% -d var2=%var2% -d var3=%var3%'

# real value of curl_params variable
# during execution of swat module:
curl_param='-d var1=value1 -d var2=value2 -d var3=value3'

Use %[\w\d_]+% placeholders in a curl_params variable to insert module variables here

Access to a module variables is provided by `module_variable' function:

# hook.pm
module_variable('var1');
module_variable('var2');

One word about sharing state between upstream story and swat modules. As swat modules get executed in the same process as upstream story there is no magic about sharing data between upstream and downstream stories. The straightforward way to share state is to use global variables :

# upstream story hook:
our $state = [ 'this is upstream story' ]

# downstream story hook:
push our @$state, 'I was here'

Of course more proper approaches for state sharing could be used as singeltones or something else.

swat variables accessors

There are some accessors to a common swat variables:

See test suite ini file section for details.

Be aware of that these functions are readers.

meta stories

Meta stories are special type of swat stories.

The essential property of meta story is it has no related http request:

# foo/bar story
mkdir foo/bar

# it's a meta story
touch foo/bar/meta.txt

As meta story does not have any related http request, it has no `http method' file either.

A `meta.txt' file should be treated as meta request file denoted that story is meta.

You may live `meta.txt' empty file or add some useful description to be printed when story is executed:

nano foo/bar/meta.txt

    This is my cool story. 
    Take a look at this!

To run meta story you define story as path/META, for example:

swat -t META # run meta story defined in the project root directory

swat -t foo/META # run meta story defined in the foo/ folder

echo META > swat.ini # the same as the first but via swat.ini file

How one could use meta stories?

Meta stories are just containers for other downstream stories. Usually one defines some downstream stories call inside meta story's hook file:

nano foo/bar/hook.pm

    run_swat_module( POST => '/baz' );
    run_swat_module( POST => '/baz/bar' );

Meta stories are very close to upstream stories with spoofed server response, with the only exclusion that as meta story has no real http request related to it, there is no need for spoofing.

Meta stories can be also called as downstream stories:

# I am downstream story
# you can call me from somewhere else
nano foo/bar/swat.ini
    swat_module=1

PERL5LIB

Swat adds `project_root_directory/lib' path to PERL5LIB path, which make it easy to add some modules and use them:

# my-app/lib/Foo/Bar/Baz.pm
package Foo::Bar::Baz;
...

# hook.pm
use Foo::Bar::Baz;
...

Swat runner

Swat runner - is a script to run swat stories. It is called swat.

Runner consequentially goes several phases:

A compilation phase.

Stories are converted into perl test files *.swat ( compilation phase ) and saved into temporary directory.

An execution phase.

TODO:

Suite configuration

Swat test suites could be configurable. Configuration files contain a supplemental data to adjust suite behavior

There are two type of configuration files are supported:

.Ini style configuration files are passed by `--ini' parameter

$ swat --ini /etc/suites/foo.ini

$ cat /etc/suites/foo.ini

[main]

foo = 1
bar = 2

There is no special magic behind ini files, except this should be Config Tiny compliant configuration file.

Or you can choose YAML format for suite configuration by using --yaml parameter:

$ swat --yaml /etc/suites/foo.yaml

$ cat /etc/suites/foo.yaml

main:
  foo : 1
  bar : 2

Unless user sets path to configuration file explicitly by --ini or --yaml swat runner looks for the files named suite.ini and then ( if not found ) looks for the file named suite.yaml at the current working directory.

If configuration file is passed and read a related configuration data is accessible via config() function, for example in story.pm file:

# cat story.pm

my $foo = config()->{main}{foo};
my $bar = config()->{main}{bar};

Misc settings

Swat client

Once swat is installed you get swat client at the `PATH':

swat [[project_root_dir] [host:port]] [swat_command_line_parameters]

In case you don't set one, swat assume it equal to current working directory. Examples:

# setup project root directory explicitly
swat /foo/bar/baz

# project root directory is CWD
swat 

In case host parameter is missing , swat tries to read it up from `host' file. Examples:

# setup host explicitly
swat /foo/bar/baz 127.0.0.1

# host entry gets read from CWD/host file 
swat /foo/bar/baz

# project root directory is CWD
# host entry gets read from CWD/host file 
swat

List of swat command line parameters:

Sets a certain swat story to executed, see Running none default swat story

See suite configuration section for details.

Override the value for swat debug variable, see swat variables section:

swat --debug 2 # set debug to 2

Running none default swat story

Use `-t' options to execute specific swat story, you should use $http_resource/$http_method syntax, for example:

# run `POST foo/bar' story
swat example/my-app 127.0.0.1 -t foo/bar/POST

# run `META /' story
swat example/my-app 127.0.0.1 -t META

Examples

There is plenty of examples at ./examples directory

AUTHOR

Aleksei Melezhik

See also

Swat version 0.2.0 BREAKING CHANGES

As with version 0.2.0 swat removes the usage of prove ( Test::More, Test::Harness modules ), there are consequences of that:

Thanks

To God as the One Who inspires me to do my job!