Figure out how to add tagging to API tests

[dsuch]

The underlying library that the API tests use is behave - https://pythonhosted.org/behave/

One of the things behave allows for is tagging and we should start using it.

All the tests should be tagged with an 'enabled' tag and during development if at one point we don't want to run a given test, the tag could be changed to 'disabled' hence letting behave know that we don't want to run it.

In Zato codebase, there is a line containing all the options which is passed into behave directly, it is behave.options in the ./code/apitest/features/config.ini file. This is the place to enable tags in.

[ivaano]

Instead of just using the enabled tag, I think it would be better if we tag each feature by its parent category name (channels, definitions, http-soap,kvdb,outgoing,ping,scheduler,security,server,service,stats) that way we could have more control over what tests to run or what tests to skip.

for example, I've added tags to kvdb and ping test, and now I can run only kvdb specific tests only, by using the --tags=kvdb switch in config.ini :

apitest/features/zato_kvdb_data_dict_dictionary_create.feature  ....
apitest/features/zato_kvdb_data_dict_dictionary_delete.feature  ....
apitest/features/zato_kvdb_data_dict_dictionary_edit.feature  ....
apitest/features/zato_kvdb_remote_command.feature  .............
apitest/features/zato_ping.feature  S
apitest/features/zato_scheduler_job_interval_based.feature  SSSSSS
apitest/features/zato_scheduler_job_one_time.feature  SSSSSSSSSSS
apitest/features/zato_security_basic_auth.feature  SSSSSSSSS
apitest/features/zato_security_tech_account.feature  SSSSSSSSSS
apitest/features/zato_security_wss.feature  SSSSSSSSSSSSSS
4 features passed, 0 failed, 6 skipped
25 scenarios passed, 0 failed, 51 skipped
243 steps passed, 0 failed, 621 skipped, 0 undefined
Took 0m12.273s

or ping tests by specifying the tag--tags=ping:

apitest/features/zato_kvdb_data_dict_dictionary_create.feature  SSSS
apitest/features/zato_kvdb_data_dict_dictionary_delete.feature  SSSS
apitest/features/zato_kvdb_data_dict_dictionary_edit.feature  SSSS
apitest/features/zato_kvdb_remote_command.feature  SSSSSSSSSSSSS
apitest/features/zato_ping.feature  .
apitest/features/zato_scheduler_job_interval_based.feature  SSSSSS
apitest/features/zato_scheduler_job_one_time.feature  SSSSSSSSSSS
apitest/features/zato_security_basic_auth.feature  SSSSSSSSS
apitest/features/zato_security_tech_account.feature  SSSSSSSSSS
apitest/features/zato_security_wss.feature  SSSSSSSSSSSSSS
1 feature passed, 0 failed, 9 skipped
1 scenario passed, 0 failed, 75 skipped
6 steps passed, 0 failed, 858 skipped, 0 undefined
Took 0m0.051s

we also can use --tags=kvdb,ping to run all the scenarios tagged with those tags, and if we want to run all tests we can simply use --tags=-enabled:

apitest/features/zato_kvdb_data_dict_dictionary_create.feature  ....
apitest/features/zato_kvdb_data_dict_dictionary_delete.feature  ....
apitest/features/zato_kvdb_data_dict_dictionary_edit.feature  ....
apitest/features/zato_kvdb_remote_command.feature  .............
apitest/features/zato_ping.feature  .
apitest/features/zato_scheduler_job_interval_based.feature  ......
apitest/features/zato_scheduler_job_one_time.feature  ...........
apitest/features/zato_security_basic_auth.feature  .........
apitest/features/zato_security_tech_account.feature  ..........
apitest/features/zato_security_wss.feature  ......F.......

Failing scenarios:
  apitest/features/zato_security_wss.feature:118  Invoke zato.ping over the previusly created http channel with valid credentials

9 features passed, 1 failed, 0 skipped
75 scenarios passed, 1 failed, 0 skipped
862 steps passed, 1 failed, 1 skipped, 0 undefined
Took 0m47.208s

[dsuch]

Thanks @ivaano - yes, that makes sense!