Could not continue execute other cases after failure case

[hungpk-bap]

I write API test base on the guild runn Grpc server

I have 2 test files:

• login_test.yml (multiple case):

  desc: Login
  vars:
  request:
  - "001-Success-request.json"
  - "002-IncorrectPassword-request.json"
  - "003-IncorrectUsername-request.json"
  response:
  - "001-Success-response.json"
  - "002-IncorrectPassword-response.json"
  - "003-IncorrectUsername-response.json"
  steps:
  - loop:
    count: len(vars.request)
  include:
    path: requestBase.yml
    vars:
      request: "json://../test-data/login/{{ vars.request[i] }}"
      response: "json://../test-data/login/{{ vars.response[i] }}"
      testPath: "/login"
      testCase: "Login { vars.request[i] }}"

• logout_test.yml (multiple case)

1 requestBase.yml file as below:

runners:
  greq:
    addr: ${BASE_GRPC_URL}
    tls: false 
    protos:
      - ../../../*.proto
steps:
  - greq:
      "{{ vars.testPath }}":
        headers:
          test-case: "{{ vars.testCase }}"
        message: "{{ vars.request }}"
    test: compare(current.res.message, vars.response)

main_test.go

func TestAPI(t *testing.T) {
    flag.Parse() // Parse the flags

    err := godotenv.Load(".env")
    if err != nil {
        log.Fatal("Error loading .env file")
    }

    opts := []runn.Option{
        runn.T(t),
        runn.Debug(true),
    }

    runner, err := runn.Load(
        *testPath,
        opts...,
    )

    if err != nil {
        t.Error(err)
    }

    ctx := context.TODO()

    if err := runner.RunN(ctx); err != nil {
        t.Log(err)
    }
}

When I run all test. When case 001 login success, case 002 login failure. It stop (do not run case 003 login, although logout_test still run) I change: t.Fatal(err) -> t.Error(err) purpose continue execute other case if in test_file.yml has failure case. But it does not work. It still stop. Could you share idea or something wrong here? Thank you very much!

[k1LoW]

For runn, login_test.yml is one scenario and logout_test.yml is one scenario. This is because you are running two scenarios. Two scenarios are always run (until the step that fails)

[hungpk-bap]

Hi @k1LoW , Thanks for quickly response. Currently, in the login_test scenario has 3 cases. Expected: when case 002 failure then case 003 continue execute. Actual: when case 002 failure then all case remaining of login_test do not execute. I think cause by: t.Fatal() stop it. So I change t.Fatal() to t.Error() to avoid that but it doesn't seem to work. Forgive my ignorance on this matter. 🙇‍♂️

[k1LoW]

Currently, in the login_test scenario has 3 cases.

In runn, these are only steps, which must be consecutive and successful.

If you want them to be test cases, you need to separate the runbook.

There is an option called force:, but again this is only for forcing all the steps to run, and it does not take into account the cases you mention.

[dungtqbap]

Could you provide an example requestBase for a multi-instance runbook? Suppose you are looping 3 cases in turn Then when the 2nd case reports: test failed on "Generated by runn new".steps[0]: condition is not true We will continue to run the third case If I misunderstand something, please let me know 🙇

[k1LoW]

Separate login_test.yml into three separate files ( not loop ).

[hungpk-bap]

Thanks for support @k1LoW . I got it! When any step failure => The scenario will be failure. So we should split scenarios into sperate files.