servicex-sh / httpx

httpx - CLI to test HTTP/gRPC/RSocket/Kafka... services by HTTP DSL
https://servicex.sh
Apache License 2.0
132 stars 9 forks source link

The assertions are not being verified. #11

Open Marek00Malik opened 2 years ago

Marek00Malik commented 2 years ago

I'm trying to use this tool to run tests outside of my IDE in CICD. So far the called are executed, but the problem I have is that non of my assertions are being tested at all.

Bellow the simplest test for your HTTP example:

### get my ip
//@name get-my-ip
GET https://httpbin.org/ip

> {%
    client.test("Request executed successfully", function() {
        client.log(response.status);
        client.log(response.contentType);
        client.log(response.body);
        client.log("Testing");
        client.assert(response.status == 400, "this should fail");
        client.assert(true, "this should not fail");
    });
%}

Expected result:

Exactly as in IntelliJ, I would imagine the tool to fail/return non zero error code.

Marek00Malik commented 2 years ago

➜ httpx --version 0.33.0

linux-china commented 2 years ago

Now client.assert just calls console.assert(), and code assert following:

    assert(condition, message) {
        console.assert(condition, message);
    }

You mean if assertion failed, and exit httpx and return none zero error code like following:

  assert(condition, message) {
        if(!condition) {
           console.error(message);
           process.exit(1);
        }
    }
Marek00Malik commented 2 years ago

if I would assume to get some error code when the assertion is failing. This is working when running with docker.

linux-china commented 2 years ago

I can add -W for this feature, and it means warning. If assertion failed, and httpx will exit with none zero code.

tino-fernandez commented 2 years ago

thanks @linux-china i'll try that option, but i am wondering, if this is a testing tool to automate checks, should that not be the default behaviour?

linux-china commented 2 years ago

-W not available now, and I will implement this week.

if this is a testing tool to automate checks, should that not be the default behaviour?

Because this behaviour is different with HTTP Client in IDE. But it's ok to add -W option for CLI. :)

mis4s commented 1 year ago

Hi, any update on this? I would love to see this feature for our CI/CD processes