AnWeber / vscode-httpyac

Quickly and easily send REST, Soap, GraphQL, GRPC, MQTT and WebSocket requests directly within Visual Studio Code
https://marketplace.visualstudio.com/items?itemName=anweber.vscode-httpyac
MIT License
237 stars 20 forks source link

VSCode http tests are not working the same way as in CLI #180

Closed woolfas closed 1 year ago

woolfas commented 1 year ago

Consider this http file:

### Set user 1
# @name set_user_1
{{
    $global.UserData = {
            id: "1",
            name: "User 1"
        }
}}

//=========================================
### Set user 2
# @name set_user_2
{{
    $global.UserData = {
            id: "2",
            name: "User 2"
        }
}}

//=========================================
### Send User
# @name send_user
POST https://httpbin.org/anything
Content-Type: application/json

{{ $global.UserData }}

//=========================================
### Send User 1
# @forceRef set_user_1
# @forceRef send_user

//=========================================
### Send User 2
# @forceRef set_user_2
# @forceRef send_user

If I execute this file using HttpYac CLI, everything works as expected: all regions are executed one by one from top to bottom. Testing of regions Set user 1 and Set user 2 is skipped and testing for regions Send User, Send User 1 and Send User 2 is invoked and passed successfully.

However if I run tests for the same file in VSCode, regions are executed backwards - from bottom to top. Also testing is done for all regions including Set user 1 and Set user 2 which does not have any requests. Moreover testing fails for Set user 1 and Set user 2 with the error message no response received.

image

I would expect that VSCode tests are executed in the same order as using CLI - from top to bottom. Also testing of Set user 1 and Set user 2 should not fail. I would expect that either they are skipped, or they are executed and passed successfully.

AnWeber commented 1 year ago

Also testing of Set user 1 and Set user 2 should not fail.

All regions with a name or a request are considered executable regions. The difference is that httpyac CLI does not consider a response as an error and VSCode does. I remove the behavior in VSCode.

AnWeber commented 1 year ago

Also testing is done for all regions including Set user 1 and Set user 2 which does not have any requests.

I will not change the behavior. For me regions with name or a request are executable region (not global). Because of that Send User 1 and Send User 2 are also executed. In my opinion http files are not a good location for JSON or javascript files. For this reason I have explicitly implemented the simple import of JSON files and the execution of external Javascript. I would recommend not creating regions just for running javascript and instead outsource those to proper js files.

### Send User 1
{{
  exports.UserData = require("./data.js").user1();
}}
# @forceRef send_user

However if I run tests for the same file in VSCode, regions are executed backwards

I have already implemented a simple fix for the order. However, it does not yet ensure that all regions of a file are executed sequentially when executed in parallel. Since this is mostly done by other TestRunners like jest, I will check this.

AnWeber commented 1 year ago

I have extended the declaration of variables so that $global can also be filled directly with variables from files.

### Send User 2
@global.UserData = {{< ./userData2.json}}
# @forceRef send_user
AnWeber commented 1 year ago

Now it is ensured that during test execution the tests of a file are executed sequentially from top to bottom. Exception is when you execute each one explicitly