Inso-CLI not persisting environment variables, same test suite works in Insomnia App

lcsmobileqa commented 10 months ago

Expected Behavior

If a collection runs successfully through the desktop app, the same collection should succeed when running through the inso-cli tool.

Actual Behavior

I ran the collection with the verbose flag and got this information:

C:\Users\gcovarrubias>inso run test uts_295496 -e QA --verbose
» Data store configured from app data directory at C:\Users\gcovarrubias\AppData\Roaming\Insomnia
» Load api specification with identifier uts_295496 from data store
» Found 0.
» Load workspace with identifier uts_295496 from data store
» Found 0.
» Load unit test suite with identifier uts_295496 from data store
» Found 1.
» Load base environment for the workspace wrk_f01a1c01eb254881997aa5a9c5586501 from data store
» Found 1.
» Load sub environment with identifier QA from data store
» Found 1

  eNum Check
[response tag] No response found
Error rendering template Template render error: (unknown path)
  Error: No responses for request
    at Object._prettifyError (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
    at eval (eval at _compile (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js), <anonymous>:10:11)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
Failed to render element Environment[6].RM12AccessToken {% response 'body', 'req_131d781c39614850ac94743203995f13', 'b64::JC4=::46b', 'never', 60 %}
[response tag] No response found
Error rendering template Template render error: (unknown path)
  Error: No responses for request
    at Object._prettifyError (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
    at eval (eval at _compile (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js), <anonymous>:10:11)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
Failed to render element Environment[6].RM12AccessToken {% response 'body', 'req_131d781c39614850ac94743203995f13', 'b64::JC4=::46b', 'never', 60 %}
[response tag] No response found
Error rendering template Template render error: (unknown path)
  Error: No responses for request
    at Object._prettifyError (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
    at eval (eval at _compile (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js), <anonymous>:10:11)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
Failed to render element Environment[6].RM12AccessToken {% response 'body', 'req_131d781c39614850ac94743203995f13', 'b64::JC4=::46b', 'never', 60 %}
[network] Response succeeded req=req_131d781c39614850ac94743203995f13 status=200
    ✔ RM12 API Authorization (568ms)
[response tag] No response found
Error rendering template Template render error: (unknown path)
  Error: No responses for request
    at Object._prettifyError (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
    at eval (eval at _compile (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js), <anonymous>:10:11)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
Failed to render element Environment[6].RM12AccessToken {% response 'body', 'req_131d781c39614850ac94743203995f13', 'b64::JC4=::46b', 'never', 60 %}
[response tag] No response found
Error rendering template Template render error: (unknown path)
  Error: No responses for request
    at Object._prettifyError (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
    at eval (eval at _compile (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js), <anonymous>:10:11)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
Failed to render element Environment[6].RM12AccessToken {% response 'body', 'req_131d781c39614850ac94743203995f13', 'b64::JC4=::46b', 'never', 60 %}
[response tag] No response found
Error rendering template Template render error: (unknown path)
  Error: No responses for request
    at Object._prettifyError (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
    at eval (eval at _compile (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js), <anonymous>:10:11)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
Failed to render element Environment[6].RM12AccessToken {% response 'body', 'req_131d781c39614850ac94743203995f13', 'b64::JC4=::46b', 'never', 60 %}
[response tag] No response found
Error rendering template Template render error: (unknown path)
  Error: No responses for request
    at Object._prettifyError (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
    at eval (eval at _compile (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js), <anonymous>:10:11)
    at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
Failed to render element headers[0].value {% response 'body', 'req_131d781c39614850ac94743203995f13', 'b64::JC4=::46b', 'never', 60 %}
    1) Count of eNums and values are unchanged and errorcode 3 not returned

  1 passing (644ms)
  1 failing

  1) eNum Check
       Count of eNums and values are unchanged and errorcode 3 not returned:
     Error: No responses for request
      at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
      at C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js
      at RawTask.call (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js)
      at flush (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js)
      at Timeout.handleTimer (C:\snapshot\insomnia\packages\insomnia-send-request\dist\index.js)
      at listOnTimeout (node:internal/timers:564:17)
      at process.processTimers (node:internal/timers:507:7)

Reproduction Steps

In Insomnia Document

Environment - has a variable meant to hold changing information, such as URL and authorization token. Create one if it doesn't already exist.
Environment/variable - Control+Space at either variable and select "Response => Body Attribute"
Environment/variable - Click to open the Edit Tag form and complete as needed. Example as follows: Attribute: Body Attribute - value of response body Request: Select the authorization request Filter: Enter the filter needed and confirm the Live Preview displays the correct targeted response value. This may display and error and you will need to manually run the request first in order to display a response value. Trigger Behavior: When Expired - resend when existing response has expired Click "Done" - make sure the tag is not displayed in red. This means the value from the authorization request is not being captured.
Close Environments tab
Collection - Open any request that will be using the authorization token variable and navigate to where the request will be using the variable. Headers for example.
Request/Headers - Control+Space and select the variable from the drop down. Make sure it is not displayed in red, this means the value pulled from the authorization request is not being captured.
Tests - Should have a set of tests that run the authorization request first, then any subsequent requests that will be using the authorization token generated. Click "Run Tests". All responses should return successful.

In Command terminal

Open command prompt
Type "Inso run test"
Select the test suite from the setup above
Select the environment used for the test run above - inso CLI will not pull the still active authorization token.
Go back to the request in step 6 above and copy/paste the authorization token into the field that held the tag
Go back to the command prompt and run the same tests again - this time the tests pass because the token is hard-coded and put pulling from a variable.

Capture available on request.

Is there an existing issue for this?

[X] I have searched the issue tracker for this problem.

Additional Information

Desktop app runs test suite successfully:

Inso CLI fails same test suite:

Inso CLI version:

It's whichever is installed from the Github on 11/10/2023

Insomnia Version

Version: Insomnia 8.4.5 Build date: 11/23/2023 OS: Windows_NT x64 10.0.22631 Electron: 27.0.3 Node: 18.17.1 Node ABI: 118 V8: 11.8.172.16-electron.0 Architecture: x64

What operating system are you using?

Windows

Operating System Version

Windows 11 Pro, Version 23H2, OS Build 22631.2428

Installation method

download from Insomnia

Last Known Working Insomnia version

N/A

gatzjames commented 4 months ago

Hi @lcsmobileqa I've tested this using the latest Inso and it works for all types of request chaining for me. I tested on Git repositories, exported files and Insomnia data folder and get consistent results.

Can you double-check that you are using the latest version of Inso?

gatzjames commented 4 months ago

I just realized that you are trying to use the environment to store the response. This is not recommended as the environments don't re-evaluate responses and/or send requests. My suggestion is to move from an env variable to using the Response Body tag directly on the requests you want to use it.

Note that we are currently working a new scripting capability that will allow manipulating the environment based on the response which is probably coming on the next release.

lcsmobileqa commented 4 months ago

Hi Gatzjames, thank you for your response, I really appreciate you taking the time to look into this! I tried updating the authorization tag to the response body directly in each header, but I'm still running into the same issue where it fails for inso-cli, but not the desktop. Something new that's happening is that one of my requests is running out of order--at the top of the run, instead of where it's listed in the test list. But back to what I'm seeing as far as what's reported. Here is a screenshot of the environment edited to remove that authorization token--so when I add the response body tag to the request headers, there's no confusion about where it should be pulling the value from:

Screenshot of one of the request headers showing the response body tag:

Then the setup for the response body tag:

Screenshot of the test passing in insomnia desktop:

And screenshot of the same collection run failing in inso-cli:

Not sure if this matters, but I did close and reopen insomnia desktop after making the above changes, then running it again in inso-cli, just in case that was needed for the changes to show up. I'm not sure what I'm doing differently from your setup to get this running correctly, any ideas? Anything else I should/could check on my end? Thank you again! You're a wonderful human being!

gatzjames commented 3 months ago

Happy to help and thanks for the kind feedback ☺️

A couple of things that will help debug this more:

Can you run inso in verbose mode? inso run test --verbose
- Here you should see your dependent request being run before the test.
It looks like you are using a plugin to store some values. If that's the case the diff between inso/insomnia is that Inso doesn't install/use any plugins you might have installed in Insomnia.

Kong / insomnia