extract_binds with imports

[complexanomaly]

I'm trying to use a variable extracted from the headers through my tests. My main .yaml file is simply:

---
- config:
    - testset: "App tests"

- import: test/auth.yaml
- import: test/items.yaml

Both imported files use the same test set, and the lifecycle docs say:

Parse YAML, reading top-level imports and building TestSets

and also:

Generate a Context for each test set, populated with generators and variables defined in the TestConfig

And:

Contexts are persistent within a TestSet. Once a variable is set, it can be used in all following tests

But a variable declared in the first import isn't available in the second. I've moved the tests from the second import into the first import and it all works perfectly, so the variable is definitely being set correctly.

Example from auth.yaml:

---
- config:
    - testset: "App tests"

- test:
    - group: "Authentication"
    - name: "User successful login"
    - url: "/api/v0.1/auth/"
    - method: "POST"
    - body: '{"username": "xxxx", "password": "xxxx", "pin": xxxx }'
    - headers: {"Content-Type": "application/json"}
    - expected_status: [200]
    - validators:
      - json_schema: {schema: {file: "auth-schema.json"}}
    - extract_binds:
      - "auth_key": {"jsonpath_mini": "token"}

And then items.yaml:

---
- config:
    - testset: "App tests"

- test:
    - group: "Items"
    - name: "User item listing successful"
    - url: "/api/v0.1/items/?limit=10"
    - method: "GET"
    - headers: { template: {"Api-Key": "$auth_key", "Content-Type": "application/json"}}
    - expected_status: [200]

Do I need a config section in each imported file? Adding or removing it seems to make no difference to how the tests run.

The item listing call using verbose mode shows:

Api-Key:$auth_key

[svanoort]

Hi @tentoseven - thank you for submitting your question. Unfortunately, you're the first to be hit by a complexity of how the imports work vs. the contexts:

From the docs:

import: import another test set file so you Don't Repeat Yourself

So, every import you run creates a new test set to be run (after the current one). Each TestSet is intended to run as a separate test scenario, so it gets a fresh Context. Thus, while you can use imports to construct scenarios by writing to the database, the variables they write to the Context are lost after each run (as you've observed).

I'm aware that this behavior limits the ability to use imports to create setup/teardown scenarios, and have plans on the future roadmap to fix this: https://github.com/svanoort/pyresttest/issues/146 It may be reasonable as well to provide an override option for this behavior in the future as well (something like reuse_context_between_testsets?). The latter is something that could be done easily enough in the next release. The former depends on architectural changes that I'm incorporating with the next 2 releases.

For now, my recommendation is to do exactly as you are doing and include setup logic in each testset (even though this may cause some duplication).

Does that answer your question and give some context?

[complexanomaly]

Hi @svanoort,

It does indeed. At the very least, I can stop trying to make it work now! Duplicating the setup logic in this case isn't too much of an overhead, thankfully.

Great job overall though, thanks.

[svanoort]

@tentoseven Thanks, and I've added an update to the README to make this clearer so others do not get tripped up by it.

[andyr]

Just for clarity on this issue (and the following statement in the docs):

They may also be used to create sample data or perform cleanup as long as you don't rely on variables to store this information.

This means sample data or cleanup if that stuff is stored in a database/persistence layer right?

I am trying to refactor a whole suite of tests with a LOT of duplication. One of the simplest use cases is that I want to create a yaml file to store constants (api key, etc) that are used in every file (over 100). (Sounds like I'm trying to accomplish the setup/teardown issue that is tracked in issue 146.)

[svanoort]

@andyr

This means sample data or cleanup if that stuff is stored in a database/persistence layer right?

It means that if you run a set of API calls to set up data (test users, test user data, mock records) for your test scenario or remove results afterward, you can create an import for that. Similarly you can create a testset import to run API actions that delete the test data afterward (and test this worked correctly).

You just can't rely on the PyRestTest contexts to store variables/generators between testsets, because each one gets a fresh context.

One of the simplest use cases is that I want to create a yaml file to store constants (api key, etc) that are used in every file (over 100).

There's two ways to handle this currently. The most succinct one is to pass them as command-line arguments (it takes an inline YAML argument 'vars'). The feature is fairly new and AFAIK hasn't been used yet, so it may have issues, but it's worth trying. In this case, you'd run pyresttest from a shellscript that builds up this argument (possibly creating a shell variable, i.e. $PYRRESTEST_VARS that is used to store it).

Option 2 is a bit uglier - use environment variables to store individual values, and set them by a shellscipt before testing. You define named env_variable generators in the testconfigs, and then do a generator_bind element in each test that needs them. This involves a lot of ugly boilerplate though.

Added https://github.com/svanoort/pyresttest/issues/178 to enable imports to better aid this. You're the second user to request it, and now I have a plan. Dynamic scoping #79 and make-everything-dynamic #101 are probably the issues that will assist with this too.