Figure out pattern for recording authenticated API calls

[simonw]

The pattern I need to figure out for many of the tests I write is how to safely use this with an authenticated API endpoint. Basically I want to:

• Have code that talks to a protected API, e.g. the OpenAI one
• Ensure that the cassette recordings redact the Authorization: Bearer ... key or similar
• Ensure that pytest uses just the cassettes, but I know how to recreate the cassettes when I want to change something
• Figure out the most responsible way to make an API key available to the code that generates the cassettes, without having it in the code

Originally posted by @simonw in https://github.com/simonw/pytest-recording-example/issues/1#issuecomment-2019315790

[simonw]

I'll try this out with Claude, because it has slightly unconventional HTTP headers (all required):

curl https://api.anthropic.com/v1/messages \
     --header "x-api-key: $ANTHROPIC_API_KEY" \
     --header "anthropic-version: 2023-06-01" \
     --header "content-type: application/json" \
     --data \
'{
    "model": "claude-3-haiku-20240307",
    "max_tokens": 50,
    "messages": [
        {"role": "user", "content": "Haiku about a pelican"}
    ]
}'

[simonw]

I added this test to test_httpx.py:

@pytest.mark.vcr()
def test_httpx_anthropic_claude():
    url = "https://api.anthropic.com/v1/messages"
    headers = {
        "x-api-key": os.environ.get("ANTHROPIC_API_KEY") or "mock-api-key",
        "anthropic-version": "2023-06-01",
        "content-type": "application/json",
    }
    data = {
        "model": "claude-3-haiku-20240307",
        "max_tokens": 50,
        "messages": [{"role": "user", "content": "Haiku about a pelican"}],
    }

    with httpx.Client() as client:
        response = client.post(url, json=data, headers=headers)
        assert response.status_code == 200

        content = response.json()
        assert content["content"][0]["text"] == (
            "Here is a haiku about a pelican:\n\n"
            "Graceful wings unfurl,\n"
            "Diving for its fishy feast,\n"
            "Pelican soars high."
        )

I ran it once with:

pytest --record-mode new_episodes

Which failed because I didn't have the right assertion in place - but did capture the request in a cassette. I edited the test to match the correct Haiku and now pytest passes.

But.. the cassette it generated included my API key:

      host:
      - api.anthropic.com
      user-agent:
      - python-httpx/0.27.0
      x-api-key:
      - sk-ant-api...

Need to figure out redaction.

[simonw]

OK, this works. I have to set the ANTHROPIC_API_KEY environment variable before I run the pytest --record-mode new_episodes (or all) command, otherwise the cassette is recorded with an authentication denied page.

It correctly redacts the API key thanks to this:

https://github.com/simonw/pytest-recording-example/blob/236cf7bc05c24e1ef0399bc019fa2ee2e9ef1d55/test_httpx.py#L6-L8

Hence no API key here: https://github.com/simonw/pytest-recording-example/blob/236cf7bc05c24e1ef0399bc019fa2ee2e9ef1d55/cassettes/test_httpx/test_httpx_anthropic_claude.yaml#L5-L21

[simonw]

I think the easiest way to re-record a cassette is to delete the file and then run this:

ANTHROPIC_API_KEY=... pytest --record-mode once