schemathesis / schemathesis

Supercharge your API testing, catch bugs, and ensure compliance
https://schemathesis.readthedocs.io
MIT License
2.29k stars 163 forks source link

[BUG] Response validation fails when using authentication and custom session #2409

Closed flacerdk closed 2 months ago

flacerdk commented 2 months ago

Checklist

Describe the bug

Validating responses fails when an endpoint specifies a security parameter and a custom test client is passed as the session argument to case.call_and_validate(), such as in this example from the documentation.

To Reproduce

🚨 Mandatory 🚨: Steps to reproduce the behavior:

Running this test example with pytest should reproduce the issue:

from typing import Annotated
from fastapi import FastAPI, Security
from fastapi.security import APIKeyHeader
from starlette_testclient import TestClient
import schemathesis

app = FastAPI()

@app.get("/", responses={200: {"model": {}}, 403: {"model": {}}})
def root(api_key: Annotated[str, Security(APIKeyHeader(name="x-api-key"))]):
    return {"message": "Hello world"}

schemathesis.experimental.OPEN_API_3_1.enable()

schema = schemathesis.from_asgi("/openapi.json", app)

@schema.parametrize()
def test_api(case):
    client = TestClient(app)
    case.call_and_validate(session=client)

This gives me the following error:

FAILED tests/test_schema.py::test_api[GET /] - requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7c2e41ffa6...

This is the OpenAPI schema:

{
  "openapi": "3.1.0",
  "info": {
    "title": "FastAPI",
    "version": "0.1.0"
  },
  "paths": {
    "/": {
      "get": {
        "summary": "Root",
        "operationId": "root__get",
        "responses": {
          "200": {
            "description": "Successful Response",
            "content": {
              "application/json": {
                "schema": {}
              }
            }
          },
          "403": {
            "description": "Forbidden"
          }
        },
        "security": [
          {
            "APIKeyHeader": []
          }
        ]
      }
    }
  },
  "components": {
    "securitySchemes": {
      "APIKeyHeader": {
        "type": "apiKey",
        "in": "header",
        "name": "x-api-key"
      }
    }
  }
}

Expected behavior

The test given above should pass

Environment

- OS: Linux
- Python version: 3.12
- Schemathesis version: 3.34.1
- Spec version: 3.1.0

Additional context

I believe the issue is in this check: https://github.com/schemathesis/schemathesis/blob/master/src/schemathesis/specs/openapi/checks.py#L351 The check is creating a requests.Session which will lead to an actual HTTP call. It should be using the specified session instead.

b3n4kh commented 2 months ago

We have the same issue with similar code

We mitigated the issue for now by pinning schemathesis to 3.33.3

Stranger6667 commented 2 months ago

Thank you for reporting! Fixed in 3.34.2