Endava / cats

CATS is a REST API Fuzzer and negative testing tool for OpenAPI endpoints. CATS automatically generates, runs and reports tests with minimum configuration and no coding effort. Tests are self-healing and do not require maintenance.
Apache License 2.0
1.17k stars 74 forks source link

Response body is not being validated by OpenAPI schema #123

Closed AbdulinRuslan closed 4 months ago

AbdulinRuslan commented 5 months ago

Describe the bug No errors or warnings appear in case of difference between actual response body and response body described in OpenAPI contract.

To Reproduce Steps to reproduce the behaviour:

  1. Run cats --contract=fastapi.json --server=http://127.0.0.1:8000 --httpMethods=GET --fuzzers=HappyPath --urlParams="user_id:2945"
  2. Using which OpenAPI file fastapi.json
  3. Here's the simple service I'm using:

from fastapi import FastAPI, HTTPException, Request, status from pydantic import BaseModel, Field import json import psycopg2 from psycopg2.extras import RealDictCursor from psycopg2 import connect

app = FastAPI()

conn = connect(dbname='mydatabase', user='myuser', password='mypassword', host='localhost')

class User(BaseModel): name: str = Field(min_length=1, max_length=50) last_name: Optional[str] = Field(min_length=1, max_length=100)

@app.get("/users", response_model=list[User]) def get_users(): with conn.cursor(cursor_factory=RealDictCursor) as cursor: cursor.execute("SELECT * FROM Users;") users = cursor.fetchall() return users

@app.post("/users", response_model=User, status_code=status.HTTP_201_CREATED) def create_user(user: User): with open(f'data/create.txt', mode='a') as file: data = user.dict() file.write(f"Request POST /users with body: {json.dumps(data)}\n") with conn.cursor(cursor_factory=RealDictCursor) as cursor: try: cursor.execute( "INSERT INTO Users (name, last_name, age) VALUES (%s, %s, %s) RETURNING *;", (user.name, user.last_name, user.age) ) created_user = cursor.fetchone() conn.commit() return created_user except Exception as e: conn.rollback() raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(e))

@app.get("/users/{user_id}", response_model=User) def get_user(user_id: int): with conn.cursor(cursor_factory=RealDictCursor) as cursor: cursor.execute("SELECT * FROM Users WHERE id = %s;", (user_id,)) user = cursor.fetchone() if user is None: raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="User not found") return user

@app.put("/users/{user_id}", response_model=User) def update_user(user_id: int, user: User): with open(f'data/put.txt', mode='a') as file: data = user.dict() file.write(f"Request PUT /users/{user_id} with body: {json.dumps(data)}\n") with conn.cursor(cursor_factory=RealDictCursor) as cursor: try: cursor.execute( """ UPDATE Users SET name = COALESCE(%s, name), last_name = COALESCE(%s, last_name), age = COALESCE(%s, age) WHERE id = %s RETURNING *; """, (user.name, user.last_name, user.age, user_id) ) updated_user = cursor.fetchone() conn.commit() if updated_user is None: raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="User not found") return updated_user except Exception as e: conn.rollback() raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(e))

@app.delete("/users/{user_id}", status_code=status.HTTP_204_NO_CONTENT) def delete_user(user_id: int): with conn.cursor(cursor_factory=RealDictCursor) as cursor: try: cursor.execute("DELETE FROM Users WHERE id = %s RETURNING id;", (user_id,)) deleted_user_id = cursor.fetchone() conn.commit() if deleted_user_id is None: raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="User not found") except Exception as e: conn.rollback() raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=str(e))

4. As you can see - there's a different in the response between service and OpenAPI schema = the response body (by Service) doesn't contain **age** parameter (in OpenAPI this parameter is mandatory).
5. And details about the error/issue you are facing

**Expected behaviour**
Regarding the documentation - https://endava.github.io/cats/docs/getting-started/filtering-reports#ignoring-response-body-checks, the test result should be marked as a warn, not a success.

**Environment:**
* Provide the output of: `cats info` or `java -jar cats.jar info`:
Key Value
OS Name Mac OS X
OS Version 14.3.1
OS Arch aarch64
Binary Type native
Cats Version 11.4.0
Cats Build 2024-04-03T17:31:02Z
Term Width 80
Term Type xterm-256color
Shell /bin/zsh

* OpenAPI file - 
[fastapi.json](https://github.com/Endava/cats/files/15062082/fastapi.json)
* I use HappyPath Fuzzer in this case.
en-milie commented 5 months ago

Hi @AbdulinRuslan. Currently, responses are not fully matched. It's only checking if any field in the response exists in the schema, but not the other way around. This is because responses can have a lot of variations, especially when using anyOf/oneOf combinations. This might be a feature that will be added in the future, but currently, indeed, is not supported.

en-milie commented 4 months ago

I will close this for now as the functionality is not there at the moment.