APIFuzzer does not parse $ref in /paths

[jmaister]

Describe the bug

The OpenAPI definition that I am using has references ($ref) in the /paths section. The parser from APIFuzzer does not take it as valid.

APIFuzzer debug log

jordi@DESKTOP-NISTS11:/mnt/c/dev/workspace/APIFuzzer$ ./APIFuzzer -s ../saas-api/openapi/main.yml
2508 [WARNING] APIFuzzer.ResolveReferences: Failed to find paths.~1v1~1tenants because: Parse error at 1:6 near token ~ (~)
2508 [WARNING] APIFuzzer.ResolveReferences: No /paths/~1v1~1tenants in {"components": {"parameters": {}, "schemas": {"Tenant": {"allOf": [{"$ref": "./types.yml#/components/schemas/Id"}, {"$ref": "#/components/schemas/TenantData"}], "description": "Tenant model"}, "Tenant
2508 [WARNING] APIFuzzer.ResolveReferences: 2 Nothing to do with <class 'NoneType'>: None
2508 [WARNING] APIFuzzer.ResolveReferences: Failed to find paths.~1v1~1tenants~1%7BtenantId%7D because: Parse error at 1:6 near token ~ (~)
2508 [WARNING] APIFuzzer.ResolveReferences: No /paths/~1v1~1tenants~1%7BtenantId%7D in {"components": {"parameters": {}, "schemas": {"Tenant": {"allOf": [{"$ref": "./types.yml#/components/schemas/Id"}, {"$ref": "#/components/schemas/TenantData"}], "description": "Tenant model"}, "Tenant
2508 [WARNING] APIFuzzer.ResolveReferences: 2 Nothing to do with <class 'NoneType'>: None
2508 [WARNING] APIFuzzer.ResolveReferences: Failed to find paths.~1v1~1tenants~1%7BtenantId%7D~1customers because: Parse error at 1:6 near token ~ (~)
2508 [WARNING] APIFuzzer.ResolveReferences: No /paths/~1v1~1tenants~1%7BtenantId%7D~1customers in {"components": {"schemas": {"Customer": {"allOf": [{"$ref": "./types.yml#/components/schemas/Id"}, {"$ref": "#/components/schemas/CustomerData"}], "description": "Customer model"}, "CustomerData": {"d
2508 [WARNING] APIFuzzer.ResolveReferences: 2 Nothing to do with <class 'NoneType'>: None
2508 [WARNING] APIFuzzer.ResolveReferences: Failed to find paths.~1v1~1tenants~1%7BtenantId%7D~1customers~1%7BcustomerId%7D because: Parse error at 1:6 near token ~ (~)
2508 [WARNING] APIFuzzer.ResolveReferences: No /paths/~1v1~1tenants~1%7BtenantId%7D~1customers~1%7BcustomerId%7D in {"components": {"schemas": {"Customer": {"allOf": [{"$ref": "./types.yml#/components/schemas/Id"}, {"$ref": "#/components/schemas/CustomerData"}], "description": "Customer model"}, "CustomerData": {"d
2508 [WARNING] APIFuzzer.ResolveReferences: 2 Nothing to do with <class 'NoneType'>: None
2508 [  ERROR] APIFuzzer: Exception: 'NoneType' object has no attribute 'keys'
Traceback (most recent call last):
  File "/home/jordi/.local/lib/python3.8/site-packages/apifuzzer/fuzzer.py", line 37, in prepare
    template_generator.process_api_resources()
  File "/home/jordi/.local/lib/python3.8/site-packages/apifuzzer/openapi_template_generator.py", line 100, in process_api_resources
    self._process_request_body()
  File "/home/jordi/.local/lib/python3.8/site-packages/apifuzzer/openapi_template_generator.py", line 110, in _process_request_body
    for method in paths[resource].keys():
AttributeError: 'NoneType' object has no attribute 'keys'
Unexpected exception happened during fuzz test preparation: 'NoneType' object has no attribute 'keys'. Feel free to report the issue

Related API definition

main.yml

openapi: 3.0.2  
info:
  title: Some API
  version: 1.0.0
  description: Main file to hold the rest of the OpenAPI definitions.
  contact:
    name: Some
    email: some@gmail.com
servers:
  - url: 'http://localhost:8080'
paths:
  /v1/tenants:
    $ref: 'tenants.yml#/paths/~1v1~1tenants'
  /v1/tenants/{tenantId}:
    $ref: 'tenants.yml#/paths/~1v1~1tenants~1%7BtenantId%7D'

tenants.yml (reduced)

openapi: 3.0.2
info:
  title: Tenants
  version: 1.0.0
  contact:
    name: Some
    email: some@gmail.com
  description: Tenants API
servers:
  - url: 'http://localhost:8080'
paths:
  /v1/tenants:
    post:
      tags:
        - tenant
      summary: Create a new tenant
      responses:
        '201':
          description: Created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TenantData'
      operationId: post-v1-tenants
      description: Create a new tenant
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/TenantData'
components:
  securitySchemes: {}
  schemas:
    TenantData:
      title: TenantData
      type: object
      x-examples:
        example-1:
          name: Cool Startup
      description: Holds the information for creating a new Tenant.
      properties:
        name:
          type: string
          example: Cool Startup
          minLength: 1
          maxLength: 256
      required:
        - name
  parameters: {}
tags:
  - name: tenant
    description: Tenant Tag

Software environment (please complete the following information):

• OS: Windows with DSL Ubuntu 20.04.3
  • Python version: 3.8.10
• APIFuzzer Version: 0.9.11

Additional context Add any other context about the problem here.

[KissPeter]

Can you please double chech if tenants.yml (reduced) generates this error? I tried with the latest version several times, but didn't see the error. Thanks

[jmaister]

Yes, the reduced file of tenants.xml works fine. I get that error when running it on the main.xml file

APIFuzzer -s main.xml

However, the full version gives me a recursion error. I'll create a new issue when I can create a reduced file with the error.

[KissPeter]

I thought the reduced API is for me. Can you please test if the issue still exists with the new APIFuzzer version?

[jmaister]

Running with the new version, I get these error logs:

jordi@DESKTOP-NISTS11:/mnt/c/dev/workspace/saas-api/openapi$ APIFuzzer -s main.yml -u 192.168.1.34:8080 --headers '[{"Authorization": "Bearer ..."}]'
209 [  ERROR] APIFuzzer: Exception: 'list' object has no attribute 'get'
Traceback (most recent call last):
  File "/mnt/c/dev/workspace/APIFuzzer/apifuzzer/fuzzer.py", line 45, in prepare
    template_generator.process_api_resources()
  File "/mnt/c/dev/workspace/APIFuzzer/apifuzzer/openapi_template_generator.py", line 109, in process_api_resources
    self._process_request_body()
  File "/mnt/c/dev/workspace/APIFuzzer/apifuzzer/openapi_template_generator.py", line 123, in _process_request_body
    paths[resource][method].get("requestBody", {}).get("content", [])
AttributeError: 'list' object has no attribute 'get'
Unexpected exception happened during fuzz test preparation: 'list' object has no attribute 'get'. Feel free to report the issue

[KissPeter]

After removed the following lines from the tenants.xml " /v1/tenants/{tenantId}: $ref: 'tenants.yml#/paths/~1v1~1tenants~1%7BtenantId%7D" The latest version started fuzzing.

The exception reports a strange thing: "APIFuzzer/apifuzzer/openapi_template_generator.py", line 123, in _process_request_body paths[resource][method].get("requestBody", {}).get("content", []) AttributeError: 'list' object has no attribute 'get'" either the method or the requestBody key is a list instead of the expected dict. Please make sure your API definiton is valid.

[mbeckerle-xqueue]

I am not really sure if it is related so I decided to open a new bug report even if the error message is kind of the same. See #49