search

python-openapi / openapi-core

Openapi-core is a Python library that adds client-side and server-side support for the OpenAPI v3.0 and OpenAPI v3.1 specification.
BSD 3-Clause "New" or "Revised" License
302 stars 132 forks source link

Test failures from openapi-schema-validator 0.4.2 #495

Closed musicinmybrain closed 1 year ago

musicinmybrain commented 1 year ago 
$ gh repo clone p1c2u/openapi-core
$ cd openapi-core
$ python3 -m venv _e
$ . _e/bin/activate
(_e) $ pip install poetry
(_e) $ poetry install
(_e) $ pytest

The tests pass.

Now:

(_e) $ pip install --upgrade openapi-schema-validator openapi-spec-validator
(_e) $ pytest
============================================== short test summary info ===============================================
FAILED tests/unit/unmarshalling/test_unmarshal.py::TestOAS30SchemaUnmarshallerCall::test_string_format_datetime_invalid - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_date_invalid[true] - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_uuid_invalid[true0] - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_invalid[true0] - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_invalid[true1] - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_invalid[1989-01-02] - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_strict_rfc3339[1989-01-02T00:00:00Z] - AttributeError: <module 'openapi_schema_validator._format' from '/home/ben/src/forks/openapi-core/_e/lib64/python...
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_strict_rfc3339[2018-01-02T23:59:59Z] - AttributeError: <module 'openapi_schema_validator._format' from '/home/ben/src/forks/openapi-core/_e/lib64/python...
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_isodate[1989-01-02T00:00:00Z] - AttributeError: <module 'openapi_schema_validator._format' from '/home/ben/src/forks/openapi-core/_e/lib64/python...
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_isodate[2018-01-02T23:59:59Z] - AttributeError: <module 'openapi_schema_validator._format' from '/home/ben/src/forks/openapi-core/_e/lib64/python...
ERROR tests/integration/contrib/django/test_django_project.py::TestPetListView::test_get_no_required_param - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetListView::test_get_valid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetListView::test_post_server_invalid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetListView::test_post_required_header_param_missing - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetListView::test_post_media_type_invalid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetListView::test_post_required_cookie_param_missing - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetListView::test_post_valid[data_json0] - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetListView::test_post_valid[data_json1] - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetDetailView::test_get_server_invalid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetDetailView::test_get_unauthorized - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetDetailView::test_delete_method_invalid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestPetDetailView::test_get_valid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/django/test_django_project.py::TestDRFPetListView::test_post_valid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetListResource::test_get_no_required_param - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetListResource::test_get_valid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetListResource::test_post_server_invalid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetListResource::test_post_required_header_param_missing - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetListResource::test_post_media_type_invalid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetListResource::test_post_required_cookie_param_missing - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetListResource::test_post_valid[data_json0] - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetListResource::test_post_valid[data_json1] - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetDetailResource::test_get_server_invalid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetDetailResource::test_get_path_invalid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetDetailResource::test_get_unauthorized - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetDetailResource::test_get_valid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/contrib/falcon/test_falcon_project.py::TestPetDetailResource::test_delete_method_invalid - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/schema/test_spec.py::TestPetstore::test_spec - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_response - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_response_no_schema - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_invalid_response - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_ids_param - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_tags_param - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_parameter_deserialization_error - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_wrong_parameter_type - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_raises_missing_required_param - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_empty_value - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_allow_empty_value - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_none_value - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_param_order - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pets_param_coordinates - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_birds - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_cats - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_cats_boolean_string - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_no_one_of_schema - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_cats_only_required_body - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_pets_raises_invalid_mimetype - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_pets_missing_cookie - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_pets_missing_header - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_pets_raises_invalid_server_error - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pet - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pet_not_found - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_pet_wildcard - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_get_tags - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_tags_extra_body_properties - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_tags_empty_body - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_tags_wrong_property_type - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_tags_additional_properties - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_tags_created_now - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_tags_created_datetime - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_post_tags_created_invalid_type - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_delete_tags_with_requestbody - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_delete_tags_no_requestbody - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_petstore.py::TestPetstore::test_delete_tags_raises_missing_required_response_header - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_request_server_error - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_invalid_path - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_invalid_operation - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_missing_parameter - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_get_pets - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_get_pets_webob - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_missing_body - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_invalid_content_type - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_invalid_complex_parameter - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_post_pets - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_post_pets_plain_no_schema - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_get_pet_unauthorized - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestRequestValidator::test_get_pet - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestResponseValidator::test_invalid_server - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestResponseValidator::test_invalid_operation - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestResponseValidator::test_invalid_response - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestResponseValidator::test_invalid_content_type - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestResponseValidator::test_missing_body - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestResponseValidator::test_invalid_media_type - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestResponseValidator::test_invalid_media_type_value - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestResponseValidator::test_invalid_value - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestResponseValidator::test_invalid_header - RecursionError: maximum recursion depth exceeded in comparison
ERROR tests/integration/validation/test_validators.py::TestResponseValidator::test_get_pets - RecursionError: maximum recursion depth exceeded in comparison
============= 10 failed, 677 passed, 5 xfailed, 2 xpassed, 277 warnings, 87 errors in 125.70s (0:02:05) ==============

(The full output is too long to paste here.)

p1c2u commented 1 year ago

The reason is the issue here https://github.com/p1c2u/openapi-spec-validator/issues/192

p1c2u commented 1 year ago

The issue was fixed and new openapi-spec-validator version 0.5.4 released hence closing

musicinmybrain commented 1 year ago

The steps in this report still produce test failures (albeit fewer of them) even with openapi-spec-validator 0.5.4.

============================================ short test summary info =============================================
FAILED tests/integration/validation/test_petstore.py::TestPetstore::test_post_tags_created_now - openapi_core.validation.request.exceptions.InvalidRequestBody: Request body error
FAILED tests/integration/validation/test_petstore.py::TestPetstore::test_post_tags_created_invalid_type - assert <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaFormatValue'> is InvalidSchemaValue
FAILED tests/unit/unmarshalling/test_unmarshal.py::TestOAS30SchemaUnmarshallerCall::test_string_format_datetime_invalid - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_date_invalid[true] - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_uuid_invalid[true0] - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_invalid[true0] - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_invalid[true1] - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_invalid[1989-01-02] - Failed: DID NOT RAISE <class 'openapi_core.unmarshalling.schemas.exceptions.InvalidSchemaValue'>
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_strict_rfc3339[1989-01-02T00:00:00Z] - AttributeError: <module 'openapi_schema_validator._format' from '/home/ben/src/forks/openapi-core/_e/lib64/py...
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_strict_rfc3339[2018-01-02T23:59:59Z] - AttributeError: <module 'openapi_schema_validator._format' from '/home/ben/src/forks/openapi-core/_e/lib64/py...
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_isodate[1989-01-02T00:00:00Z] - AttributeError: <module 'openapi_schema_validator._format' from '/home/ben/src/forks/openapi-core/_e/lib64/py...
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_isodate[2018-01-02T23:59:59Z] - AttributeError: <module 'openapi_schema_validator._format' from '/home/ben/src/forks/openapi-core/_e/lib64/py...
======================= 12 failed, 762 passed, 5 xfailed, 2 xpassed, 842 warnings in 4.97s =======================

If it matters, the above output is from Python 3.11.1.

musicinmybrain commented 1 year ago

The steps in this report still produce test failures (albeit fewer of them) even with openapi-spec-validator 0.5.4.

I noticed that openapi-schema-validator 0.4.3 is not yet on PyPI. If I install it manually, I’m down to 7 failures:

=========================== short test summary info ============================
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_date_invalid[true]
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_uuid_invalid[true0]
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_invalid[true0]
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_strict_rfc3339[1989-01-02T00:00:00Z]
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_strict_rfc3339[2018-01-02T23:59:59Z]
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_isodate[1989-01-02T00:00:00Z]
FAILED tests/unit/unmarshalling/test_validate.py::TestSchemaValidate::test_string_format_datetime_isodate[2018-01-02T23:59:59Z]
====== 7 failed, 728 passed, 5 xfailed, 2 xpassed, 980 warnings in 2.02s =======