search

python-restx / flask-restx

Fork of Flask-RESTPlus: Fully featured framework for fast, easy and documented API development with Flask
https://flask-restx.readthedocs.io/en/latest/
Other
2.16k stars 335 forks source link

Marshal expect with validate true doesn't work with nested field #66

Open filipealc opened 4 years ago

filipealc commented 4 years ago

Hello Folks,

When running a test case against an endpoint I'm getting the following error:

 "Unresolvable JSON pointer: %r" % fragment
jsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'definitions/test_case_params'

Pip freeze:

alembic==1.4.0
aniso8601==8.0.0
astroid==2.3.3
attrs==19.3.0
autopep8==1.5
certifi==2019.11.28
chardet==3.0.4
Click==7.0
Flask==1.1.1
Flask-Cors==3.0.8
Flask-Migrate==2.5.2
flask-restx==0.1.1
Flask-SQLAlchemy==2.4.1
Flask-Testing==0.7.1
gunicorn==20.0.0
idna==2.8
importlib-metadata==1.5.0
isort==4.3.21
itsdangerous==1.1.0
Jinja2==2.11.1
jsonschema==3.2.0
lazy-object-proxy==1.4.3
Mako==1.1.1
MarkupSafe==1.1.1
mccabe==0.6.1
psycopg2-binary==2.8.3
pycodestyle==2.5.0
pylint==2.4.4
pyrsistent==0.15.7
python-dateutil==2.8.1
python-dotenv==0.10.3
python-editor==1.0.4
pytz==2019.3
requests==2.22.0
six==1.14.0
SQLAlchemy==1.3.13
typed-ast==1.4.1
urllib3==1.25.8
Werkzeug==0.16.1
wrapt==1.11.2
zipp==2.2.0

Full traceback:

ERROR: test_test_case_post_success_only_required_params (app.test.test_test_case_controller.TestTestCaseController)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/validators.py", line 811, in resolve_fragment
    document = document[part]
KeyError: 'definitions'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/cleivianecosta/apps/geekhunter-dexter/app/test/test_test_case_controller.py", line 124, in test_test_case_post_success_only_required_params
    follow_redirects=True)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/werkzeug/test.py", line 1039, in post
    return self.open(*args, **kw)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/testing.py", line 227, in open
    follow_redirects=follow_redirects,
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/werkzeug/test.py", line 993, in open
    response = self.run_wsgi_app(environ.copy(), buffered=buffered)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/werkzeug/test.py", line 884, in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/werkzeug/test.py", line 1119, in run_wsgi_app
    app_rv = app(environ, start_response)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_restx/api.py", line 599, in error_router
    return original_handler(f)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_restx/api.py", line 597, in error_router
    return self.handle_error(e)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_restx/api.py", line 599, in error_router
    return original_handler(f)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_restx/api.py", line 597, in error_router
    return self.handle_error(e)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_restx/api.py", line 339, in wrapper
    resp = resource(*args, **kwargs)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_restx/resource.py", line 42, in dispatch_request
    self.validate_payload(meth)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_restx/resource.py", line 88, in validate_payload
    self.__validate_payload(expect, collection=False)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_restx/resource.py", line 73, in __validate_payload
    expect.validate(data, self.api.refresolver, self.api.format_checker)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/flask_restx/model.py", line 104, in validate
    validator.validate(data)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/validators.py", line 352, in validate
    for error in self.iter_errors(*args, **kwargs):
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
    for error in errors:
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/_validators.py", line 286, in properties
    schema_path=property,
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
    for error in self.iter_errors(instance, schema):
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
    for error in errors:
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/_legacy_validators.py", line 55, in items_draft3_draft4
    for error in validator.descend(item, items, path=index):
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/validators.py", line 344, in descend
    for error in self.iter_errors(instance, schema):
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/validators.py", line 328, in iter_errors
    for error in errors:
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/_validators.py", line 259, in ref
    scope, resolved = validator.resolver.resolve(ref)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/validators.py", line 766, in resolve
    return url, self._remote_cache(url)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/validators.py", line 781, in resolve_from_url
    return self.resolve_fragment(document, fragment)
  File "/Users/cleivianecosta/apps/geekhunter-dexter/.env/lib/python3.7/site-packages/jsonschema/validators.py", line 814, in resolve_fragment
    "Unresolvable JSON pointer: %r" % fragment
jsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'definitions/test_case_params'

The endpoint


...

API = Namespace("test_cases", description="test cases related operations")
TEST_CASE_PARAMS = API.model("test_case_params", {
    "question_id": fields.Integer(
        required=True, description="The related question id"
    ),
    "name": fields.String(
        required=True, description="The test case name"
    ),
    "points": fields.Integer(
        required=True, description="How many points is the test worth"
    ),
    "test_input": fields.String(
        required=True, description="The input for the test"
    ),
    "test_output": fields.String(
        required=True, description="The expected output for the test"
    ),
    "is_visible": fields.Boolean(description="True if the test case is an example")
})

TEST_CASES_POST_PARAMS = API.model("test_cases_post_params", {
    "test_cases": fields.List(fields.Nested(TEST_CASE_PARAMS), required=True)
})

...

@API.route("/")
@API.expect(PARSER)
class TestCaseList(Resource):
  @API.expect(TEST_CASES_POST_PARAMS, validate=True)
  @API.marshal_with(LIST_TEST_CASES_MODEL)
  @token_required
  def post(self):
    data = request.json
    test_cases = create_test_cases(data["test_cases"])
    if test_cases is None:
      return {
          "error": "Could not create test_cases at this time"
      }, HTTPStatus.INTERNAL_SERVER_ERROR
    result = {"test_cases": [CreateGetTestCase(
        test_case) for test_case in test_cases]}
    return result, HTTPStatus.OK

The Test Case

def test_test_case_post_success_only_required_params(self):
    """
    Test if the /test-cases/ endpoint is working in case of success
    when only required params are passed
    """
    breakpoint()
    temp_params = {"test_cases": [self.test_case_required_params]}
    response = self.client.post("/test-cases/",
                                headers={
                                    "Content-Type": "application/json",
                                    "Authorization": APP.config["DEFAULT_AUTH_TOKEN"]
                                },
                                data=json.dumps(temp_params),
                                follow_redirects=True)

    self.assertTrue(response.status_code == HTTPStatus.OK)

    output_data = json.loads(response.data)
    for key in self.list_output_keys:
      self.assertTrue(key in output_data)
    self.assertTrue(output_data["error"] is None)

Note that before trying to migrate restplus to restX this was working as expected but on restX this error starts to pop up and if I set on the endpoint validate=False the problem would go away! So I think that there's some kind of problem when restX is trying to validate the params

dengqianyi commented 4 years ago

hi, all. how to fixed it?I already get this problem

jamiedoornbos commented 3 years ago

I'm seeing this as well, and my project relies heavily on nested validation. My example is pretty similar to the OP but I have a production endpoint using Nested and I get this trace in postman and in integration tests.

EDIT: For reference I'm running restx 0.3.0

alembic==1.5.8
AMQPStorm==2.8.4
aniso8601==9.0.1
attrs==20.3.0
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
click==7.1.2
coverage==5.5
cryptography==3.4.7
decorator==5.0.7
directly==0.0.30
Flask==1.1.2
Flask-Cors==3.0.10
flask-restx==0.3.0
flask-talisman==0.7.0
greenlet==1.0.0
idna==2.10
iso8601==0.1.14
itsdangerous==1.1.0
Jinja2==2.11.3
jsonpath-ng==1.5.2
jsonschema==3.2.0
Mako==1.1.4
MarkupSafe==1.1.1
mysqlclient==2.0.3
pamqp==2.3.0
ply==3.11
psutil==5.8.0
pycparser==2.20
pycrypto==2.6.1
PyJWT==1.7.1
pyrsistent==0.17.3
python-dateutil==2.8.1
python-editor==1.0.4
pytz==2021.1
regex==2021.4.4
requests==2.25.1
six==1.15.0
SQLAlchemy==1.4.8
urllib3==1.26.4
Werkzeug==0.16.1
ander2 commented 3 years ago

I came across this issue while I was editing a model and started using a Nested field. When running my tests, I got the error referenced in the issue:

"Unresolvable JSON pointer: %r" % fragment
jsonschema.exceptions.RefResolutionError: Unresolvable JSON pointer: 'definitions/test_case_params'

Having a look to the error message and to the endpoints, I realised that the endpoint was defined with doc=False, so I guessed that no schema was being generated. After removing doc=False from the endpoint, the errors disappeared and validations started to work.

I know it's not the same use case as in the original issue, but it may help someone in the future. Tested in flask-restx 0.4.0

Elysium1436 commented 3 years ago

I have been running into the same problem. If anyone has a solution it'd be much appreciated

Elysium1436 commented 3 years ago

I have been running into the same problem. If anyone has a solution it'd be much appreciated

Just realised people are remaking the whole marshalling.

andreikitaitsev commented 2 years ago

Same problem while parsing open API specification via yaml.loader. Is there any other way to validate nested schemas from open API specification apart from parsing it into json and using json schema validation?