The goal of this PR is, as mentioned in the title, unmarshaling performance enhancement.
NOTE: A similar PR will be published for applying the same approach described here for the marshaling process. I would like to have this verified first in order to avoid having two massive branches open at the same time
In order to achieve unmarshaling performance boost I've rewritten the whole bravado_core.unmarshal module.
The current, before this PR, logic of unmarshaling is:
extract the type from the schema
if the schema if of type x then call the appropriate unmarshaling function
if the schema is of type object: call the top unmarshaling function for the property (point 1)
if the schema is of type object: call the top unmarshaling function for each item of the list (point 1)
if the schema is of primitive type: identify, each time, the right formatting function to use
once the unmarshaling function is identified then perform the real unmarshaling
This PR aims to cache results of point 1 and 2 for later "cost-free" usage.
While making this changes I've noticed that there might be no real good reason to have all the unmarshaling functions public, users of the library commonly use bravado_core.unmarshal.unmarshal_schema_object.
In order to simplify future maintenance work I've marked all the other functions as deprecated in the next major release. Doing so will allows us to modify more, if needed, the internals of unmarshaling without worrying about breaking backward compatibility.
Be aware that:
bravado_core.unmarshal.get_unmarshaling_method is wrapped by _decorators.wrap_recursive_call_exception and memoize_by_id decorators.
_decorators.wrap_recursive_call_exception decorator is needed as models could be recursive and so invoking get_unmarshaling_method might end up on an infinite recursion.
In order to proactively deal with it I've modified @memoize_by_id decorator to raise a controlled exception if an unbounded recursion is identified (ℹ️ the interpreter would raise anyway with a RuntimeError: maximum recursion depth exceeded)
memoize_by_id decorator is needed to speed up evaluation of the marshaling method for the same spec and schema. This effectively is one of the major performance boost (basically the expensive point 1 and 2 of the original flow become, more or less, a dictionary lookup)
~unmarshaling of a polymorphic object does not raise the traditional SwaggerMappingError.
ℹ️ : The change is intentional as we should not do validation if validation is disabled (and as it was it was not possible to enable/disable that validation)~ (addressed by #333 )
~Add support for unmarshaling additionalProperties (it was not done previously) -> test_unmarshal_object_with_additional_properties test~ (addressed by #333 )
Coverage decreased (-0.2%) to 98.065% when pulling c98a9c03ce95ee118334949901969d1a66cfef1c on macisamuele:maci-unmarshal-performances into c48be8acf013628c6eb8f434f6e24390c1ad4143 on Yelp:master.
The goal of this PR is, as mentioned in the title, unmarshaling performance enhancement. NOTE: A similar PR will be published for applying the same approach described here for the marshaling process. I would like to have this verified first in order to avoid having two massive branches open at the same time
In order to achieve unmarshaling performance boost I've rewritten the whole
bravado_core.unmarshal
module.The current, before this PR, logic of unmarshaling is:
This PR aims to cache results of point 1 and 2 for later "cost-free" usage.
While making this changes I've noticed that there might be no real good reason to have all the unmarshaling functions public, users of the library commonly use
bravado_core.unmarshal.unmarshal_schema_object
. In order to simplify future maintenance work I've marked all the other functions as deprecated in the next major release. Doing so will allows us to modify more, if needed, the internals of unmarshaling without worrying about breaking backward compatibility.Be aware that:
bravado_core.unmarshal.get_unmarshaling_method
is wrapped by_decorators.wrap_recursive_call_exception
andmemoize_by_id
decorators._decorators.wrap_recursive_call_exception
decorator is needed as models could be recursive and so invokingget_unmarshaling_method
might end up on an infinite recursion. In order to proactively deal with it I've modified@memoize_by_id
decorator to raise a controlled exception if an unbounded recursion is identified (ℹ️ the interpreter would raise anyway with aRuntimeError: maximum recursion depth exceeded
)memoize_by_id
decorator is needed to speed up evaluation of the marshaling method for the same spec and schema. This effectively is one of the major performance boost (basically the expensive point 1 and 2 of the original flow become, more or less, a dictionary lookup)SwaggerMappingError
. ℹ️ : The change is intentional as we should not do validation if validation is disabled (and as it was it was not possible to enable/disable that validation)~ (addressed by #333 )~Add support for unmarshaling additionalProperties (it was not done previously) ->
test_unmarshal_object_with_additional_properties
test~ (addressed by #333 )Let's get to the juicy part 😄 Raw data on: https://gist.github.com/macisamuele/9ff23d3e5e81c40cd884425a7eae5751
This PR provides a massive performance improvement: