Marshmallow 2 & 3 support

[atmo]

Hi @fuhrysteve ! I decided to create a separate PR for marshmallow 2 & 3 support, because in your PR #71 you're trying to achieve both marshmallow and jsonschema compatibility and I think that it's easier to go for marshmallow compatibility first and then support different versions of jsonschema (or only the newest one) in a different PR.

Aside from the obvous changes needed to support marshmallow 3, I've also added these improvements:

• Test for all possible combinations of Python (2.7, 3.4, 3.5, 3.6, 3.7, pypy, pypy3) with tox both locally and on CI.
• Move requirements to text files so that make and tox can easily access them.
• Add .coveragerc config for coverage and measure coverage with pytest-cov plugin.

Questions:

• I suggest getting rid of Python 3.4 (in a different PR) as it has reached end of life on 2019-03-16 (everything online points to this date, it is not available in the link above, but the fact that there is no Python 3.4 in that table at all seems to support this information). Never mind, I am forced to drop Python 3.4 support in this PR as the latest version of marshmallow 3 does not support it.
• I'm not entirely sure whether there is any value in testing Python 2 for marshmallow 3 support as Python 2 support has been removed in marshmallow 3.0.0rc6. Currently I'm testing against 3.0.0rc5 as a last version of marshmallow that supports Python 2, but I'm not sure if it is the right thing to do.

[coveralls]

Coverage increased (+1.9%) to 100.0% when pulling 40b2f683f06c110430b19ddfd6fe8f5a2c7cbbb4 on atmo:marshmallow_2_3_support into c51c6fdae00c09a9d2cdace5d1f5cfdbc88ae552 on fuhrysteve:master.

[atmo]

I've compiled here the list of breaking changes from marshmallow's upgrading to newer releases guide (as of 2019-07-13, version 3.0.0rc7), so that it's easier to reason about marshmallow 3 support. I add the following annotations to these changes:

:ok: - Does not require changes. :exclamation: - Requires changes. :question: - Does not require immediate changes but needs attention/should be considered as an additional feature.

Schemas are always strict

• The strict parameter was removed. Schemas are always strict. :ok:
• Schema().load and Schema().dump don't return a (data, errors) tuple any more. Only data is returned. :ok:

Schema().validate always returns a dictionary of validation errors (same as 2.x with strict=False). :ok:

Setting the strict option on class Meta has no effect on Schema behavior. :ok:

Passing strict=True or strict=False to the Schema constructor will raise a TypeError. :ok:

Decorated methods receive many and partial

• Methods decorated with pre_load, post_load, pre_dump, post_dump, and validates_schema receive many as a keyword argument. In addition, pre_load, post_load, and validates_schema receive partial. To account for these additional arguments, add **kwargs to your methods. :exclamation:

Deserializing invalid types raises a ValidationError

Numbers, booleans, strings, and None are considered invalid input to Schema.load. :ok:

ValidationError.fields is removed

ValidationError no longer stores a list of Field instances associated with the validation errors. :ok:

ValidationError expects a single field name

ValidationError no longer accepts a list of field names. It expects a single field name. If none is passed, the error refers to the schema. :ok:

ValidationError error messages are deep-merged

When multiple :ValidationError are raised, the error structures are merged in the final ValidationError raised at the end of the process. :ok:

Schemas raise ValidationError when deserializing data with unknown keys

Marshmallow 3.x schemas can deal with unknown keys in three different ways, configurable with the unknown option:

• EXCLUDE: drop those keys (same as marshmallow 2)
• INCLUDE: pass those keys/values as is, with no validation performed
• RAISE (default): raise a ValidationError :question:

Overriding get_attribute

If your Schema overrides get_attribute, you will need to update the method's signature. The positions of the attr and obj arguments were switched for consistency with Python builtins, e.g. getattr. :ok:

pass_original=True passes individual items when many=True

When pass_original=True is passed to validates_schema, post_load, or post_dump, the original_data argument will be a single item corresponding to the (de)serialized datum. :ok:

utils.get_func_args no longer returns bound arguments

The utils.get_func_args function will no longer return bound arguments, e.g. 'self'. :ok:

Handling AttributeError in Method and Function fields

The Method and Function fields no longer swallow AttributeErrors. Therefore, your methods and functions are responsible for handling inputs such as None. :ok:

Adding additional data to serialized output

Use a post_dump to add additional data on serialization. The extra argument on Schema was removed. :ok:

Schema-level validators are skipped when field validation fails

By default, schema validator methods decorated by validates_schema won't execute if any of the field validators fails (including required=True validation). :ok:

SchemaOpts constructor receives ordered argument

Subclasses of SchemaOpts receive an additional argument, ordered, which is True if the ordered option is set to True on a Schema or one of its parent classes. :ok:

ContainsOnly accepts empty and duplicate values

validate.ContainsOnly now accepts duplicate values in the input value. :ok:

json_module option is renamed to render_module

The json_module class Meta option is deprecated in favor of render_module. :ok:

missing and default Field parameters are passed in deserialized form

:ok:

Pass default as a keyword argument

fields.Boolean now receives additional truthy and falsy parameters. Consequently, the default parameter should always be passed as a keyword argument. :ok:

Email and URL fields do not validate on serialization

fields.Email and fields.URL only validate input upon deserialization. They do not validate on serialization. This makes them more consistent with the other fields and improves serialization performance. :ok:

load_from and dump_to are merged into data_key

The same key is used for serialization and deserialization. :ok:

Pre/Post-processors must return modified data

In marshmallow 2.x, None returned by a pre or post-processor is interpreted as "the data was mutated". In marshmallow 3.x, the return value is considered as processed data even if it is None. :ok:

Nested field no longer supports plucking

In marshmallow 2.x, when a string was passed to a Nested field's `only parameter, the field would be plucked. In marshmallow 3.x, the Pluck field must be used instead. :ok:

Accessing attributes on objects within a list

In order to serialize attributes on inner objects within a list, use the Pluck field. :ok:

Float field takes a new allow_nan parameter

In marshmallow 2.x, Float field would serialize and deserialize special values such as nan, inf or -inf. In marshmallow 3, those values trigger a ValidationError unless allow_nan is True. allow_nan defaults to False. :ok:

DateTime field dateformat Meta option is renamed datetimeformat

The Meta option dateformat used to pass format to DateTime field is renamed as datetimeformat. :ok:

The prefix Schema parameter is removed

The prefix parameter of Schema is removed. The same feature can be achieved using a post_dump method. :ok:

fields.FormattedString is removed

fields.FormattedString field is removed. Use fields.Function or fields.Method instead. :ok:

attribute or data_key collision triggers an exception

When a Schema is instantiated, a check is performed and a ValueError is triggered if

• several fields have the same attribute value (or field name if attribute is not passed), excluding dump_only fields, or :ok:
• several fields have the same data_key value (or field name if data_key is not passed), excluding load_only fields :ok:

python-dateutil recommended dependency is removed

In marshmallow 2, python-dateutil was used to deserialize RFC or ISO 8601 datetimes if it was installed. In marshmallow 3, datetime deserialization is done with no additional dependency.

python-dateutil is no longer used by marshmallow. :ok:

As we can see, the only change that needs changes is that many is passed to processors, and I've added it to the PR.

There is also other change that requires attention, the unknown option (EXCLUDE, INCLUDE and RAISE). I wanted to emphasize it because I plan to convert into additionalProperties support for jsonschema later on.

[atmo]

Hey @fuhrysteve , could you please tell what are your thoughts on this?

[fuhrysteve]

Excellent work!! Thanks for the pull, this is wonderful!

Thanks for adding tox in there, I had meant to do that myself to accomplish all the multiple-versions of everything problem.

I think we're probably going to have to continue to support marshmallow 2 & 3 for a good bit after marshmallow v3 finally gets released. For better or worse it's not as though we gain a ton in this project in particular by dropping py2 or marshmallow2 support.