search

scrapy-plugins / scrapy-jsonschema

Scrapy schema validation pipeline and Item builder using JSON Schema
BSD 3-Clause "New" or "Revised" License
44 stars 12 forks source link

scrapy==2.2.0 breaks this #18

Closed BurnzZ closed 4 years ago

BurnzZ commented 4 years ago

Steps to Reproduce:

  1. Install scrapy==2.2.0
  2. Install scrapy_jsonschema=0.3.0
  3. Invoke import scrapy_jsonschema

It would error out like the one below: image

@asadurski has pointed out that it was due to this commit from scrapy: https://github.com/scrapy/scrapy/commit/5256eae60d3685de51c1f3891abe157e15d14def

BurnzZ commented 4 years ago

I've cloned this project and started out a new environment, ran pytest and received the breaking errors from the latest version of Scrapy.

I'm pasting the failed tests below while developing a fix:

============================================================================ test session starts ============================================================================
platform darwin -- Python 3.8.0, pytest-6.0.0, py-1.9.0, pluggy-0.13.1
rootdir: /Users/burnzz/dev/scrapinghub/scrapy-jsonschema
collected 0 items / 4 errors

================================================================================== ERRORS ===================================================================================
____________________________________________________________________ ERROR collecting tests/test_draf.py ____________________________________________________________________
../../../.pyenv/versions/3.8.0/envs/scrapy-jsonschema-3.8.0/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:961: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:783: in exec_module
    ???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
tests/__init__.py:1: in <module>
    from scrapy_jsonschema.draft import JSON_SCHEMA_DRAFT_7
scrapy_jsonschema/__init__.py:1: in <module>
    from scrapy_jsonschema.item import JsonSchemaItem
scrapy_jsonschema/item.py:118: in <module>
    class JsonSchemaItem(DictItem):
../../../.pyenv/versions/3.8.0/envs/scrapy-jsonschema-3.8.0/lib/python3.8/site-packages/six.py:878: in wrapper
    return metaclass(cls.__name__, cls.__bases__, orig_vars)
scrapy_jsonschema/item.py:65: in __new__
    cls = super(JsonSchemaMeta, mcs).__new__(mcs, class_name, bases, attrs)
../../../.pyenv/versions/3.8.0/envs/scrapy-jsonschema-3.8.0/lib/python3.8/abc.py:85: in __new__
    cls = super().__new__(mcls, name, bases, namespace, **kwargs)
E   TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
____________________________________________________________________ ERROR collecting tests/test_item.py ____________________________________________________________________
tests/test_item.py:7: in <module>
    from scrapy_jsonschema.item import JsonSchemaItem, _merge_schema
scrapy_jsonschema/__init__.py:1: in <module>
    from scrapy_jsonschema.item import JsonSchemaItem
scrapy_jsonschema/item.py:118: in <module>
    class JsonSchemaItem(DictItem):
../../../.pyenv/versions/3.8.0/envs/scrapy-jsonschema-3.8.0/lib/python3.8/site-packages/six.py:878: in wrapper
    return metaclass(cls.__name__, cls.__bases__, orig_vars)
scrapy_jsonschema/item.py:65: in __new__
    cls = super(JsonSchemaMeta, mcs).__new__(mcs, class_name, bases, attrs)
../../../.pyenv/versions/3.8.0/envs/scrapy-jsonschema-3.8.0/lib/python3.8/abc.py:85: in __new__
    cls = super().__new__(mcls, name, bases, namespace, **kwargs)
E   TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
________________________________________________________________ ERROR collecting tests/test_item_schema.py _________________________________________________________________
tests/test_item_schema.py:5: in <module>
    from scrapy_jsonschema.item import JsonSchemaItem, _merge_schema
scrapy_jsonschema/__init__.py:1: in <module>
    from scrapy_jsonschema.item import JsonSchemaItem
scrapy_jsonschema/item.py:118: in <module>
    class JsonSchemaItem(DictItem):
../../../.pyenv/versions/3.8.0/envs/scrapy-jsonschema-3.8.0/lib/python3.8/site-packages/six.py:878: in wrapper
    return metaclass(cls.__name__, cls.__bases__, orig_vars)
scrapy_jsonschema/item.py:65: in __new__
    cls = super(JsonSchemaMeta, mcs).__new__(mcs, class_name, bases, attrs)
../../../.pyenv/versions/3.8.0/envs/scrapy-jsonschema-3.8.0/lib/python3.8/abc.py:85: in __new__
    cls = super().__new__(mcls, name, bases, namespace, **kwargs)
E   TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
__________________________________________________________________ ERROR collecting tests/test_pipeline.py __________________________________________________________________
tests/test_pipeline.py:6: in <module>
    from scrapy_jsonschema.pipeline import JsonSchemaValidatePipeline
scrapy_jsonschema/__init__.py:1: in <module>
    from scrapy_jsonschema.item import JsonSchemaItem
scrapy_jsonschema/item.py:118: in <module>
    class JsonSchemaItem(DictItem):
../../../.pyenv/versions/3.8.0/envs/scrapy-jsonschema-3.8.0/lib/python3.8/site-packages/six.py:878: in wrapper
    return metaclass(cls.__name__, cls.__bases__, orig_vars)
scrapy_jsonschema/item.py:65: in __new__
    cls = super(JsonSchemaMeta, mcs).__new__(mcs, class_name, bases, attrs)
../../../.pyenv/versions/3.8.0/envs/scrapy-jsonschema-3.8.0/lib/python3.8/abc.py:85: in __new__
    cls = super().__new__(mcls, name, bases, namespace, **kwargs)
E   TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
============================================================================= warnings summary ==============================================================================
scrapy_jsonschema/item.py:90
  /Users/burnzz/dev/scrapinghub/scrapy-jsonschema/scrapy_jsonschema/item.py:90: SyntaxWarning: "is not" with a literal. Did you mean "!="?
    if p is not 'additionalProperties'

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================== short test summary info ==========================================================================
ERROR tests/test_draf.py - TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
ERROR tests/test_item.py - TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
ERROR tests/test_item_schema.py - TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
ERROR tests/test_pipeline.py - TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================= 1 warning, 4 errors in 0.81s ========================================================================
BurnzZ commented 4 years ago

So I've made two (2) approaches here:

EDIT: As per our Slack convo, it would make much more sense for the longer term approach and drop Python2 support.