search

scrapinghub / spidermon

Scrapy Extension for monitoring spiders execution.
https://spidermon.readthedocs.io
BSD 3-Clause "New" or "Revised" License
526 stars 94 forks source link

Schematics seems to no longer be supported. #370

Closed paulvav closed 4 months ago

paulvav commented 1 year ago

Hi Spidermon team,

Was looking over the schematics project and noticed it looks like it is no longer maintained. I note that it doesn't support Python 3.10 and doesn't appear to have any code changes for over a year. See also: https://github.com/schematics/schematics/issues/628

Going forward should the recommendation be to use jsonschema exclusively for item validation? Or will another validation library be considered - maybe https://github.com/pydantic/pydantic ?

rennerocha commented 1 year ago

Hello @paulvav ,

schematics looks dead to me as well and I have no good hope that they will merge this 3.10 PR soon (and 3.11 is already there for a while).

From my point of view, considering the projects that I saw when I worked at Zyte (ex-Scrapinghub) which is the company that sponsor Spidermon, I don't remember any of them that uses schematics. jsonschema is the standard for item validation there. Perhaps outside Zyte this could be the same?

I would suggest to deprecate schematics (adding a Deprecation warning for anyone using it) in the next release (1.18.0) and them remove it from Spidermon in 1.19.0. This would make the item validation pipeline much cleaner too.

Any thoughts on that @VMRuiz @Gallaecio ?

VMRuiz commented 1 year ago

Hello @paulvav , @rennerocha ,

Schematics looks abandoned indeed. Even the doc warms is currently somewhat outdated. So I'm good with deprecating this eventually.

If there is enough interest on other validation tools we could think of adding them. But that is a different topic that doesn't need to wait for this one to resolve.

HaddadJoe commented 1 year ago

@VMRuiz schematics is broken starting python 3.10 and beyond. Tightly coupling spidermon with a dead project (latest update from 2021) is not a great idea for spidermon moving forward. this is preventing us from upgrading our spider's python version. Would you be open for PRs that drop support of schematics in favor of other validation libraries? If yes, any preference on which solutions to adopt?

VMRuiz commented 1 year ago

Hello @HaddadJoe, thanks for you interest on the topic!

We are open to PR both for deprecating schematics and to add new validations libraries. They don't necessarily needs to be done simultaneously

raphapassini commented 1 year ago

I'm working on that. I'll have a PR ready soon.