No errors on Python 3.12 when none of `url`, `urls`, or `urls_file` is given

[BurnzZ]

Overview

From the following PRs:

• https://github.com/zytedata/zyte-spider-templates/pull/41, https://github.com/zytedata/zyte-spider-templates/pull/50
• https://github.com/zytedata/zyte-spider-templates/pull/38

We have respectively introduced urls_file and urls as new parameters to indicate input URLs to the crawls, alongside the existing url parameter.

Should none of these 3 parameters are supplied to a crawl, the expected behavior would be to have the following error message:

  File "/some_dir/zyte-spider-templates-project/venv/lib/python/site-packages/pydantic/main.py", line 193, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for EcommerceSpiderParams
  Value error, No input parameter defined. Please, define one of: url, urls, urls_file. [type=value_error, input_value={}, input_type=dict]

However, it would seem that when using Python 3.12, the error does not exist.

Code to Reproduce

from scrapy_spider_metadata import Args
from zyte_spider_templates.spiders.ecommerce import EcommerceSpiderParams
from zyte_spider_templates.spiders.base import BaseSpider

class Spider(Args[EcommerceSpiderParams], BaseSpider):
    name = "spider"

if __name__ == "__main__":
    Spider()

Python 3.11

• ✅ python file.py
• ✅ scrapy crawl spider

Python 3.12

• ✅ python file.py
• ❌ scrapy crawl spider (no error at all)

[wRAR]

It's basically https://github.com/scrapy/scrapy/issues/6047

The exception is bubbled up to the deferred created with self.crawler_process.crawl() in the crawl or runspider command, but that deferred has no errback.

(No idea why is this situation handled differently on 3.11 and 3.12 :shrug:)

[wRAR]

So ideally we just shouldn't rely on unhandled exceptions, unless we fix Scrapy.

[Gallaecio]

So in Python 3.12+ Twisted no longer reports (exceptions in) unhandled deferreds?

[wRAR]

Not sure what could have changed.

from scrapy import Spider

class MySpider(Spider):
    name = "spider"

    def __init__(self, *args, **kwargs):
        1/0

This shows an unhandled exception on both Python versions.

[wRAR]

So far I was able to minimize it to this:

import scrapy
from pydantic import BaseModel, model_validator

class Model(BaseModel):
    @model_validator(mode="after")
    def foo(self):
        raise ValueError()

class Spider(scrapy.Spider):
    name = "spider"

    def __init__(self, *args, **kwargs) -> None:
        Model()
        super().__init__(*args, **kwargs)

Just having e.g. a required field is not enough to trigger this.