scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.
https://scrapy.org
BSD 3-Clause "New" or "Revised" License
50.98k stars 10.34k forks source link

Fix WrappedRequest.get_header raising TypeError if default is None #6310

Closed VMRuiz closed 4 weeks ago

VMRuiz commented 4 weeks ago

Fixes https://github.com/scrapy/scrapy/issues/6308

Alternatives:

    def get_header(self, name, default=None):
        value = self.request.headers.get(name, default)
        if not value:
            return value
        return to_unicode(value, errors="replace")

But I don't like repeating value so much

    def get_header(self, name, default=None):
        if not (value := self.request.headers.get(name, default)):
            return value
        return to_unicode(value, errors="replace")

The parenthesis and walrus operator make the line too convoluted

     def get_header(self, name, default=None):
         try: 
             to_unicode(self.request.headers.get(name, default), errors="replace")
         except TypeError:
             return default

Using try/except this way is pythonic, but I'm always afraid I may unexpectedly capture another unrelated exception