rmax / scrapy-redis

Redis-based components for Scrapy.
http://scrapy-redis.readthedocs.io
MIT License
5.49k stars 1.59k forks source link

[spiders] remove duplicate check setting types #260

Closed LuckyPigeon closed 1 year ago

LuckyPigeon commented 1 year ago

Description

When a spider get parameter from settings, there's a duplicate type checking show as follow.

if self.redis_batch_size is None:
  self.redis_batch_size = settings.getint('CONCURRENT_REQUESTS', defaults.REDIS_CONCURRENT_REQUESTS)
try:
  self.redis_batch_size = int(self.redis_batch_size)
except (TypeError, ValueError):
  raise ValueError("redis_batch_size must be an integer")

code sample: spiders.py

Solution

Since showing an error message is recommended code style, then replace getint or getxxx to get can do the job.