search

groupbwt / scrapy-boilerplate

Scrapy project boilerplate done right
MIT License
42 stars 25 forks source link

feat(rmq): update scrapy to 2.11.0 version #136

Closed LyricalToxic closed 5 months ago

LyricalToxic commented 5 months ago

Update note version 2.3.0

  1. Updated dependency Scrapy 2.5.1 -> 2.11.0
  2. Updated dependency mysqlclient 1.4.6 -> 2.2.1
  3. Changed min python version 3.9 -> 3.11
  4. Optparse lib has been replaced with argparse lib in rpc commands according to Scrapy usage.
    1. Removed callback option because argparse not supported it from the box. Parameter "type" now validates types. Callbacks are no longer needed
  5. Added new "delay" option for Producer. It allows changing default value of command sleeping if not new records detected.
  6. Added explicitly value for scrapy setting REQUEST_FINGERPRINTER_IMPLEMENTATION, because warning appears in log.
  7. Removed spider parameter for engine crawl method, because it was deprecated.
  8. Renamed "opts" argument -> "namespace", because argparse lib uses this term.
  9. Added two util function for sql expression compiling.
  10. Changed rmq command expression executing. Now using parameter bypassing without binds.

Changes have been manually tested.