search

scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.
https://scrapy.org
BSD 3-Clause "New" or "Revised" License
51.16k stars 10.35k forks source link

Solve incompatibility errors in _compression.py #6261

Closed ygalvao closed 2 months ago

ygalvao commented 2 months ago

Increase compatibility across different virtual environments, particularly with Conda, when using brotli.Decompressor()

wRAR commented 2 months ago

Is this only required when brotlipy is installed instead of/in addition to brotli?

ygalvao commented 2 months ago

Yes. I don't know why, but when setting up conda environments and installing scrapy and other packages it seems Conda / Mamba only installs brotlipy.

Gallaecio commented 2 months ago

Maybe we should do something to make Scrapy distinguish the 2, and only enable brotli support when the right one is installed.

wRAR commented 2 months ago

Yeah.

wRAR commented 2 months ago

when setting up conda environments and installing scrapy and other packages it seems Conda / Mamba only installs brotlipy.

Note that Scrapy currently does not depend on brotli (or on brotlipy), neither on PyPI nor on Conda, and if something installs brotlipy as opposed to not installing anything it's either "other packages" you mentioned or some other things you do when "setting up".

ygalvao commented 2 months ago

I understand. However, in my current conda env I just have a few packages installed, most of them are well-known ones like Scrapy, Selenium, Pandas, Numpy, etc. and somehow, Conda or Mamba installed brotlipy, which throws an AttributeError when running Scrapy. So this simple change solves this issue, adding compatibility with brotlipy and hence with other packages installed using Conda / Mamba as well.

codecov[bot] commented 2 months ago

Codecov Report

Merging #6261 (9cbdacb) into master (ee51958) will increase coverage by 0.00%. The diff coverage is 100.00%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #6261 +/- ## ======================================= Coverage 88.89% 88.90% ======================================= Files 161 161 Lines 11776 11786 +10 Branches 1913 1913 ======================================= + Hits 10468 10478 +10 Misses 964 964 Partials 344 344 ``` | [Files](https://app.codecov.io/gh/scrapy/scrapy/pull/6261?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | | |---|---|---| | [scrapy/utils/\_compression.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6261?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL19jb21wcmVzc2lvbi5weQ==) | `91.89% <100.00%> (+1.26%)` | :arrow_up: |