ArchiveTeam / wpull

Wget-compatible web downloader and crawler.
GNU General Public License v3.0
554 stars 77 forks source link

Always send the Host header first #472

Open Pokechu22 opened 1 year ago

Pokechu22 commented 1 year ago

This pull request fixes #468.

I remembered to:

I ran tests by doing pip install . -t thematrix/ --no-dependencies followed by cd thematrix/ && nosetests --with-coverage --cover-package=wpull --cover-branches, but that didn't seem to actually run request_test.py (in that I didn't see any relevant error output if I changed the test but not the rest of the code). I also did python3 -m unittest wpull.protocol.http.request_test which did fail before my change and now passes.

Unfortunately, it seems like cloudflare doesn't hate me, so I'm not able to tell if this actually fixes anything in practice.