scholarly-python-package / scholarly

Retrieve author and publication information from Google Scholar in a friendly, Pythonic way without having to worry about CAPTCHAs!
https://scholarly.readthedocs.io/
The Unlicense
1.29k stars 292 forks source link

fix: set minimum free-proxy version #487

Closed ltalirz closed 1 year ago

ltalirz commented 1 year ago

scholarly requires free-proxy >= 1.1.0, see

  File "/.../scholarly/_proxy_generator.py", line 509, in _fp_coroutine
    all_proxies = freeproxy.get_proxy_list(repeat=False)  # free-proxy >= 1.1.0
TypeError: get_proxy_list() got an unexpected keyword argument 'repeat'

Fixes #Enter the associated issue number.

Description

Describe your addition briefly to help the reviewers understand your contribution.

Checklist

arunkannawadi commented 1 year ago

Did you get an error there actually? This is within a try/except block, so it should be insensitive to whether we are using fp before or after the breaking change.

ltalirz commented 1 year ago

You are right - this excerpt I copied was part of the stack trace (and I incorrectly assumed it was the source of the error), but the actual error was:

  File "/.../fp/fp.py", line 24, in get_proxy_list
    proxies = [f'{tr_elements[i][0].text_content()}:{tr_elements[i][1].text_content()}' for i in
  File "/.../fp/fp.py", line 24, in <listcomp>
    proxies = [f'{tr_elements[i][0].text_content()}:{tr_elements[i][1].text_content()}' for i in
IndexError: list index out of range

This was with free-proxy 1.0.2. Upgrading to 1.1.0 fixed the issue

I guess the try/except can also be removed if one requires free-proxy>=1.1.0 .

arunkannawadi commented 1 year ago

I hadn't seen this error before. Perhaps it's best to require this. I'd leave the try/except block as it is for now.

arunkannawadi commented 1 year ago

Thanks for the two PRs.