j-andrews7 / kenpompy

A simple yet comprehensive web scraper for kenpom.com.
https://kenpompy.readthedocs.io/en/latest/?badge=latest
GNU General Public License v3.0
70 stars 21 forks source link

Build URLs with urllib instead of manually #70

Closed esqew closed 9 months ago

esqew commented 10 months ago

Nice to have: a much cleaner way to generate the target scraping URLs and their parameters would be to use urllib.parse.urlencode, which would preclude us from having to do escapes, etc. ourselves.

A non-exhaustive list of some examples of where this could be leveraged:

Wolfmyths commented 10 months ago

Hey I'd like to take a look at this issue, but before I do. Would you rather have urls formatted like this?

# Current Format
url = url + '?' + urllib.parse.urlencode({"y":str(season)})
# New Format
url = f'{url}?{urllib.parse.urlencode({"y":str(season)})}'

We could also do something like this

url += f'?{urllib.parse.urlencode({"y":str(season)})}'
esqew commented 10 months ago

The first would be most preferable, however we generally put more of an emphasis on readability than other projects. Something akin to the following would be preferable:

params = {}
if season:
    params['y'] = str(season)

url = url + '?' + urllib.parse.urlencode(params)