Filing this after terribot's encouragement earlier in the evening:
<+dgw> !how much deprecation?
<+terribot> dgw: 816 deprecations
On a more serious note, we had a look at what's actually useful in the tools.web module. The answer? Not much.
In my fact-based-but-still-opinionated view, the only possibly-useful utilities left are search_urls(), trim_url(), iri_to_uri() (maybe), USER_AGENT, and DEFAULT_HEADERS.
search_urls() is used in core code (but should probably be called find_urls() or extract_urls() instead)
trim_url() is used in search_urls()
iri_to_uri() seems helpful vs. just using encodings.idna because it understands the whole URL and IDN-encodes components of the domain name + urlencodes non-ASCII components of the path
urlencode_non_ascii is probably replaceable with something in stdlib, but I haven't found anything yet
USER_AGENT and DEFAULT_HEADERS are for plugin authors' convenience
DEFAULT_HEADERS should also be better insulated against being modified by plugins (i.e. some module-level equivalent to a property would be great, something that looks like a module attribute but returns a copy of its default-header dictionary)
Or we could go the simple way and replace DEFAULT_HEADERS with a function like get_default_headers() that returns a copy plugins can tamper with without affecting other code
Unless I've missed some other important stuff, that's going to leave a pretty barebones module behind. We might consider moving the remaining bits around (again; sopel.web moved to sopel.tools.web in #1616 / 7.0) but I'd feel weird about having the few bits and bobs left live in the root sopel.tools module.
Filing this after terribot's encouragement earlier in the evening:
On a more serious note, we had a look at what's actually useful in the
tools.webmodule. The answer? Not much.In my fact-based-but-still-opinionated view, the only possibly-useful utilities left are
search_urls(),trim_url(),iri_to_uri()(maybe),USER_AGENT, andDEFAULT_HEADERS.search_urls()is used in core code (but should probably be calledfind_urls()orextract_urls()instead)trim_url()is used insearch_urls()iri_to_uri()seems helpful vs. just usingencodings.idnabecause it understands the whole URL and IDN-encodes components of the domain name + urlencodes non-ASCII components of the pathurlencode_non_asciiis probably replaceable with something in stdlib, but I haven't found anything yetUSER_AGENTandDEFAULT_HEADERSare for plugin authors' convenienceDEFAULT_HEADERSshould also be better insulated against being modified by plugins (i.e. some module-level equivalent to apropertywould be great, something that looks like a module attribute but returns a copy of its default-header dictionary)DEFAULT_HEADERSwith a function likeget_default_headers()that returns a copy plugins can tamper with without affecting other codeUnless I've missed some other important stuff, that's going to leave a pretty barebones module behind. We might consider moving the remaining bits around (again;
sopel.webmoved tosopel.tools.webin #1616 / 7.0) but I'd feel weird about having the few bits and bobs left live in the rootsopel.toolsmodule.Plenty of time to think about this stuff, though.