Closed jayvdb closed 4 years ago
e.g.
DEBUG pypidb._pypi:_pypi.py:313 processing Webpage: http://msgpack.org/
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(xml.name) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(xml.name) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(xml.name) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(xml.name) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(quickstart-c.md) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(quickstart-cpp.md) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(howtoinstall.md) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(time.now) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(umsgpack.py) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(umsgpack.py) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(ext.data) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(in.mp) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(out.mp) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(clojure.java.io) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(mlmsgpack.mlb) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(mlmsgpack.cm) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(tutorial.md) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(msgpack.so) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(array.map) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(key.image.name) gaierror(-11)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(key.image.data) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(key.image.name) gaierror(-11)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(key.image.name) gaierror(-11)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(key.image.data) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(msgpack.marsworks.ru) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(mp.marsw.ru) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(msgpack.marsworks.ru) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(homepage:www.diocp.org) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(p.name) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(p.name) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(homepage-example.mp) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(xx.name) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(file.mp) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(file.mp) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(opcache.so) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(opcache.so) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(msgpack.so) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(opcache.so) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(msgpack.cr) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(mpc.zip) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(api.example.com) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(compatibility.md) gaierror(-2)
INFO urlextract:urlextract_core.py:567 Unknown exception during gethostbyname(notepack.io) gaierror(-2)
This limit was implemented and should be in latest release.
Parsing https://msgpack.org/ is very expensive, esp when DNS checking is enabled.
One way to avoid DOS is to limit the number of results. The caller could do this themselves by using
gen_urls
and manually building a result dict and stopping at a limit they define.However, default operation should behave in a sane manner. I suggest having a high limit to
find_urls
, after which an error is logged and results returned, or exception raised.IMO the limit should be provided when instantiating the
URLExtract
instance, and the caller can uselimit=False
to disable the limit. The limit could be even higher than needed for https://msgpack.org/ - maybe even10,000
. The importance is that building a limit into the library ensures that users also think about potential for DOS.