Closed ronnix closed 4 years ago
Scrapy uses ThreadedResolver (see https://github.com/scrapy/scrapy/blob/master/scrapy/resolver.py), and it uses stdlib socket.gethostbyname which doesn't support IPv6. So yes, Scrapy doesn't support IPv6 now.
It looks like Twisted itself supports IPv6, and Scrapy may start supporting it by switching to some other resolver (from twisted.names?), but I haven't checked the details.
Thanks for the explanation!
https://docs.python.org/2/library/socket.html#socket.gethostbyname suggests using getaddrinfo, btw.
Yes, it looks like ThreadedResolver subclass which uses getaddrinfo is an option.
An implementation based on socket.getaddrinfo: https://github.com/scrapy/scrapy/pull/1104
While #1104 is not in the master
this gist (dns resolver middleware) may be helpful.
any progress in this? it seems that other than the here discussed address resolution scrapy seems to be able to do ipv6 requests:
@qknight, is this really still an issue? Can you please provide a testcase/log?
I was under the impression that current Twisted Agent did getaddrinfo by default and so is also used by ScrapyAgent
in HTTP11DownloadHandler
.
@nyov i don't have a clue about the current implementation but it seems, out of your response, that it supports ipv6 now. is there an explicit switch to force scrapy to use ipv6?
Indeed, this is still an issue. Scrapy disables Twisted's IPv6 support by installing a non-IPv6-aware resolver. The problem is here:
If you don't want to trust the operating system's DNS caching for some reason, you can use the more modern API to install a custom resolver: https://twistedmatrix.com/documents/18.9.0/api/twisted.internet.interfaces.IReactorPluggableNameResolver.html#installNameResolver
and, rather than subclassing a resolver within Twisted (you shouldn't need the internal _GAIResolver
to be made public), you can write a generalized caching layer; an twisted.internet.interfaces.IHostnameResolver
that takes another IHostnameResolver
as an argument, and caches its results; then simply pass the previous value of reactor.nameResolver
to it.
Hope that this helps!
I'm running into problems while trying to run the example on the scrapy.org home page from the FOSDEM IPv6-only Wi-Fi network. (The scraper works fine from an IPv4 network.)
If both IPv4 and IPv6 are enabled on my computer (OS X Yosemite), and the IPv4 is configured with DHCP, and thus gets a self-assigned address (169.254.x.x), then I get timeout errors:
If I turn off IPv4 completely, then scrapy fails with "No route to host" errors:
Note that I can open the blog.scrapinghub.com site in Safari, so the target web site does support IPv6 and the problem seems to be on scrapy's side.