Proxy not used when crawling on localhost network

[GuilloOme]

When launching a crawl, it seems that only the start url and robots.txt are requested through the proxy (during the validation process).

way to reproduce:

• start a crawl with: $ ./htcap.py crawl -v -p http:127.0.0.1:8080 http://localhost/index.html test.db you get:

  
  Initializing . . done
  Database test.db initialized, crawl started with 10 threads
  crawl result for: link GET http://localhost/index.html  
  new request found link GET http://localhost/test1.html 
  crawl result for: link GET http://localhost/test1.html  
  new request found link GET http://localhost/test2.html 
  new request found link GET http://localhost/index.html 
  crawl result for: link GET http://localhost/test2.html  
  new request found link GET http://localhost/test1.html 
  new request found link GET http://localhost/index.html 

Crawl finished, 3 pages analyzed in 0 minutes

- I only got 2 hits in the proxy log: 
  - `http://…/index.html `
  - `http://…/robots.txt `

[GuilloOme]

After investigating, it seems that phantomjs is not happy with playing on the localhost network… I filled a bug #14808 to them.

[GuilloOme]

It's seems a behavior of QT (the lib used by phantomJS) ; unfortunately, it's not be "fixable"… (see this response)