returneksibir / yakala

a humble web crawler framework
3 stars 1 forks source link

Need for the user agent info #23

Closed rimbi closed 13 years ago

rimbi commented 13 years ago

Some web sites presents different versions of the pages (like mobile versions) depending on the user agent information in http connection request. A valid user agent info must be used in the connections.