search

sjdirect / abotx

Cross Platform C# Web crawler framework, headless browser, parallel crawler. Please star this project! +1.
https://abotx.org
131 stars 23 forks source link

User agent #16

Closed f103mirage closed 6 years ago

f103mirage commented 6 years ago

the UA works correctly on ABOT, but in ABOTX it not working correctly

sjdirect commented 6 years ago

Tested both by xml config and setting on config object directly. Unable to reproduce. Can you give more info?

f103mirage commented 6 years ago

I setup userAgentString in app.config AS Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

ABOT use this user agent, but ABOTX when crawling ajax pages use another user agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1

sjdirect commented 6 years ago

Just deployed version 1.3.63 which should address this issue. Can you verify please?

https://www.nuget.org/packages/AbotX/1.3.63

f103mirage commented 6 years ago

perfect... thanks

sjdirect commented 6 years ago

Thanks for checking back to verify the fix.