Search snippet title truncates + description field

[kjellberg]

I'm not sure why, I think it is because of no-javascript or another parameter in the google query that truncates title. We want to fetch the full title, and not parts of it.

We also need to add the description field.

https://github.com/kjellberg/serp_scraper/blob/master/lib/engines/google.rb#L107

[hkleow]

Hey there, just discovered this repo. Was about to build my own SERP tracking with nokogiri, you saved my time! thanks! About the issue you mentioned, i think you might have to use some kind of headless browser like selenium to overcome the JavaScript issue.

Also i have a feature request, do u think it's possible to make google search with parameter such as coordinates(latitude, longitude) ? Because browser usually request for location access. Search results will be different base on coordinates. This is for serp tracking for local business. Not sure if we need to use headless browser for this

[kjellberg]

Heey @hkleow,

I'm not sure if this gem works 100% anymore since Google updated their layout and css selectors. Please let me know if it still works.

You can set custom parameters using this gem. For geolocation searches you should use the uule parameter instead of long/lat.

Example:

s = SerpScraper.new(engine: 'google', tld: 'com')

# Set location to Chicago,Illinois,United States
s.engine.parameter('uule', 'w+CAIQICIeQ2hpY2FnbyxJbGxpbm9pcyxVbml0ZWQgU3RhdGVz')

response = s.search('buy cars or whatever')
return response.organic

Check out this plugin to generate uule codes for your locations. Maybe you can find something similar in Ruby: dkarter/uule_grabber

[hkleow]

hey man i just tested it on my rails project, yes your gem is still working. Just read about uule, thanks i also found the gem for uule for ruby

https://github.com/takisawa/uule-ruby

Basically, if you don't use this gem also can. You may convert city string "Chicago,Illinois,United States" via BASE64

Here is article from moz how it works https://moz.com/ugc/geolocation-the-ultimate-tip-to-emulate-local-search

[hkleow]

Hey man, is it possible for me to set User Agent?