search

serp-spider / search-engine-google

:spider: Google client for SERPS
https://serp-spider.github.io
Other
168 stars 61 forks source link

Mobile results are not parsed #32

Closed lmahesh5 closed 7 years ago

lmahesh5 commented 8 years ago

Google is now giving new format of results for first page for certain searches and that is not being parsed by parser it seems. Please check it. Normal results which use to come in first page : https://www.mondovo.io/files/serp-test/2016-09-06/127957-1.html New format results in first page : https://www.mondovo.io/files/serp-test/2016-09-06/127813-1.html When this new format results comes, parser is not able to parse. Note : Both formats is currently sent by google, so need to handle both the formats.

gsouf commented 8 years ago

hi @lmahesh5 I dont use to have this new format, This format is very similar with the mobile format.

Can you give me the instructions to get this format please ?

lmahesh5 commented 8 years ago

Hi @gsouf I am not able to simulate the steps, but when we do crawling, out of 100 searches 1 search will come like this in first page. I store the crawled content and so able to give you the reference url. But not able to simulate it.

gsouf commented 8 years ago

ok, maybe AB testing from google... but that's weird that they output this format. Maybe should we throw a unsuported format exception? Then you know that the page was not parsable and you can call it again and parse it in the good format. What do you thing?

lmahesh5 commented 8 years ago

Hi @gsouf In any case, we need that kind of (unsupported format) exception when we have crawled page, but results are not able to be fetched, to go and try for some alternatives. But this one, what if he implements for mobile results anytime soon, then our mobile results will start to fail.

gsouf commented 8 years ago

@lmahesh5 ok, I wont fix it now, but I plan to do it. In the interim if you still encounter this kind of result, or if you have a clue about how to trigger it, please feel free to add a message in this conversation

sameerpanjwani commented 8 years ago

As of now, these seem to be purely random results. Google has been testing this new format (http://searchengineland.com/google-continues-testing-new-search-results-interfaces-including-wider-card-styled-results-249971) and shows it only for the first page result.

As of now, since it seems random, we might be able to ignore. Google seems to be experimenting with this format and is usually an indication that this will be the format in the near future. They have always experimented with new upcoming interfaces, first at random to subsets of users, and then later if the feedback is positive, they implement it globally. We can perhaps implement it when it goes live globally although that might require large sudden changes.

gsouf commented 8 years ago

@sameerpanjwani thanks for the additional details.

If it's testing, and if it's implementing in the future, it's hard to know if the format will be exactly the same.

I need to find a way to handle this anyway : either implement it, either have a notice that the format is not the good. Your output might also help me to make the good decision, if you have some data about the frequency of these results that can also be helpful to make a choice.

lmahesh5 commented 8 years ago

@gsouf Now we have got the trend for this type of results. This is happening for mobile option results, when we use mobile user agents. (almost all mobile results) I have given sample user agents which we are using, by which you can simulate the error. Mozilla/5.0 (Linux; Android 6.0.1; SM-G920I Build/MMB29K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.98 Mobile Safari/537.36 Mozilla/5.0 (Linux; Android 6.0.1; ASUS_Z00AD Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.98 Mobile Safari/537.36

gsouf commented 8 years ago

@lmahesh5 looks like this only happens with mobile results. I didnt try yet, btw thanks for the ua samples.

I just renamed the issue to make it cleaner

gsouf commented 8 years ago

@lmahesh5 cards results that google is currently testing (not the mobile results) are now parsed, but only in the master branch. That will be available in the version 0.2.0

gsouf commented 7 years ago

I'm bumping to version 0.2. If some mobile results are not parsed feel free to open an issue. For the moment app results and graph results need to be implemented

MrZANO commented 5 years ago

Hello, everyone, Currently with user agent "Mozilla/5.0 (Linux; U; Android 4.4.2; en-us; SCH-I535 Build/KOT49H) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30"

Only the AD results are parsed, never the organic results

i'm used 0.4.8