Scraping in google returns no results (chinese queries)

[richardzhu64]

Hello,

I'm working on a project where I am trying to search google with a few keywords in Chinese. Currently, there are no error messages being shown in the terminal, and the return value shows "no_results": false - however, there are no results and the num_results is empty.

My scraping scripti is here:

`const se_scraper = require('se-scraper');

(async () => { /* let browser_config = { test_evasion: true, output_file: 'scraped_urls_manual_google.json', search_engine: "baidu", sleep_range: "",//[70,150], random_user_agent: true, debug: true, keywords: clean_terms }

se_scraper.scrape(browser_config, (err, response) => {
    if (err) { console.error(err) }

    /* response object has the following properties:

        response.results - json object with the scraping results
        response.metadata - json object with metadata information
        response.statusCode - status code of the scraping process

    console.dir(response.results, {depth: null, colors: true});
});
*/
let browser_config = {
    test_evasion: true,
    debug_level: 1,
    output_file: 'scraped_urls_manual_google.json',
    log_http_headers: false,
    log_ip_address: true,
    // whether to prevent images, css, fonts and media from being loaded
    block_assets: true,
    sleep_range: "",//[70,150],
    random_user_agent: true,
    apply_evasion_techniques: true,
    debug: true,
};

let scrape_job = {
    search_engine: 'google',
    keywords: clean_terms,
    num_pages: 1,
    // add some cool google search settings
    google_settings: {
        gl: 'hk', // The gl parameter determines the Google country to use for the query.
        hl: 'en', // The hl parameter determines the Google UI language to return results.
        start: 0, // Determines the results offset to use, defaults to 0.
        num: 10, // Determines the number of results to show, defaults to 10. Maximum is 100.
    },
};

var scraper = new se_scraper.ScrapeManager(browser_config);

await scraper.start();

var results = await scraper.scrape(scrape_job);

console.dir(results, {depth: null, colors: true});

await scraper.quit();

})();`

I get the following output when running the script:

{ "\"阿尔及利亚TAMANRASSET至In Salah饮用水供给工程项下水井项目 水井项目子项目3：2眼水井及35眼观测井的成井\"": { "1": { "num_results": "", "no_results": false, "effective_query": "", "right_info": {}, "results": [], "top_products": [], "right_products": [], "top_ads": [], "bottom_ads": [], "places": [], "time": "Tue, 06 Oct 2020 04:28:36 GMT" } }, "\"突尼斯斯菲西发水坝\"": { "1": { "num_results": "", "no_results": false, "effective_query": "", "right_info": {}, "results": [], "top_products": [], "right_products": [], "top_ads": [], "bottom_ads": [], "places": [], "time": "Tue, 06 Oct 2020 04:30:13 GMT" } }, "\"苏丹石油1/2/4区CANNER地面设施及长输管道项目\"": { "1": { "num_results": "", "no_results": false, "effective_query": "", "right_info": {}, "results": [], "top_products": [], "right_products": [], "top_ads": [], "bottom_ads": [], "places": [], "time": "Tue, 06 Oct 2020 04:32:35 GMT" } }, "\"阿尔及尔BAB EZZOUAR 1号地1577套住宅商业服务层部分的设计与施工\"": { "1": { "num_results": "", "no_results": false, "effective_query": "", "right_info": {}, "results": [], "top_products": [], "right_products": [], "top_ads": [], "bottom_ads": [], "places": [], "time": "Tue, 06 Oct 2020 04:34:47 GMT" } } }

I've also attached an image of the console while my script is running. Do we know how to resolve this issue where there aren't any apparent errors, but there are no results or links returned for when scraping Google? Previously I was able to do this successfully, but running it again seems to have broken something.

Thanks!

[przemek-nowicki]

Maybe cookie consent popup that shows up when cookies are not set block somehow getting the google search results.

[przemek-nowicki]

It actually seems that problem is in parse_async method of GoogleScraper class. The parsed html changed so query selectors need to be updated ie:

let serp_obj = {
 link: _attr(el, '.rc a', 'href'),
 title: _text(el, '.rc a h3'),
 snippet: _text(el, '.rc > div:nth-child(2) span span'),
 visible_link: _text(el, '.rc cite'),
 date: _text(el, '.rc > div:nth-child(2) span.f'),
};

[przemek-nowicki]

Hi @richardzhu64 I have created PR for this issue https://github.com/NikolaiT/se-scraper/pull/82/files Could you let me know if this fixed your google results as well?

[pankajjha]

Hi @przemek-nowicki It is working for few pages and for few it doesn't Here attached - it is working for alibaba.txt but it is not working for carloan.txt alibaba.txt carloan.txt

Also, are you able to fetch ads as for me none of them are showing ads

[przemek-nowicki]

Hi @pankajjha I fixed selectors for organic search results only.