Transfer-Encoding: chunked with proxy enabled

[migliori]

Hi,

I've been using search-engine-google for a while, worked perfectly until now but I've got a recent issue with proxies.

The Google scraper works fine on my localhost, but on the production server it throws an error 500: Unable to check javascript status

The scraped results come with dom => textContent starting with "ncoding Transfer-Encoding: chunked"

I put a simple test online here: https://www.hack-hunt.com/scraping-simple-test.php The code is the code of your example here: http://serp-spider.github.io/documentation/search-engine/google/#installation

I just added: $proxy = Proxy::createFromString('https://xxx:proxy@ip'); $browser->setProxy($proxy);

It works fine on localhost, or on production server if I remove the proxy, but it fails on production with proxy.

Not sure if the issue comes from my server or search-engine-google.

Any help much appreciated, thanks

[LunarDevelopment]

Can you post the contents of your local composer.json and production composer.json & composer.lock ?

On Fri, 5 Oct 2018 at 09:28 Gilles Migliori notifications@github.com wrote:

Hi,

I've been using search-engine-google for a while, worked perfectly until now but I've got a recent issue with proxies.

The Google scraper works fine on my localhost, but on the production server it throws an error 500: Unable to check javascript status

The scraped results come with dom => textContent starting with "ncoding Transfer-Encoding: chunked"

I put a simple test online here: https://www.hack-hunt.com/scraping-simple-test.php The code is the code of your example here: http://serp-spider.github.io/documentation/search-engine/google/#installation

I just added: $proxy = Proxy::createFromString('https://xxx:proxy@ip'); $browser->setProxy($proxy);

It works fine on localhost, or on production server if I remove the proxy, but it fails on production with proxy.

Not sure if the issue comes from my server or search-engine-google.

Any help much appreciated, thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/serp-spider/search-engine-google/issues/112, or mute the thread https://github.com/notifications/unsubscribe-auth/AKNqM3yup7uJ31PvQlGsf7i29tHAmHx7ks5uhxgogaJpZM4XJx4q .

[migliori]

composer.zip

same on local & server

[LunarDevelopment]

Hm, looks alright - can you check that your production server IP is whitelisted / isn't blocked with the proxy service ?

On Fri, 5 Oct 2018 at 09:39 Gilles Migliori notifications@github.com wrote:

composer.zip https://github.com/serp-spider/search-engine-google/files/2449573/composer.zip

same on local & server

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/serp-spider/search-engine-google/issues/112#issuecomment-427289573, or mute the thread https://github.com/notifications/unsubscribe-auth/AKNqMyZ_H7fZB4KOi7Nn2t3aBW7ZY7kqks5uhxrPgaJpZM4XJx4q .

[gsouf]

@migliori Please check your CURL version. If curl version is not the same on the server, please try to upgrade and let us know what's going on.

[migliori]

No, it isn't, if you open https://www.hack-hunt.com/scraping-simple-test.php you'll see the string added before Google content: `public 'textContent' => string 'ncoding Transfer-Encoding: chunked

simpsons - Recherche Google(function(){window.google=...`

I suspected that headers could be added by Apache pagespeed module, but tried to disable it without success.

I can't change my PHP Curl version, it's built-in with PLESK PHP. version =>7.26.0 ssl_version => OpenSSL/1.0.1t libz_version => 1.2.7

I just tested with nginx instead of apache: same result.

[gsouf]

@migliori not php-curl, just curl itself. Run curl --version

[migliori]

I already did it: apt-get update && apt-get install curl libcurl curl --version curl 7.26.0 (x86_64-pc-linux-gnu) libcurl/7.26.0 OpenSSL/1.0.1t zlib/1.2.7 libidn/1.25 libssh2/1.4.2 librtmp/2.3 Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp Features: Debug GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

[gsouf]

Your version of curl is very old. Try to upgrade to version 7.61 and see if it works.

Additionally curl <7.48 has issue with cookies, preventing SERPS to work correctly with cookies.

[migliori]

I'm in touch with my server provider & let you know if it's ok or not as soon as the upgrade is done - may take 1 or 2 days.

Thanks so much for your reactivity & help