FriendsOfPHP / Goutte

Goutte, a simple PHP Web Scraper
MIT License
9.26k stars 1.01k forks source link

Memory Leak while iterating over few thousand links #413

Closed dragonsnapx closed 4 years ago

dragonsnapx commented 4 years ago

I'm using Goutte 4.0.0, and when iterating over multiple thousand links using php artisan command, it quickly takes up memory and eventually throws a memory limit error. It throws this:

 PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 16777224 bytes) in /home/forge/project/vendor/symfony/browser-kit/History.php on line 38

In History.php line 38:

Allowed memory size of 536870912 bytes exhausted (tried to allocate 16777224 bytes) 

Here is my code:

$client = new Client();
foreach($links as $link){
    $crawler= $client->request('POST', $link);
    // Commit to Database
}

I tried running with a --env=prod but it doesn't work.

dragonsnapx commented 4 years ago
$crawler = $client->request('POST', 'http://fifaonline4.nexon.com/datacenter/PlayerPriceGraph'
                    , $params, array(), array(), null, false);

Set the $changeHistory = false to fix this.