getall may not work with low memory systems

[anzenketh]

Code Used: https://github.com/EtienneLamoureux/durmand-scriptorium/blob/master/docs/COLLECTION_CONSUMER.md#getall

This returned memory error: $ php test.php PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 72 bytes) in /home/user/projects/gw2/gw2-items/vendor/guzzlehttp/guzzle/src/functions.php on line 303 PHP Stack trace: PHP 1. {main}() /home/user/projects/gw2/gw2-items/test.php:0 PHP 2. Crystalgorithm\DurmandScriptorium\v2\collection\CollectionConsumer->getAll() /home/user/projects/gw2/gw2-items/test.php:11 PHP 3. Crystalgorithm\DurmandScriptorium\v2\collection\CollectionConsumer->getAllPages() /home/user/projects/gw2/gw2-items/vendor/crystalgorithm/durmand-scriptorium/src/v2/collection/CollectionConsumer.php:40 PHP 4. Crystalgorithm\DurmandScriptorium\Consumer->convertResponseToArray() /home/user/projects/gw2/gw2-items/vendor/crystalgorithm/durmand-scriptorium/src/v2/collection/CollectionConsumer.php:77 PHP 5. GuzzleHttp\Message\Response->json() /home/user/projects/gw2/gw2-items/vendor/crystalgorithm/durmand-scriptorium/src/Consumer.php:83 PHP 6. GuzzleHttp\json_decode() /home/user/projects/gw2/gw2-items/vendor/guzzlehttp/guzzle/src/Message/Response.php:136 PHP 7. json_decode() /home/user/projects/gw2/gw2-items/vendor/guzzlehttp/guzzle/src/functions.php:303

[EtienneLamoureux]

This is indeed troublesome. I'll add a way to reliably fetch the pages one by one. I'll at least allow systems with memory limitations to be able to get the full collections.

A quick workaround would be to increase (double or quadruple) the value of Crystalgorithm\DurmandScriptorium\utils\Constants::MEMORY_LIMIT_IN_BYTES (Settings::MEMORY_LIMIT_IN_BYTES if you are using the dev branch). I'll have to precisely diagnose of much memory the biggest call to the API needs.

[EtienneLamoureux]

After some investigation. there is definitely a memory problem with the batch requests. The response from a call to https://api.guildwars2.com/v2/items?page=0&page_size=200 weights 112 014 bytes, so there is no way that 193 of those exceed the memory limit (even if it does mean that it weights in at nearly 22MB of raw data). I'll continue to look into it.

[EtienneLamoureux]

Looks like PHP's array_merge and json_decode with their pass-by-copy mechanics, are responsible for the memory overflow. Also, PHP arrays aren't the most memory-effective structures... I'll have to implement a custom iterator to get it working as intended.

[anzenketh]

Would it be a good idea to cache the responses as you receive them. This way you are not holding the entire database in a array.

[EtienneLamoureux]

Pure caching is not an option, since you want to always fetch fresh data from some endpoints (prices, listings...). But using a file system to manage the bigger collection may be a good idea.

Caching as a separate feature will come in due time!

[EtienneLamoureux]

With issue #7 closed and version 1.1.0 published, there's now a reliable way of fetching all pages, although it's not batched. This bug is next on my list.

[EtienneLamoureux]

Fixed as of v.1.1.1-beta1