Fatal error: Allowed memory size of 1073741824 bytes exhausted

[brechtvdv]

When generating arrivals/departures from NMBS, I get a memory problem in create_arrivals_departures.php.

MySQL uses 99% CPU and PHP uses 1.4GB memory when running with: ini_set('memory_limit', '-1');

Futher investigation is needed to process the data in batches.

[brechtvdv]

Instead of processing < date, serviceId > pairs, use < date, [serviceIds] >.

Important for performance of feeds that only have calendar_dates.txt

[pietercolpaert]

Make sure to stream out results instead of loading them in memory...

[brechtvdv]

This is still an issue for datasets like De Lijn