search

techfromsage / tripod-php

Object Graph Mapper for managing RDF data in Mongo
MIT License
29 stars 4 forks source link

Discover impacted subjects job should handle large amounts of subjects #103

Open rgubby opened 8 years ago

rgubby commented 8 years ago

When a discover impacted subject job runs, it finds all subjects grouped by queue and adds them to an apply operation job.

https://github.com/talis/tripod-php/blob/d441fdf5184298afb9e741711e43d4694edfdbea/src/mongo/jobs/DiscoverImpactedSubjects.class.php#L129

In some instances, the $subjects param can contain a lot of data and when the apply operation job tried to turn the subjects into an array, it can easily run out of memory:

https://github.com/talis/tripod-php/blob/d441fdf5184298afb9e741711e43d4694edfdbea/src/mongo/jobs/ApplyOperation.class.php#L82

We should think about we can queue up an apply operation job without being completed bound by how much memory is allocated to a PHP process.