rachel3834
commented
11 months ago I've reviewed the OGLE harvester code and modified it so that it does not attempt to harvest the photometry for all OGLE targets but rather just the set for which it was configured to run. I also reduced the defaults so that it selects a random 100 events rather than 300. New events are prioritized, with older events being updated if there is room. Lastly I reduced the frequency with which the code runs to 4x daily from every 30min.
Tests on mop-dev indicate that this code runs cleanly
Even in MOP-prod, this harvester is dying with OOM errors.
Inspecting the code, it is still harvesting Gaia data for all targets instead of the reduced selection given. It also makes no check to see if this data has been harvested previously.
Since this is not the lightcurve data, the results are relatively static, so this function would be better executed once when the target is ingested.