Open hroberts opened 4 years ago
I think this is because they have been OOMing constantly as 100k stories (the default chunk size) is quite a bit to load into RAM at once.
I've increased the RAM limit from 4 GB to 8 GB and they seem to be slowly catching up. If it doesn't fix the issue, we can also reduce the chunk size to 50k or so stories and just import more often.
the jobs were still dying every 20 minutes or so, so I decreased the job size to 20k. it is catching up now.
On Tue, Nov 26, 2019 at 1:23 PM Linas Valiukas notifications@github.com wrote:
I think this is because they have been OOMing constantly as 100k stories (the default chunk size) is quite a bit to load into RAM at once.
I've increased the RAM limit from 4 GB to 8 GB and they seem to be slowly catching up. If it doesn't fix the issue, we can also reduce the chunk size to 50k or so stories and just import more often.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_berkmancenter_mediacloud_issues_621-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAAN66T3LLF4YQOFREYRLKFLQVVZRHA5CNFSM4JGJ5ZXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFHFBFI-23issuecomment-2D558780565&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=0c5FW2CrwCh84ocLICzUHjcwKK-QMUDy4RRw_n18mMo&m=JB7cfAgBoQmjVneNDvn6CamTHh-eN5SW0Fb5qRz_goY&s=bbWx9tQY7CIVx4DygRMK8eUx-E6aE9amdEOV1qtB9g4&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAN66T2RAAOA5OD3YZTUU53QVVZRHANCNFSM4JGJ5ZXA&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=0c5FW2CrwCh84ocLICzUHjcwKK-QMUDy4RRw_n18mMo&m=JB7cfAgBoQmjVneNDvn6CamTHh-eN5SW0Fb5qRz_goY&s=NAU7bc0vFMz1UQk0NyRIVY3AmEWAqNMosBA4aM9R9hc&e= .
solr imports are very slow.
here are the last ten imports and the size of the import queue:
The current container log shows a bunch of entries like this:
I think an ancillary part of the problem is that the solr imports are slow enough that the occasional deployment resets the process and requires starting again.