Fewer porter jobs/callbacks

PRX / cms.prx.org

CMS API for PRX

https://cms.prx.org

GNU Affero General Public License v3.0

4 stars 2 forks source link

Open cavis opened 4 years ago

cavis commented 4 years ago

CMS worker gets real bogged down during feed imports. Attempt to minimize redundant jobs/callbacks. Crazy ideas:

Make sure we only send each audio_file / image to Porter once. We appear to be sending duplicates for segments > 1 right now.
Make sure Porter callbacks don't change audio_file timestamps when they don't need to. Since those will generate announcements to feeder and more CMS-search-index jobs.
Look into upgrading shoryuken for better parallelization

cavis commented 4 years ago

Also per @kookster ... the audio_file acts_as_list code may be causing the specific issue where we reprocess segments > 1.

cavis commented 4 years ago

There may also be a problem with the podcast importer unlocking the Feeder podcast too early. Seeing a ton of publish feed jobs over there.