search

Poniverse / Pony.fm

The pony music community.
https://pony.fm/
GNU Affero General Public License v3.0
41 stars 15 forks source link

Workers keep dying D: #110

Open ghost opened 7 years ago

ghost commented 7 years ago

This has been a long standing issue for a while. Occasionally the workers responsible for processing tracks will suddenly fall over. There has been an attempt at alerting users that there's a problem by returning a 418 error when a track file is not found. There needs to be some sort of detection system to detect when these 418's are fired a few times and restart the workers

EventideGlow commented 7 years ago

@logic-dev Have you found anything interesting in the error logs when the workers crash? It would be nice to figure out what causes this, in addition to setting up more reliable monitoring.

The workers are currently monitored with Supervisord - sounds like our config for that in Ansible might need tweaking.

EventideGlow commented 7 years ago

This might be related to #114 - if there are issues with what the workers are doing, normally, those would be getting logged in the failed_jobs table.