Feature/3959 bg monitoring fixed

[richard-jones]

• Issue: https://github.com/DOAJ/doajPM/issues/3959

Replaces PR: https://github.com/DOAJ/doaj/pull/2419

update bg monitoring settings for mission critical monitoring changes

Applies tighter monitoring constraints on the background jobs, as per this spreadsheet: https://docs.google.com/spreadsheets/d/1XWq-7R9hZmZsBA27eOgs9cO-SfJnLEE0dW50DVtKhpI/edit?gid=543591434#gid=543591434

This PR...

• [ ] has scripts to run
• [ ] has migrations to run
• [ ] adds new infrastructure
• [ ] changes the CI pipeline
• [ ] affects the public site
• [ ] affects the editorial area
• [ ] affects the publisher area
• [x] affects the monitoring

Developer Checklist

Developers should review and confirm each of these items before requesting review

• [x] Code meets acceptance criteria from issue
• [x] Unit tests are written and all pass
• [x] User Test Scripts (if required) are written and have been run through
• [x] Project's coding standards are met
  • No deprecated methods are used
  • No magic strings/numbers - all strings are in constants or messages files
  • ES queries are wrapped in a Query object rather than inlined in the code
  • Where possible our common library functions have been used (e.g. dates manipulated via dates)
  • Cleaned up commented out code, etc
  • Urls are constructed with url_for not hard-coded
• [x] Code documentation and related non-code documentation has all been updated
  • Core model documentation has been added to if needed: https://docs.google.com/spreadsheets/d/1lun2S9vwGbyfy3WjIjgXBm05D-3wWDZ4bp8xiIYfImM/edit
  • Events and consumers documentation has been added if needed: https://docs.google.com/spreadsheets/d/1oIeG5vg-blm2MZCE-7YhwulUlSz6TOUeY8jAftdP9JE/edit
• [x] Migation has been created and tested
• [x] There is a recent merge from develop

Reviewer Checklist

Reviewers should review and confirm each of these items before approval If there are multiple reviewers, this section should be duplicated for each reviewer

• [ ] Code meets acceptance criteria from issue
• [ ] Unit tests are written and all pass
• [ ] User Test Scripts (if required) are written and have been run through
• [ ] Project's coding standards are met
  • No deprecated methods are used
  • No magic strings/numbers - all strings are in constants or messages files
  • ES queries are wrapped in a Query object rather than inlined in the code
  • Where possible our common library functions have been used (e.g. dates manipulated via dates)
  • Cleaned up commented out code, etc
  • Urls are constructed with url_for not hard-coded
• [ ] Code documentation and related non-code documentation has all been updated
  • Core model documentation has been added to if needed: https://docs.google.com/spreadsheets/d/1lun2S9vwGbyfy3WjIjgXBm05D-3wWDZ4bp8xiIYfImM/edit
  • Events and consumers documentation has been added if needed: https://docs.google.com/spreadsheets/d/1oIeG5vg-blm2MZCE-7YhwulUlSz6TOUeY8jAftdP9JE/edit
• [ ] Migation has been created and tested
• [ ] There is a recent merge from develop

Testing

This tightens the monitoring on a number of bg jobs, so would be good to validate that this won't cause any major surge in unnecessary reporting.

Probably need to send it live and then adjust.

Deployment

Configuration changes

Adds new values in:

• BG_MONITOR_QUEUED_CONFIG
• BG_MONITOR_ERRORS_CONFIG
• BG_MONITOR_DEFAULT_CONFIG

Monitoring

This may trigger new types of "unstable" alerts in the background jobs reporting, so we should ensure that those alerts come through