Batch all sentinel operations

[dianabarsan]

Describe the issue Currently, Sentinel runs a number of tasks on a schedule. Some of these tasks query the database, searching for specific docs that need to be updated. The two examples I have are due_tasks and outbound. https://github.com/medic/cht-core/blob/master/sentinel/src/schedule/outbound.js#L29 https://github.com/medic/cht-core/blob/master/shared-libs/transitions/src/schedule/due_tasks.js#L41 Both of these (and maybe others too) run queries without limit over the whole database. They run smoothly when Sentinel is up to date and running correctly. If it fails or stops for whatever reason, and the queue becomes too large, we will be in an irredeemable situation where we would have to hotpatch instances in order to clear the queue.

Describe the improvement you'd like We should not have any unlimited database queries that are prone to have documents queued up. The selected limit should be a big enough number that it doesn't lengthen process periods with too long pause intervals but also be safe to execute under any circumstance. My initial though would be a limit of 1000, but that's just an idea.

Describe alternatives you've considered We could also consider refactoring these scripts so they operate in batches until their queue is finished, although it wouldn't necessarily be an improvement, as for long queues you could end up blocking Sentinel from executing other scheduled tasks for long periods.

[garethbowen]

Related to #6108

[SCdF]

Ready for AT in 6125-sentinel-batches

AT:

• Large numbers of scheduled items (scheduled tasks, outbound requests etc) shouldn't block other scheduled items

[newtewt]

@SCdF @dianabarsan I was able to create a bunch of outbound tasks and they triggered fine. I'm not real sure how to generate a ton of schedueld tasks and have them process. Any ideas?

[dianabarsan]

I would suggest you create a schedule that would add multiple schedule messages in the current week and an SMS form with a registration that generates these messages, then just spam the Admin -> SMS -> Test Message with that form.

~I'll update this post with a sample config shortly.~

Sample config:

{
  "transitions": {
    "registration": true,
    "update_clinics": true
  },
  "forms": {
    "ADD_SCHED": {
      "meta": {
        "code": "ADD_SCHED",
        "icon": "icon-danger",
        "translation_key": "report sickness"
      },
      "fields": {
        "notes": {
          "labels": {
            "short": {
              "translation_key": "notes"
            }
          },
          "position": 0,
          "type": "string",
          "length": [
            1,
            100
          ],
          "required": false
        }
      },
      "public_form": false,
      "use_sentinel": true
    }
  },
  "schedules":
  [
      {
      "name": "Busy",
      "translation_key": "Busy",
      "summary": "",
      "description": "",
      "start_from": "reported_date",
      "messages": [
        {
          "message": [{"content": "Message 1"}],
          "group": 1,
          "offset": "10 seconds",
          "send_day": "",
          "send_time": "",
          "recipient": "reporting_unit"
        },
        {
          "message": [{"content": "Message 2"}],
          "group": 1,
          "offset": "10 seconds",
          "send_day": "",
          "send_time": "",
          "recipient": "reporting_unit"
        },
        {
          "message": [{"content": "Message 3"}],
          "group": 1,
          "offset": "10 seconds",
          "send_day": "",
          "send_time": "",
          "recipient": "reporting_unit"
        },
        {
          "message": [{"content": "Message 4"}],
          "group": 1,
          "offset": "10 seconds",
          "send_day": "",
          "send_time": "",
          "recipient": "reporting_unit"
        },
        {
          "message": [{"content": "Message 5"}],
          "group": 1,
          "offset": "10 seconds",
          "send_day": "",
          "send_time": "",
          "recipient": "reporting_unit"
        }
      ]
    }
  ],
  "registrations": [
    {
      "form": "ADD_SCHED",
      "events": [
        {
          "name": "on_create",
          "trigger": "assign_schedule",
          "params": "Busy"
        }
      ]
    }
  ]
}

You can add multiple messages to the schedule, if you like, I added a random number of 5. You can just send ADD_SCHED from a known phone number. The delay of when the messages will be scheduled is 10 seconds, so after 10 seconds, you can restart Sentinel (or wait for max 5 minutes) and you should see all the scheduled tasks become pending. Also, don't forget to enable "update_clinics" and "registrations" transitions :)

[dianabarsan]

Also, if you create > 1000 scheduled tasks, it would process only 1000 per run.

[newtewt]

This is working now as expected, seeing the batched amounts go out. @dianabarsan feel free to merge and close.

[dianabarsan]

Merged.