Analysis to process assessments in the queue

[JasonCTang]

Describe the task Analysis to trigger Camunda assessment workflows using queues. Right now, most of the time, the assessment/reassessment records are created and the call to the queue happens right away. If the queue call fails or the queue is cleared there is no other attempt to start the workflow again.

Acceptance Criteria

• [x] Strategy to trigger the workflow for the different scenarios.
  • Student application submission.
  • Scholastic change
  • Bulk withdrawal
  • Student appeal
  • Bulk offering change
  • Application offering change
• [x] Ensure that we are not triggering the workflow multiple times for an assessment. Whatever the duration between the scheduler runs, only trigger workflow for assessments that have not been triggered within that time period.
  • [x] On the occasion where the workflow is called twice, check how to cancel the workflow in a nice way to avoid generating an unnecessary Camunda incident.
• [x] Define how exclusive the execution of the workflow needs to be. Can we safely execute in parallel workflows for the same student for different applications?
• [x] How to handle multiple assessment records creation for the same student application? Right now some features (Student Appeals/Scholastic Standing) verify if there is already another workflow under execution for the same application assertAllAssessmentsCompleted.
  • Should we allow multiple assessment records creation for the same application?
  • How the "current assessment" relationship in the applications table be updated?
  • Should we ensure that an assessment will be calculated for every single workflow?
• [x] Create an epic to group the upcoming tickets needed to have the Camunda workflow-related queues ready for production MVP as much as possible.

[andrewsignori-aot]

Analysis

• Camunda workflows will no longer be invoked when new student_assessment records are created. Triggering a workflow will be a responsibility of a new scheduler.
• All the student_assessment records will be associated with the application's current assessment alongside its creation, which means that the current assessment will always point to the most recent assessment record.
• Multiple student_assessment records can be created and all will be executed in the creation order.
• The student_assessment records for the same application should be executed in order, but different applications for the same students can be executed in parallel.
• Two new schedulers will be created.
  • Assessment workflow execution (every few seconds): used to start and cancel workflows.
  • Assessment workflow execution retry (every few hours): used to retry start and cancelation operations that are taking too long.
• Two existing queues will be still used.
  • start-application-assessment
  • cancel-application-assessment
• New status will be created for the student_assessment table to allow its control by the schedulers.
  • Add also an extra column, similar to sims.applications.application_status_updated_on to track the date that the status was updated (e.g. student_assessment_status_updated_on).
• Add a new column current_processing_assessment_id to the sims.applications to be updated by the scheduler. Every time that a record is queued for execution this new column will also be updated.

New status for the assessment

• Created: when the student_assessment is created.
  • Set by the code creating the record. Can be the default value for the new column.
• Queued: when the student_assessment is queued to be executed on Camunda. Change the status to Queued and call the queue after, in this way, if the Camunda call fails it will retry later because the status is already set.
  • Set by the new scheduler: Assessment workflow execution.
• In progress: when the workflow id is associated with the student_assessment.
  • Se from a Camunda worker (e.g. associate-workflow-instance-task).
• Completed: at the end by a dedicated worker.
  • Set by a new Camunda worker to be created.
• Cancellation requested: when a workflow needs to be canceled, for instance, when the student cancels the application or the "Bulk offering update" happens and the application is in progress. The student_assessment can have the status updated to Cancellation requested to be selected by the specific queue and then have its status finally updated to Cancelled.
  • Set by the application code requesting the cancellation.
• Cancellation queued: create a new item in the queue cancel-application-assessment. This queue should also be changed to update the student_assessment to Cancelled.
  • Set by the new scheduler Assessment workflow execution.
• Cancelled: when the workflow is canceled or a call to Camunda to cancel the workflow returns a "Not found".
  • Set by the existing queue cancel-application-assessment.

Regular Assessment Lifecycle

For a single workflow execution, the assessments will progress as shown below.

Stage 1

The student_assessment record is saved to the database with the default Created status.

Stage 2

The record with Created status:

1. is selected by the assessment-workflow-execution-queue scheduler.
2. its status is updated to Queued.
3. Its id is also set in the application's current_processing_assessment_id column.
4. it is added to the start-application-assessment queue.

Stage 3

1. The workflow has its execution started by Camunda.
2. The associate-workflow-instance worker, besides setting the workflow_id in DB (already happening), also sets the workflow status to In progress.

During this stage, alongside the associate-workflow-instance execution, if another workflow was already set to process the student_assessment, use the job.cancelWorkflow(); to terminate its execution.

Stage 4

A new workflow worker is created at the end of the workflow to have the student_assessment record changed to completed.

Queued Assessment Lifecycle

On the occasion when more than one student_assessment record is created, a scheduler will coordinate their ordered execution. At any given time, assessments for one particular application can have 0 to 1 "Queued" records and 1 to n created records waiting to be processed. Every time that a student_assessment record is queued its id is also set in the application's current_processing_assessment_id column.

Stage 1

Stage 2

Stage 3

Assessment Cancellation Lifecycle

The cancellation of a workflow is currently executed by the queue cancel-application-assessment. The Offering bulk change, for instance, executes also the bulk cancellation of any In progress application. To allow the cancellation to also be handled by the schedulers, we can use the assessment status Cancellation requested to update any student-assessment to be canceled. The scheduler can then look for these records and try to execute the process and set the student-assessment as canceled. The workflow cancellation process happen immediately and right now it is processed by cancel-application-assessment.processor.ts.

Schedulers

Assessment workflow execution (assessment-workflow-execution-queue)

Coordinate the execution of all the student_assessments to ensure its execution. Allow multiple student_assessments to be created and ensures their ordered execution and its association with the application's current assessment.

Execution interval

Executed every minute or some short interval to be discussed.

Target records

• Any application where the current assessment is in Created status.
• Any application where the current assessment is in Completed status and there is 1 to n Created assessment (waiting to be executed). Only the oldest assessment record in Created status will be selected for each application.

Process description

• If the current assessment is in Created status. 1 - Change the status to Queued. 2 - Queue the assessment for execution (already existing start-application-assessment).
• If the current assessment is NOT in Created status. 1 - Change the status to Queued. 2 - Update the application's current assessment to point to this record. 3 - Queue the assessment for execution (already existing start-application-assessment).

Assessment Workflow execution retry (assessment-workflow-execution-queue-retry)

Checks for any stalled Queued student_assessments that did not start after a long period.

Execution interval

Executed every few hours or some long interval to be discussed.

Target records

Any application where the current assessment is in Queued status for some long period of time (e.g. 6 hours).

Process description

Queue the assessment for execution (already existing start-application-assessment).

Refactors

Schedulers should trigger workflows

Stop triggering the start-application-assessment and the cancel-application-assessment from the applications. Only the new schedulers would be responsible for triggering the Camunda workflows and canceling them. We can consider maybe letting the "Original Assessment" trigger the assessment for a better user experience once the application is submitted.

Current points triggering the assessment workflow:

• Application submission application.students.controller.ts
• Student Appeal Request student-appeal.aest.controller.ts
• Offering bulk change education-program-offering.service.ts
• Scholastic standing student-scholastic-standings.institutions.controller.ts

Current points for canceling the assessment workflow:

• Offering bulk change education-program-offering.service.ts

Notifications

Some assessment creations, like Scholastic Standing, also create notifications for the students. We should consider triggering these notifications when the specific assessment becomes the application's current assessment.

Other refactors

• Check if the code current inside method startAssessment can be moved to the start-application-assessment queue.
• Remove the no longer needed code assertAllAssessmentsCompleted.