Closed amprokop closed 6 years ago
This looks awesome!
Great document @amprokop! From the implementation standpoint, everything looks pretty straight forward. We should definitely invest some time to ensure we have good logging and monitoring in place so we can track auto assignments.
I like Option 1: Self-service button
more than Option 2: Auto-assign cases to judges from case storage
because it would allow a judge to control the assignment. I doubt there is a way for us to determine programmatically if a judge is on PTO. I also think we should provide a judge with an option to select a batch size.
Each judge has a set individual assignment batch size. This could be a uniform number for everyone, or could be determined by the size of the judge's team or other factors.
The sum of all judges' assignment batch sizes is the total assignment batch size. Let's say there are 90 judges with an average assignment batch size of 40 cases. This would give us a total assignment batch size of 3,600.
We will divide this assignment batch size among the dockets according to logic to be determined. Each docket will return cases according to its own logic. Initially, the only docket is the legacy docket, so the total docket batch size for the legacy docket will be 100% of the assignment batch size, or 3,600. Subsequent bullets only apply to the legacy docket.
Find the number of AOD cases ready to be assigned. Let's say there are 600 AOD cases. Subtracting this from the total docket batch size will give us the net docket range, 3,000. This number cannot be less than zero.
Retrieve the (net docket range)-oldest non-AOD cases that are ready to be assigned. In combination with the AOD cases that are ready to be assigned, these are our assignable cases. (If there are more AOD cases than the total docket batch size, we only use the (total docket batch size)-oldest AOD cases.)
Divide the assignable cases into hearing cases and genpop cases. Divide the hearing cases into cases that are assignable to this judge (the judge who has pushed the "I can haz appealz?" button), own hearing cases and cases that are not, other hearing cases. (A case with a panel decision would be assignable to multiple judges.)
Subtract the number of hearing cases, including AOD and non-AOD, from the total docket batch size to arrive at the net non-hearing docket batch size. Let's say there are 1,600 hearing cases; that would give us 2,000.
Divide the total number of non-hearing AOD cases by the net non-hearing docket batch size to arrive at the genpop AOD rate. Let's say there were 400 non-hearing AOD cases; that would give us a rate of 20%.
Now we have sufficient information to retrieve a number of cases equal to the individual assignment batch size, in this example 40. We do so in the following order:
@amprokop shall we close this ticket in favor of the 3rd draft, documented in #4985
sweet sg
On Thu, Apr 5, 2018 at 3:23 PM, Lauren Peterson notifications@github.com wrote:
@amprokop https://github.com/amprokop shall we close this ticket in favor of the 3rd draft, documented in #4985 https://github.com/department-of-veterans-affairs/caseflow/issues/4985
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/department-of-veterans-affairs/caseflow/issues/4954#issuecomment-379048752, or mute the thread https://github.com/notifications/unsubscribe-auth/ADm0y91P7ZrjWnLinM1X3QEYqQvZq3hgks5tlm8wgaJpZM4S7js8 .
Context
See https://github.com/department-of-veterans-affairs/appeals-design-research/issues/645
Overview
We'll develop a Case Storage auto-assignment model that:
Implementation
Assigning a case from case storage to a judge
VACOLS::CaseDocket.docket_date_of_nth_appeal_in_case_storage
Assignment parameters
By law, we need to prioritize AODs, and the board has slated hearing cases as a priority. Currently, AOD decision output is somewhere around 20%, which we see as a healthy figure. However, it appears that judges and case storage staff manually keep case category throughput in balance. If we develop a relatively simple auto-assignment model, but keep certain parameters configurable, we can control the throughput of various case categories centrally and monitor throughput to ensure it meets compliance and production goals.
Proposed configurable parameters:
Provisional auto-assignment logic
Auto-assignment will happen for one judge at a time.
Find and assign (batch size (Max % AOD)) AOD cases to a judge. Find and assign the minimum of (batch size - already assigned cases) and (batch size (Max % hearing)) hearing cases to a judge. Find and assign (batch size - already assigned cases) gen pop cases to the judge.
Example
Batch size is 30, AOD % is 20, Hearing % is 50.
A judge requests a batch of 30 cases.
The judge has 25 hearing cases awaiting decision, there are 500 AOD cases currently on docket, and 7000 genpop cases. The judge requests a batch of cases from case storage.
30 .2 = 6, so 6 AOD cases are assigned to the judge. 30 .5 = 15 is the max # of hearing cases the judge can receive in a batch. The judge has 30 - 6 = 24 cases that can still be assigned, so 15 hearing cases are assigned to the judge. 30 - 6 - 15 = 9, so 9 genpop cases are assigned to the judge.
Monitoring
To validate our model is behaving as we intend and so we can watch for any needed adjustments, we'll need to track the flow of all types of cases to judges separately. However, in the end, the top-line metric we care about is decision output, not case assignment. We'll want to also ensure descision output by case type is surfaced and easily available. We might opt to do this in Datadog at first, but ideally we'd make this easy to view for BVA employees as well, and ensure Tableau or a business metric tool can surface necessary data.
AMA appeal preparation
The business requirements for AMA case storage auto-assignment are not yet clear. It may be the case that separate judge/attorney teams exclusively work AMA appeals.
In this case, we could just create a separate case storage auto-assignment code path with its own configuration variables.
Additionally, we want to make it easy to track case storage assignments. We'll defer a final decision until later, but we'll consider creating a CaseStorageAssignment record for all appeals, so we ensure we're tracking assignments in Caseflow.
Specialty case teams
Business requirements for specialty case teams are not clear yet, but adding specialty case teams may significantly increase the complexity of this effort.
A few options:
These questions probably deserve their own separate spec.
Manual assignment
It's likely that there will be assignment hiccups. When we initially roll out case storage auto-assignment, we should ensure we can manually tweak case assignments as needed. At first, this can just be through a method exposed to the Rails console that a system admin engineer can run. This manual method must be able to support switching cases from judge to judge.
User interface
We'll weigh two different approaches for triggering the assignment of cases from case storage to judges.
Option 1: Self-service button
When a judge loads Caseflow Queue and views the appeals assigned to them, we could present the judge a "Fetch more from Case Storage" button. Upon clicking a button, we'd assign a batch of case storage cases to the judge, and they'd appear in the Queue list view immediately after assignment is complete.
Goal: Judges should understand the methodology behind automatic case allocation
Goal: Cases should not be allocated to attorneys and judges on PTO/leave
Goal: Cases are only allocated to active staff
Goal: Cases should not sit on judges plates to assign or to sign for long times
Option 2: Auto-assign cases to judges from case storage
We wouldn't need any UI for this option. We could approach a full automation option in several ways. Two that come to mind:
A recurring job could run on some interval and ensure each judge has at least N cases on their plate to assign to attorneys.
Each time a judge assigns an appeal to an attorney, if the judge has less than N cases on their plate, we add a batch of new cases from case storage
However, a fully automated option will create some challenges and risks.
Goal: Judges should understand the methodology behind automatic case allocation
Goal: Cases should not be allocated to attorneys and judges on PTO/leave
Goal: Cases are only allocated to active staff
Goal: Cases should not sit on judges plates to assign or to sign for long times
Test Plan
Using FACOLS, we'll be able to thoroughly test the Case Storage assignment to judge logic locally.
Rollout Plan
We can develop this in parallel with the judge queue, but we shouldn't roll it out until we roll out the judge queue. We could test this process change with one judge at first by putting it behind a feature toggle.