[spec] Automating Case Storage assignment to judges: 3rd draft

[lpciferri]

Context

See https://github.com/department-of-veterans-affairs/appeals-design-research/issues/645 Earlier draft of this spec https://github.com/department-of-veterans-affairs/caseflow/issues/4954

Overview

We'll develop a Case Storage auto-assignment model that:

• removes any need for a Case Storage staff member to push cases to judges and eliminates lag time
• allows the Board to monitor and adjust the throughput of different types of cases so it can remain compliant & prevent logjams
• will leave room to add three types of AMA appeals
• will allow us to add specialty case teams. Specialty cases can be distributed directly to attorneys, bypassing judges. This is out of scope for this spec.

Implementation

Chris's drafty draft proposal for auto-assignment logic:

• Each judge has a set individual assignment batch size. This could be a uniform number for everyone, or could be determined by the size of the judge's team or other factors.

• The sum of all judges' assignment batch sizes is the total assignment batch size. Let's say there are 90 judges with an average assignment batch size of 40 cases. This would give us a total assignment batch size of 3,600.

• We will divide this assignment batch size among the dockets according to logic to be determined. Each docket will return cases according to its own logic. Initially, the only docket is the legacy docket, so the total docket batch size for the legacy docket will be 100% of the assignment batch size, or 3,600. Subsequent bullets only apply to the legacy docket.

• Find the number of AOD cases ready to be assigned. Let's say there are 600 AOD cases. Subtracting this from the total docket batch size will give us the net docket range, 3,000. This number cannot be less than zero.

• Retrieve the (net docket range)-oldest non-AOD cases that are ready to be assigned. In combination with the AOD cases that are ready to be assigned, these are our assignable cases. (If there are more AOD cases than the total docket batch size, we only use the (total docket batch size)-oldest AOD cases.)

• Divide the assignable cases into hearing cases and genpop cases. Divide the hearing cases into cases that are assignable to this judge (the judge who has pushed the "I can haz appealz?" button), own hearing cases and cases that are not, other hearing cases. (A case with a panel decision would be assignable to multiple judges.)

• Subtract the number of hearing cases, including AOD and non-AOD, from the total docket batch size to arrive at the net non-hearing docket batch size. Let's say there are 1,600 hearing cases; that would give us 2,000.

• Divide the total number of non-hearing AOD cases by the net non-hearing docket batch size to arrive at the genpop AOD rate. Let's say there were 400 non-hearing AOD cases; that would give us a rate of 20%.

• Now we have sufficient information to retrieve a number of cases equal to the individual assignment batch size, in this example 40. We do so in the following order:

  • AOD cases from the own hearing cases, in time pending order order. Let's say there were 2 such cases; leaving us with 38 slots.
  • Other cases from the own hearing cases, in docket order. Let's say there were 18 such cases, leaving us with 20 slots.
  • If there are still slots remaining, multiply the number of slots by the **genpop AOD rate. The 20 remaining slots, times 20%, gives us 4. So we'll assign 4 genpop AOD cases, in time pending order, leaving us with 16 slots.
  • Any remaining slots are given to non-AOD genpop cases. So we'll assign 16 genpop non-AOD cases, in docket order.
  • If there are not enough non-hearing cases to fill the remaining slots, we'll look outside the assignable cases we've retrieved and find the next n cases that are either own hearing cases or non-hearing cases, up to either the total number of remaining slots or until there are no more cases older than the Board's docket month.

Monitoring

To validate our model is behaving as we intend and so we can watch for any needed adjustments, we'll need to track the flow of all types of cases to judges separately. However, in the end, the top-line metric we care about is decision output, not case assignment. We'll want to also ensure descision output by case type is surfaced and easily available. We might opt to do this in Datadog at first, but ideally we'd make this easy to view for BVA employees as well, and ensure Tableau or a business metric tool can surface necessary data.

Manual assignment

It's likely that there will be assignment hiccups. When we initially roll out case storage auto-assignment, we should ensure we can manually tweak case assignments as needed. At first, this can just be through a method exposed to the Rails console that a system admin engineer can run. This manual method must be able to support switching cases from judge to judge.

User Interface: Self-service button

When a judge loads Caseflow Queue and views the appeals assigned to them, we could present the judge a "Fetch more from Case Storage" button. Upon clicking a button, we'd assign a batch of case storage cases to the judge, and they'd appear in the Queue list view immediately after assignment is complete.

Goal: Judges should understand the methodology behind automatic case allocation

• The button would require minimal training and explanation

  Goal: Cases should not be allocated to attorneys and judges on PTO/leave

• Judges are likely in the best position to know who is on PTO/leave, and they won't request cases for themselves while they're on leave

  Goal: Cases are only allocated to active staff

• Cases will not be assigned to judges unless they click a button, and judges will know not to assign cases to inactive attorneys

  Goal: Cases should not sit on judges plates to assign or to sign for long times

• A "Fetch more from Case Storage" button could be disabled until after a judge has assigned all existing cases

Test Plan

Using FACOLS, we'll be able to thoroughly test the Case Storage assignment to judge logic locally.

Rollout Plan

We can develop this in parallel with the judge queue, but we shouldn't roll it out until we roll out the judge queue. We could test this process change with one judge at first by putting it behind a feature toggle.

[lpciferri]

@cmgiven @amprokop

Today, Senior Counsel request gen pop cases from case storage so they can assign overtime cases to attorneys in their chief group. This is currently not a duty judges are authorized to do, and it would be a larger Board leadership decision to ask judges, instead of senior counsel, to do this.

Some Senior Counsel (Jebby) also request AOD and CAVC cases to distribute evenly among their chief group. We are eliminating their need to do this with the logic listed above for automatic case assignment, and I've been questioning the efficiency of senior counsel and others doing so much active "case management", so I initially was thinking we could do the same with overtime cases

But posing the question to you both - what if we added this senior counsel caveat to our automatic case assignment logic?

Proposed requirements

• Senior Counsel can request cases from case storage in Caseflow, just like we are proposing for judges.
• When Senior Counsel requests cases, they are assigned x number of gen pop cases - defined as non-hearing or retired judge hearing, non-AOD cases.

[cmgiven]

That would be pretty easily incorporated. I would suggest just not changing the total batch size used for calculating the AOD rate, as it seems like this will be more sporadic.

The other option, though, would be to enable the lawyer to request the next available genpop case directly. Depending on what we want, that could either be never AOD, the same probability of AOD as judge-requested cases, or always AOD if genpop AOD is available.

[lpciferri]

Interesting thought. Given the approval process for overtime cases, I'm not sure the Board would buy into attorneys requesting overtime cases directly. @nicholasholtz what do you think?

[lpciferri]

potential wrinkle - chief judges will have to request cases because they have to sign 5 cases per week (compared to the VLJ quota of 20 cases per week), but they don't have any attorneys. how will we define their batch size?

[cmgiven]

We can handle that. We'll just define their batch size at 5, and add (number of chiefs * 5) to the total batch size. Or we can do 10 if they prefer more of a biweekly schedule.