search

department-of-veterans-affairs / caseflow

Caseflow is a web application that enables the tracking and processing of appealed claims at the Board of Veterans' Appeals.
Other
54 stars 18 forks source link

Automatic case distribution rollout plan #7838

Closed amprokop closed 5 years ago

amprokop commented 5 years ago

Related to https://github.com/department-of-veterans-affairs/caseflow/issues/5575.

  1. Backend of legacy auto-case-distribution
  2. Frontend of legacy auto-case distribution
  3. Manual testing
    • Test 10 near simultaneous distributions to 10 different judges. They should all complete successfully.
    • Ensure that FACOLS data exists to test case distribution locally.
  4. Create Auto Case Distribution Looker dashboard for monitoring and metrics
    • number of completed distributions per day
    • Mean time to completion of each distribution (created at - completed at)
    • [later] Count # of appeal ids in multiple distributions? (should always be zero)
    • [later] Mean proportion of priority non-genpop / non priority non-genpop / priority genpop / non-priority genpop cases in each distribution per day?
    • Priority pending: How many priority cases are waiting to be distributed at any given time?
    • Priority timeliness: How long does Caseflow take to distribute a new priority case?
    • Distribution diversity: Are we balancing priority cases among judges?
    • Docket efficiency: How deep in the docket does Caseflow have to look to find enough cases?
  5. Pilot legacy auto-case distribution for two judges
  6. Increase pilot size to ~20 judges
  7. Roll out legacy auto-case distribution to all judges 8. Integrate RAMP cases into auto-case distribution? (needs separate spec)
amprokop commented 5 years ago

@cmgiven — just got some thoughts down here since I was thinking through a few things. Feel free to edit or add.

cmgiven commented 5 years ago

I like the idea of doing 10 simultaneous distributions.

The looker dashboard should have some of the same metrics we used when simulating:

What are your thoughts on 8? Is running the round robin script a pain? My initial sense is that it's not worth doing further work as it has no real connection to how things will work after AMA day, but you know better here.

amprokop commented 5 years ago

Running the round robin script isn't too much of a pain, although it would be better to automate it!

The vague thought I had was that adding RAMP appeals to legacy case distribution could possibly be a step towards adding all AMA appeals. We could just add 1 RAMP appeal from the top of that queue to each legacy case distribution, and test a bit of the logic we'll need on February 14th (task creation, etc) that way.

It might be increased effort explaining/selling this to the Board, or maybe I'm overestimating how much of a halfway step it would be, though. Any thoughts?

cmgiven commented 5 years ago

I don't immediately see anything that advances us toward AMA case distribution. It's a single case, it uses different definitions of ready than post-2/14, etc.

amprokop commented 5 years ago

Got it, sounds good then 👍

lpciferri commented 5 years ago

Suggested two judges to DVC Osborne. Looking like Judges Amy Ishizawar and Paul Sorisio will participate.

amprokop commented 5 years ago

Cool. Continuing with testing on this today.

amprokop commented 5 years ago

Dumping some steps i took to test this locally here:

  1. fire up shoryuken shoryuken start -q caseflow_development_high_priority -R

  2. ensure that the StartDistributionJob is run with perform_later even in dev mode so it hits shoryuken, so we can test what happens when a bunch of jobs are queued up at the same time

  3. Run the following code

    
# clean up
JudgeTask.destroy_all && Distribution.destroy_all && DistributedCase.destroy_all  && VACOLS::Decass.destroy_all

destroy all hearings so those appeals become genpop / don't need to be assigned to a specific judge

VACOLS::CaseHearing.destroy_all

set all appeals to case storage and activated state

vacols_ids = VACOLS::Case.all.map(&:bfkey) VACOLS::Case.batch_update_vacols_location("81", vacols_ids) VACOLS::Case.update_all(bfmpro: "ACT")

find a batch of judges to assign - exclude acting judges

judges = Judge.list_all[0..10].select(&:judge_in_vacols?)

verify all judges have 3 attorneys on their teams

judges = judges.select { |judge| Constants::AttorneyJudgeTeams::JUDGES[Rails.current_env][judge.css_id].present? }

print the number of available ready/priority appeals

VACOLS::CaseDocket.counts_by_priority_and_readiness

returns => [{"n"=>130, "priority"=>1, "ready"=>1},

{"n"=>103, "priority"=>0, "ready"=>1}]

verify all 0s

judges.map { |judge| QueueRepository.tasks_for_user(judge.css_id).count }

judges.each { |judge| RequestStore[:current_user] = judge; Distribution.create!(judge: judge) }

judges.map { |judge| QueueRepository.tasks_for_user(judge.css_id).count }

returns [15, 15, 15, 15, 15, 15, 15, 6]

print the number of available ready/priority appeals

VACOLS::CaseDocket.counts_by_priority_and_readiness

returns => [{"n"=>8, "priority"=>1, "ready"=>0},

{"n"=>103, "priority"=>0, "ready"=>0},

{"n"=>122, "priority"=>1, "ready"=>1}

test that the same case was not distributed twice, will return true if no dupes

DistributedCase.all.map(&:case_id).uniq.count == DistributedCase.count

do one more distribution

judges = Judge.list_all[11..20].select(&:judge_in_vacols?) judges = judges.select { |judge| Constants::AttorneyJudgeTeams::JUDGES[Rails.current_env][judge.css_id].present? } judges.each { |judge| RequestStore[:current_user] = judge; Distribution.create!(judge: judge) }

judges.map { |judge| QueueRepository.tasks_for_user(judge.css_id).count }

returns [1, 1, 1, 1, 1, 1, 1, 1, 1]

<- we're out of nonpriority appeals and the priority target is 1 so this is expected

amprokop commented 5 years ago

@cmgiven — i poked around at this quite a bit locally and wasn't able to find any problems. 🎊 I think I'm satisfied with manual testing.

amprokop commented 5 years ago

@cmgiven @laurjpeterson — anything on your ends that we'd need to button up before starting the pilot of this? I'm pretty satisfied.

After thinking about it, I think we should create the Looker dashboard after the first 2 users make their first distributions, so we'll have data to view.

lpciferri commented 5 years ago

🎉 Excellent! The only thing left to do is to communicate to our two pilot judges and feature toggle. I'll let them know that they should stop requesting cases using the spreadsheet, and that they will need to assign out all of their cases in order to first see the functionality. I can send this email out today!

What is the feature toggle called?

Then, we can create the Looker dashboard. I'd like to pair with @cmgiven to create visualizations for each of these metrics.

cmgiven commented 5 years ago

Pairing sounds great. Feature toggle is automatic_case_distribution.

lpciferri commented 5 years ago

https://github.com/department-of-veterans-affairs/appeals-deployment/pull/1781