BUG: Some cases are missing IrsPractitioner/PrivatePractitioner entities

[jimlerza]

Describe the Bug We had a field report of a docket clerk not being able to add a practitioner (that "should already be associated") to a case. When triaging this report, I found (with a huge assist from @rosiehsmith and @mrinsin) that there was no PrivatePractitioner entity in the case partition (resulting in the practitioner not being listed in DAWSON), but there was a UserCase entity in the user partition (resulting in the practitioner not being able to be added to the case in DAWSON).

In this one instance, I deleted the UserCase entity and @ttlenard instructed the docket clerk that reported this issue to try adding the practitioner to the case again. This was successful, and in the end we now have a UserCase entity in the user partition AND a PrivatePractitioner entity in the case partition.

Rather than wait for this to be reported again, I wrote a script to find other instances of missing practitioner entities in case partitions. That script can be found here. Because neither the practitioner entities nor the UserCase entities are indexed in Elasticsearch, this script is extremely inefficient. I have attached the script's output to this ticket. There are a total of 56 "missing" practitioner entities.

Business Impact/Reason for Severity Cases can potentially be calendared with an incomplete list of practitioners.

In which environment did you see this bug? Production, though it can also be reproduced in migration and test.

Who were you logged in as? Docket clerk

What were you doing when you discovered this bug? (Using the application, demoing, smoke tests, testing other functionality, etc.) Normal docket clerk workflow of examining cases

To Reproduce Steps to reproduce the behavior:

1. Log in to mig as a docket clerk
2. Navigate to docket number 681-21
3. Click on the 'Case Information' tab, then the 'Parties' subsection
4. Enter 'SJ1623' in the 'Add counsel' input box and click 'Search'
5. Check both petitioners' checkboxes and then click 'Add to case'
6. See a 'success' message, but note that the practitioner does not appear

Expected Behavior Counsel Jonathan Edward Strouse should have been listed as a practitioner on the case (in step 3).

Actual Behavior A PrivatePractitioner entity was not added in the case partition, presumably because a UserCase entity already existed in the user partition linking this practitioner to this case.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Cause of Bug, If Known

Process for Logging a Bug:

• Complete the above information
• Add a severity tag (Critical, High Severity, Medium Severity or Low Severity). See below for priority definition.

Severity Definition:

• Critical Defect Blocks entire system's or module’s functionality No workarounds available Testing cannot proceed further without bug being fixed.

• High-severity Defect Affects key functionality of an application There's a workaround, but not obvious or easy App behaves in a way that is strongly different from the one stated in the requirements

• Medium-severity Defect A minor function does not behave in a way stated in the requirements. Workaround is available and easy

• Low-severity Defect Mostly related to an application’s UI Doesn't need a workaround, because it doesn't impact functionality

Definition of Ready for Bugs(Created 10-4-21)

Definition used: A failure or flaw in the system which produces an incorrect or undesired result that deviates from the expected result or behavior. (Note: Expected results are use cases that have been documented in past user stories as acceptance criteria and test cases, and do not include strange behavior unrelated to use cases.)

The following criteria must be met in order for the development team to begin work on the bug.

The bug must:

• Be focused on solving a user problem
• Contain data for all fields in the bug template, so the team can pick it up and begin working immediately

Process: If the unexpected results are new use cases that have been identified, but not yet built, new acceptance criteria and test cases should be captured in a new user story and prioritized by the product owner.

If the Court is not able to reproduce the bug, add the “Unable to reproduce” tag. This will provide visibility into the type of support that may be needed by the Court. In the event that the Court cannot reproduce the bug, the Court will work with Flexion to communicate what type of troubleshooting help may be needed.

Definition of Done (Updated 4-14-21)

Product Owner

• [ ] Bug fix has been validated in the Court's test environment

Engineering

• [ ] Automated test scripts have been written
• [ ] Field level and page level validation errors (front-end and server-side) integrated and functioning
• [ ] Verify that language for docket record for internal users and external users is identical
• [ ] New screens have been added to pa11y scripts
• [ ] All new functionality verified to work with keyboard and macOS voiceover https://www.apple.com/voiceover/info/guide/_1124.html
• [ ] READMEs, other appropriate docs, JSDocs and swagger/APIs fully updated
• [ ] UI should be touch optimized and responsive for external only (functions on supported mobile devices and optimized for screen sizes as required)
• [ ] Interactors should validate entities before calling persistence methods
• [ ] Code refactored for clarity and to remove any known technical debt
• [ ] Deployed to the Court's test environment

[jimlerza]

• Output of find-cases-missing-practitioners script
• Missing entities from efcms-test-alpha as of 2021-12-29

[jimlerza]

I was only looking for missing PrivatePractitioner or IrsPractitioner entities where corresponding UserCase entities exist. I am not sure if we are ALSO missing UserCase entities where corresponding PrivatePractitioner or IrsPractitioner entities exist. Further analysis of the root cause may shed light on whether we should check for missing UserCase entities.

[jimlerza]

If it helps for testing, here is the same data in a human-readable format with docket numbers for the cases and bar numbers for the practitioners:

Docket number 30094-21 is missing IRS practitioner Thomas Alan Friday (Bar number FT0140) Docket number 2177-18 is missing IRS practitioner Valerie Vlasenko (Bar number VV0029) Docket number 27377-21 is missing IRS practitioner William B. Belvin (Bar number BW0906) Docket number 19229-16 is missing IRS practitioner Audrey Marie Morris (Bar number MA0556) Docket number 19230-16 is missing IRS practitioner Audrey Marie Morris (Bar number MA0556) Docket number 3426-19 is missing IRS practitioner Audrey Marie Morris (Bar number MA0556) Docket number 5917-18 is missing IRS practitioner Audrey Marie Morris (Bar number MA0556) Docket number 9183-17 is missing IRS practitioner Audrey Marie Morris (Bar number MA0556) Docket number 9244-18 is missing IRS practitioner Audrey Marie Morris (Bar number MA0556) Docket number 27312-21 is missing IRS practitioner Catherine J. Caballero (Bar number CC0518) Docket number 603-21 is missing private practitioner John J. Eagan (Bar number EJ0281) Docket number 7100-21 is missing private practitioner Jonathan F. Andres (Bar number AJ21305) Docket number 7306-21 is missing private practitioner Jonathan F. Andres (Bar number AJ21305) Docket number 22748-14 is missing private practitioner Gilbert S. Rothenberg (Bar number RG0320) Docket number 754-21 is missing private practitioner Vivian D. Hoard (Bar number HV0055) Docket number 7642-20 is missing private practitioner Samantha S. Wisniewski (Bar number WS0486) Docket number 1124-21 is missing private practitioner Alan L Aronson (Bar number AA0209) Docket number 14545-16 is missing private practitioner Grubert Roger Markley (Bar number MG0434) Docket number 14886-08 is missing private practitioner Grubert Roger Markley (Bar number MG0434) Docket number 19940-09 is missing private practitioner Grubert Roger Markley (Bar number MG0434) Docket number 1075-21 is missing private practitioner Cruz Saavedra (Bar number SC0507) Docket number 2058-21 is missing private practitioner Jonathan Edward Strouse (Bar number SJ1623) Docket number 681-21 is missing private practitioner Jonathan Edward Strouse (Bar number SJ1623) Docket number 29091-14 is missing private practitioner Robert W. Mopsick (Bar number MR1149) Docket number 756-13 is missing private practitioner Teresa Ellen McLaughlin (Bar number MT0172) Docket number 14380-16 is missing private practitioner Johnny Craig Young (Bar number YJ0083) Docket number 23410-14 is missing private practitioner Johnny Craig Young (Bar number YJ0083) Docket number 23411-14 is missing private practitioner Johnny Craig Young (Bar number YJ0083) Docket number 2404-14 is missing private practitioner Peter H. Clark (Bar number CP0344) Docket number 2417-14 is missing private practitioner Peter H. Clark (Bar number CP0344) Docket number 17152-13 is missing private practitioner Donna F. Herbert (Bar number HD0170) Docket number 12701-11 is missing private practitioner Elizabeth S. McBrearty (Bar number ME0589) Docket number 19502-17 is missing private practitioner Rogelio A. Villageliu (Bar number VR0078) Docket number 9076-17 is missing private practitioner Rogelio A. Villageliu (Bar number VR0078) Docket number 6054-19 is missing private practitioner Michael Leonard Boman (Bar number BM0317) Docket number 13328-18 is missing private practitioner Robert W. Benjamin (Bar number BR1519)

[halprin]

What could be missing?

• representing on a Case/Private Practitioner record.
• serviceIndicator on a Case/Private and IRS practitioner record because it is configurable by case.

Going forward plan at this point.

1. Determine whether or not this data can be recovered programmatically (e.g. through a migration alone)
   1. Write the migration.
2. Determine if we can recover the missing data by logs if we can't do it programmatically.
   1. Figure out if we can write a program or just manually write the item back.
3. Determine if we can fix the data on the fly while the application is running. We have options.
4. Add additional logs in areas we think could be the culprit. E.g. whenever a practitioner is removed. E.g. Around practitioners are already on a case and someone asked to associate that practitioner with that case.
5. Finally, investigate cause of bug further.

[halprin]

For the migration...

• For the IRS Practitioner
  • We might be able to fix a missing IRS practitioner because they do not have the representing field.
  • But we don't know what they may have used as the serviceIndicator for this specific case.
• For Private Practitioner
  • We might be able to fix a missing private practitioner if there is only one practitioner and only one petitioner on the case.
  • A super big might be able to fix a missing private practitioner if there is more than one practitioner and only one petitioner on the case. Is it possible that a private practitioner can be on the case without representing that single petitioner?
  • But we don't know what they may have used as the serviceIndicator for this specific case.

[halprin]

Latest findings:

• At least three of the IRS practitioners have magically shown back up for the some of the later 2021 cases in the test environment. I believe even at least one has shown in prod. Ostensibly they might have only ever been missing from test in the first place, but since the issue does appear in prod, this might not be the case? Wut?!
• Also, a decent number (all but two) of the older cases are missing even in the first prod database backup. That leads us to believe there is was also an issue in migrating the data from Blackstone or the data was messed up in Blackstone as well.
• For at least three missing private practitioner records, the associated cases were filed incorrectly by petitioner accounts that were later replaced by practitioner accounts. This, we believe (not proven), leads to the missing case to private practitioner records.
  • Further investigation should happen for how the petitioner account is replaced by the practitioner account. Specifically, it seems that the UUID of the petitioner account is the same as the replacement practitioner account. Is this possible other than Mike or Jim munging the data manually?

[halprin]

Update on the missing IRS practitioners. Only one is missing currently in test.

{
    entityName: 'IrsPractitioner',
    pk: 'case|2177-18',
    sk: 'irsPractitioner|3c8cb95d-2c23-4745-9f67-f0e79e94e08e',
    userId: '3c8cb95d-2c23-4745-9f67-f0e79e94e08e'
  }

This makes me think that there isn't a systemic problem for IRS practitioners. We need to have the script run against prod.

[jimlerza]

Latest results, from prod:

cases-missing-practitioners2.txt missing-entities2.txt

[halprin]

Steps to reproduce:

1. A petition exists with...
   • A petitioner with no e-mail address (this can be accomplished via a petition clerk making the case or a petitioner making a case that has two petitioners, e.g. filing for myself and my spouse).
   • No representation for the no-email petitioner.
   • Served to the IRS.
2. Admissions clerk adds any email address that is already assigned to a private practitioner as the e-mail address of the non-email petitioner.
3. Admissions clerk chooses to "associate" in the dialog box.

There are a few records from before DAWSON that don't match this usecase. There is also that missing IRS practitioner. We've yet to look into how the above steps have any bearing, if any, on the pre-DAWSON or IRS practitioner ones.

[mmarcotte]

Re: Zendesk automations. The only automation that appears to be related to this bug is one that does a single thing -- removes an account from Cognito. This frees up that Email address to be used by creating a new Practitioner or adding that email to an existing practitioner record that didn't previously have an Email address.

Here's what the admissions clerks are doing:

• receiving a request from a practitioner to add an email to their account or needing to create a new practitioner
• attempting to perform that action and receiving an error that the email address is already taken
• using the zendesk automation to remove that cognito account (it does nothing with Dynamo)
• then completing the original operation successfully

Perhaps Docket or Petitions is doing something before or after the above steps performed. It's conceivable that they are attempting to add the practitioner to the case, and THIS is what's causing the data discrepancy.

So many of these are where a practitioner creates the case with a petitioner account.

[TomElliottFlexion]

Tenille/Holly to ask Admissions about whether or not a practitioner account is "allowed" be added as a petitioner on a case.

Points that still need clarity:

• When a person with a practitioner account is also a petitioner on a case, are they able to use that practitioner account on the petitioner entity? (Editing a petitioner's info and adding an email address that is associated with a practitioner account)
• If yes, then would they be a petitioner who is "pro se", or would it need to be possible for one account to be both a petitioner and the representing practitioner on the account? (Currently a practitioner account that is listed as a petitioner on a case is treated as an unassociated practitioner despite the relevant case appearing under "My Cases"

[TomElliottFlexion]

PR with temporary fix and small explanation: https://github.com/ustaxcourt/ef-cms/pull/1897

[mariahkannenberg]

Just reading through this now and realizing that this is not a use case we have developed for in the past. A story was back logged for this functionality last year, but only for internal users. We would also need to create this functionality for non-court users (as described in this bug). This bug should be converted to a story.

[TomElliottFlexion]

Determined that no changes would be made until a larger role-based refactoring can be done. Revert here. Logging to record events similar to this bug.

[TomElliottFlexion]

Important note: the script referenced above locates records that may not be faulty/broken. There are several scenarios in which a practitioner may legitimately have no PrivatePractitioner entity in the case partition but a UserCase entity in the user partition; most notably if the practitioner signs as a petitioner on the initial petition and is therefore added as a petitioner on the case.

[TomElliottFlexion]

Need to confirm logging works correctly after the update, then this can be called done

[TomElliottFlexion]