Automate redlining identical content in Records Package

[lmullane]

• As an IAO analyst
• I want the system to automate redlining the same content in a records package
• so that redactions are applied consistently and I save time

Assumptions & Scope What are the assumptions for this story?

A records package often will include duplicate content.

Analysts want to save time and ensure consistency when redlining a package. For example, if they redline a name or phone number than the name or phone number should be automatically redlined wherever it appears in the records package.

90% of the time these redactions will apply. In some instances (a minority) the same redline may not apply to the same content (context matters) elsewhere in the package. Analysts can search the content and decide to remove the system applied redline or edit if they choose.

What is IN scope? Automate redlines of the same content in the same records package.

What is NOT in scope? Editing or removing all redactions or editing or removing one redaction; can already be done in the redaction app with bulk actions.

Suggesting redlines to the same content from different FOI requests

Acceptance Criteria

Scenario 1: Automate redlining of same content GIVEN an analyst has redlined and applied section stamps to content in an FOI records package WHEN the identical content is found elsewhere in the same FOI records package THEN the system should draw a redline around the the identical content every time it appears AND the system should apply the same section stamps AND the system should notify me that the same redaction has been applied (x) of times in the records package

Scenario 2: xxxxxx

• GIVEN ... (you or your condition)
• WHEN ... (what you do)
• THEN ... (what you see)

Scenario 3: xxxxxx ...

Dependencies? What is the impact of this dependency? (If so, link dependency in the ticket, make it visible in a team´s backlog)

Validation Rules? (If yes, list here)

Design @xxx - please link the Design here

Definition of Ready

1. [ ] Is there a well articulated User Story?
2. [ ] Is there Acceptance Criteria that covers all scenarios (happy/sad paths)?
3. [ ] If there is a user interface, is there a design?
4. [ ] Does the user story need user research/validation?
5. [ ] Does this User Story needs stakeholder approval?
6. [ ] Design / Solution accepted by Product Owner
7. [ ] Is this user story small enough to be completed in a Sprint? Should it be split?
8. [ ] Are the dependencies known/ understood? (technical, business, regulatory/policy)
9. [ ] Has the story been estimated?

Definition of Done

1. [ ] Passes developer unit tests
2. [ ] Passes peer code review
3. [ ] If there's a user interface, passes UX assurance
4. [ ] Passes QA of Acceptance Criteria with verification in Dev and Test
5. [ ] Confirm Test cases built and succeeding
6. [ ] No regression test failures
7. [ ] Test coverage acceptable by Product Owner
8. [ ] Ticket ready to be merged to master or story branch
9. [ ] Developer to list Config changes/ Update documents and designs
10. [ ] Can be demoed in Sprint Review
11. [ ] Tagged as part of a Release
12. [ ] Feature flagged if required
13. [ ] Change Management activities done?

[lmullane]

@KyEggleston , @JHarrietha-AOT, @liseandtea this is the story that came up in demo yesterday.

Note, given what we were told I think we can automatically apply the redline with the same section stamps every time the same content is found in the system. While there are exceptions around section 14 the analyst they are exceptions. The analyst can always review and edit each redaction separately or edit all of them with bulk actions if necessary.

Let me know what you think?

[JHarrietha-AOT]

@lmullane I think technically there will be a lot to make this happen as we will need something to read the document content to identify the same content. We would also need some limitations. We would not want identical redactions applied for one or two words, so how many words in a row need to be the same for the same redaction to apply? Also, I don't think this should be automated and at the Demo, we heard people say the preference would be for the same content to be identified and the analyst then selects and approves or deletes whether it should be applied. I think this would fit more into the NLP work and I don't know if the R&D has begun for that work. It doesn't feel like a stand-alone improvement to the Redaction App. Just sharing my thoughts @KyEggleston not sure if you have anything else to add

[lmullane]

Agreed there is a lot technically to do.

I believe names are a common thing that analyst may wish to remove so only a couple of words may be necessary. We could confirm with the super users.

The super users felt that you could automate redlines within the same records package. Applying the same redline would be the rule rather than the exception when it was within the same records package.

Thinking we should create some designs after we talk to the developers in refinement and then show it to the super users. It may be awhile before we can actually implement it though.

[JHarrietha-AOT]

@lmullane that makes a lot of sense. Seems like we are thinking the same things. Then we could move this into refinement and let the devs discuss the technical complexities and bring them to refinement. Once we get a better technical understanding from the devs we discuss any designs they need based on that.

[lmullane]

Perfect. Let’s do that. I know this is a main priority for the super users so need to understand how much effort is required to get there!

From: Jacklyn @.> Reply-To: bcgov/foi-flow @.> Date: Friday, December 8, 2023 at 11:54 AM To: bcgov/foi-flow @.> Cc: "Mullane, Loren CITZ:EX" @.>, Mention @.***> Subject: Re: [bcgov/foi-flow] Automate redlining identical content in Records Package (Issue #4842)

[EXTERNAL] This email came from an external source. Only open attachments or links that you are expecting from a known sender.

@lmullanehttps://github.com/lmullane that makes a lot of sense. Seems like we are thinking the same things. Then we could move this into refinement and let the devs discuss the technical complexities and bring them to refinement. Once we get a better technical understanding from the devs we discuss any designs they need based on that.

— Reply to this email directly, view it on GitHubhttps://github.com/bcgov/foi-flow/issues/4842#issuecomment-1847765763, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC33VB25Y5BH23FSGUNTICTYINV6RAVCNFSM6AAAAABAJ3T64SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBXG43DKNZWGM. You are receiving this because you were mentioned.Message ID: @.***>

[KyEggleston]

From pilot user session:

• Redlining should not be automatic. I.e., when they highlight to redact a keyword or phrase, it should not automatically redline all other occurrences. This should be manually confirmed.
• After highlighting to redact a keyword or phrase, the Redaction Panel should not open by default and perform the search - this may be jarring and unnecessary, as they don't always need to search for the keyword and phrase every time.
• Instead of automatically opening the Redaction Panel and having it perform a search, users posed the potential for having the option the click on the redaction, and clicking a 'Search' icon to perform the search for that highlighted keyword/phrase. This would then open the Redaction Panel, and search for the keyword/phrase. This would allow them to decide when they want to perform the search, and also allow them to manually review the results.
• They also noted that when they use the 'Search' button/icon to search for the keyword/phrase in the Redaction Panel, the original occurrence of this keyword/phrase should be identifiable in the search result presented in the Redaction Panel. This would allow them to easily find the original keyword/phrase that they are performing the search from, and be able to go back to that page and continue their work.

cc: @JHarrietha-AOT @liseandtea @lmullane

[KyEggleston]

This ticket needs to be revised / revisited based on notes above.

[JHarrietha-AOT]

After Meeting with Pilot Users on Dec 21st - It feels like what would be most useful is a way to trigger the Find and Redact and have it auto-populate with what has been redacted in the search field. It would be triggered by a right click and added to the current menu. Pilot users agreed this is a low priority and should be pinned till they have used search more and would have better feedback to provide.