search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
284 stars 206 forks source link

Collaboration Cycle for [Virtual Agent, Chatbot, Prescription] #61873

Open Joy-Elizabeth opened 1 year ago

Joy-Elizabeth commented 1 year ago

VFS product information

VFS team name

Virtual Agent Chatbot

Product name

Chatbot

Feature name

Prescription

GitHub label for product

Virtual-Agent

GitHub label for feature

prescription

Kickoff questions

Toggle kickoff questions ### When did/will you start working on this product/feature? _April 2023_ ### Will your work result in visible changes to the user experience? _Yes_ ### Are you doing research with VA.gov users? _Yes_ ### Will your work involve changes to... Other ### Does your product/feature have Google Analytics tracking and a KPI dashboard in Domo? _Yes_ ### Do you need to capture any additional analytics or metrics? _Yes_ ### Product outline https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/virtual-agent/product/product_outline.md ### Verify all relevant materials provided to Governance Team - [X] I have provided all relevant and up-to-date links, screenshots, images, designs, etc. of the as-is version of the product ### Notify the Collaboration Cycle team of this ticket in the vfs-platform-support Slack channel. - [X] I acknowledge that I must notify \#vfs-platform-support after submitting this issue. Kickoff Slack Thread with VFS team: [Slack thread](https://dsva.slack.com/archives/CBU0KDSB1/p1689191802425839)

Collaboration Cycle touchpoints

Design Intent

Toggle Design Intent #### Before meeting ##### VFS actions - [ ] Review [Design Intent Guidance](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/design-intent) to understand what this touchpoint involves. - [ ] Schedule your Design Intent (with at least 2 business days lead time from now): - Open the [Calendly design intent calendar](https://calendly.com/collaboration-cycle/design-intent-or-midpoint-review) - Select a date and time and click "Confirm" - Add your name and email - Click "Add Guests" and enter the email addresses for VFS team attendees - Click "Schedule Event" - [ ] Link all artifacts in the `Design Intent artifacts for review` section below at least two business days before the scheduled Design Intent. Please don't add artifacts in the comments section. **Design Intent artifacts** See Platform guidance on [Design Intent artifacts](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/design-intent#Designintent-Artifacts). Platform feedback is based solely on the artifacts provided, as reviewed during the two days before the Design Intent meeting. Any work not included in the artifacts below or any ongoing work taking place during the review period may not be reflected in that feedback. - [x] User flow (required) Sketch Rx User Flows: https://www.sketch.com/s/b5a58a23-6128-421b-9869-50bad5b2686f - [x] Whiteboard sketch or other lo-fi prototypes or wireframes - Final mocks for Rx with just refill prototype: https://www.sketch.com/s/b5a58a23-6128-421b-9869-50bad5b2686f/prototype/a/E0136A5B-837B-4DF5-AEE6-986C7A365CFB - Prototype used for Rx testing: https://xd.adobe.com/view/c97e3903-3451-4696-9f87-d9cde5d83eaf-9c2d/?fullscreen&hints=on - Rx listing/tracking prototype: https://www.sketch.com/s/b5a58a23-6128-421b-9869-50bad5b2686f/prototype/a/99168B8F-2DF7-4AD3-9374-36569BBAA8FE - [x] Research plan https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/virtual-agent/research/prescriptions/MCT%20Virtual%20Agent%20Chatbot%20Research%20Facilitation%20Guide%202023-05-19%20V1.docx - [x] Any other artifacts you have so far Discovery mural: https://app.mural.co/t/thoughtworksclientprojects1205/m/thoughtworksclientprojects1205/1679944820061/4b8319b6f93ae72464fb5111ea19b4de44c2aca1?sender=u5b7c42cacbd4742036f15947 Prototype video with no sound: https://github.com/department-of-veterans-affairs/va.gov-team/assets/98479064/f73b924f-db0d-4958-9401-b6352b1dbcf8 Chatbot Showcase with Voice feature: https://dvagov-my.sharepoint.com/:v:/g/personal/alina_murphy_va_gov/EQSufboRJB1MlqtciyYhMJEBw23xXuOFsukVb3vSRVDCXw - Virtual Agent Chatbot. User Research Presentation: ##### Platform actions - [x] Design Intent Slack thread with VFS team: [Slack thread](https://dsva.slack.com/archives/CBU0KDSB1/p1689201220443029?thread_ts=1689191802.425839&cid=CBU0KDSB1) - [x] Meeting date/time: **Monday, August 7, 2023 at 3:30PM Eastern** #### After meeting ##### Platform actions - [x] If you have feedback, create feedback tickets and link to this epic. Once you have completed you review, check the box next to your practice area - [x] Accessibility has completed the review - [x] Design has completed the review - [x] IA has completed the review - [x] Update this ticket with the Zoom recording - [Zoom recording](https://oddball-io.zoom.us/rec/share/VkRouVx0Uf0LFeHuIwXf7nlfwOU2N8T9UiJJJQMdeI5UYkSaoWoXN1DvPkRdj70v.Oo7h8q3XeWpRv8R7) - Password: x*C74Fh* ##### VFS actions - [x] Review feedback tickets and comment on the ticket if there are any questions or concerns

Sitewide CAIA Intake Request

Toggle Sitewide CAIA Intake Request ##### VFS actions - [x] Complete the [Sitewide CAIA intake request process](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/Sitewide-CAIA-intake-request.2124906551.html) https://github.com/department-of-veterans-affairs/va.gov-team/issues/64157 - [x] Link to the Sitewide CAIA Intake Request ticket: [Epic] CAIA Support for VA Chatbot: Prescription Skill Feature #64157

Research Review

Toggle Research Review #### VFS actions - [ ] Complete the [research review process](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/Research-plan-review.1781891143.html)

Midpoint Review

Toggle Midpoint Review #### Before meeting ##### VFS actions Navigate to reference link: [Midpoint Review Guidance](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/midpoint-review) - [ ] Schedule your Midpoint Review when ready: - Open the [Calendly midpoint review calendar](https://calendly.com/collaboration-cycle/design-intent-or-midpoint-review) - Select a date and time and click “Confirm” - Add your name and email - Click "Add Guests" and enter the VFS meeting attendees email addresses - Invite all relevant VFS team members, including accessibility support, product owners, and other VA stakeholders - Click "Schedule Event" - [ ] Check this box if you'd like this review to be asynchronous (Please refer to the [Midpoint Review guidance](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/Midpoint-review.1781039167.html) for the difference between a synchronous meeting and an asynchronous review) - [ ] Link all artifacts **ONLY** in the Midpoint Review artifacts section below at least two days before the scheduled Midpoint Review. **Do NOT add artifacts to Comments section** **Midpoint Review artifacts** See Platform guidance on [Midpoint Review artifacts](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/midpoint-review#Midpointreview-Artifacts). Platform feedback is based solely on the artifacts provided, as reviewed during the two days before the Midpoint Review meeting. Any work not included in the artifacts below or any ongoing work taking place during the review period may not be reflected in that feedback. Provide links or documents for the following: - [ X ] Finalized design or mock up https://www.sketch.com/s/b5a58a23-6128-421b-9869-50bad5b2686f - [ X ] Specify which pages are included in the review : Rx (prescription) - [ X ] Research plan https://app.slack.com/huddle/T03FECE8V/D058LURFGDR - [ ] Conversation guide **Content artifacts** - [x] Your product’s content source of truth https://docs.google.com/spreadsheets/d/1Dqoa9U2F1iGz6f60tJpI-DoxLDYtqCgOHDYlIqh6kdw/edit#gid=0 - [ ] Github ticket for any relevant static content page and entry point updates for tool or feature (if applicable) **IA artifacts** - [x] Please include a link to any [Sitewide CAIA](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/sitewide-caia-intake-request) feedback you received, including an IA review document or intake form. https://github.com/department-of-veterans-affairs/va.gov-team/issues/64157 ##### Platform actions - [x] Midpoint review Slack thread with VFS team: [LINK](https://dsva.slack.com/archives/CBU0KDSB1/p1692808868780999) - [x] Meeting date/time: 8/29/23 at 3:30 PM ET #### After meeting ##### Platform actions - [x] If you have midpoint review feedback, create feedback ticket and link to this epic. Once you have completed your review, check the box next to your practice area - [x] Design has completed the review - [x] Accessibility has completed the review - [x] IA has completed the review - [x] Content has completed the review - [x] Update this ticket with the Zoom recording - [Midpoint review recording](https://oddball-io.zoom.us/rec/share/GDwy9tGynhG5pO391Uy_9ye_XltpguR6SdRtmxzcbqx2-UfXq2umBA6jKs8NOQyO.JYjlCkL4q7Bz31rB) - Password: F^?pC78E ##### VFS actions - [ ] Review feedback tickets and comment on the ticket if there are any questions or concerns

Analytics Request

Toggle Analytics Request #### VFS actions - [ ] Complete the [analytics request process](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/Analytics-request.1782120453.html) - [ ] Link to the Analytics Implementation and QA Request ticket below

Contact Center Review

Toggle Contact Center Review #### VFS actions - [ ] Complete the [Contact Center review process](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/Contact-center-review.1782317061.html) - [ ] Link to the Contact Center review request ticket below

Staging Review

Toggle Staging Review #### Before meeting ##### VFS actions - Navigate to reference link: [Staging Review Guidance](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/staging-review) - [ ] Schedule your Staging Review when ready: - Open the [Calendly staging review calendar](https://calendly.com/collaboration-cycle/staging-review) - Select a date and time and click “Confirm” - Add your name and email - Click "Add Guests" and enter the VFS meeting attendees email addresses - Invite all relevant VFS team members, including accessibility support, product owners, and other VA stakeholders - Click "Schedule Event" - [ ] If this product contains any [experimental design](https://design.va.gov/experimental-design/), add the `experimental-design` label and schedule a meeting with DSC to present the research findings. - [ ] Link all artifacts **ONLY** in the Staging Review artifacts section below at least four days before the scheduled Staging Review. **Do NOT add artifacts to Comments section** - [ ] I confirm the environment is available and test users have been provided. - [ ] Please verify your product information in the [Product Directory](https://depo-platform-documentation.scrollhelp.site/getting-started/vfs-product-directory). **Staging Review artifacts** See Platform guidance on [Staging Review artifacts](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/Staging-review.1810137181.html#Stagingreview-Artifacts). Platform findings are based solely on the staging URL and test users provided, as reviewed during the four days before the Staging Review meeting. Any test cases not covered or any ongoing work taking place during the review period may not be reflected in those findings. **Product artifacts** - [x] Staging URL : currently not in staging but our TL responded, "if we ask them to use Dev, we can provide any user from the TUD because we hardcode the JWT" https://dev.va.gov/contact-us/virtual-agent/ - [x] Specify which pages are included in the review trigger phrase "refill prescription" - [x] Staging test user information https://dev.va.gov/contact-us/virtual-agent/ user: [vets.gov](http://vets.gov/).[user+228@gmail.com](mailto:user+228@gmail.com) pw: 200SsNrLgPv5 https://staging.va.gov/contact-us/virtual-agent/ **Note:** Please double-check that you've provided staging access information appropriate for testing the tool or feature. Don't put staging credentials in your va.gov-team ticket; store [test user information, passwords, and tasks](https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/blob/master/Administrative/vagov-users/staging-test-accounts-accessible-example.md) in a .md file in the va.gov-team-sensitive repository. **Content artifacts** - [x] Github ticket for any relevant static content page and entry point updates for tool or feature (if applicable) *Please note: We have full output of content in the sheet, but for staging review in dev it is still going to have some holes in it. We have a planned review with Pharmacy team. https://docs.google.com/spreadsheets/d/1Dqoa9U2F1iGz6f60tJpI-DoxLDYtqCgOHDYlIqh6kdw/edit#gid=1606100598 **IA artifacts** - [x] Please include a link to any [Sitewide CAIA](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/sitewide-caia-intake-request) of the findings you received, including an IA review document or intake form. Intake form here: https://github.com/department-of-veterans-affairs/va.gov-team/issues/64157 **QA artifacts** An artifact that corresponds to each standard in the [QA Standards](https://depo-platform-documentation.scrollhelp.site/developer-docs/quality-assurance-standards). - [x] [Regression test plan](https://depo-platform-documentation.scrollhelp.site/developer-docs/quality-assurance-standards#QAstandards-regression-test-planRegressionTestPlan) same as test plan: https://wiki.mobilehealth.va.gov/pages/viewpage.action?pageId=384665671 - [x] [Test plan](https://depo-platform-documentation.scrollhelp.site/developer-docs/quality-assurance-standards#QAstandards-test-planTestPlan) https://wiki.mobilehealth.va.gov/pages/viewpage.action?pageId=384665671https://docs.google.com/spreadsheets/d/18ydxtkiP_vVG4pEu_xt98RXICOifyMsx5nmXTXMqsmc/edit#gid=0 - [x] [Coverage for References](https://depo-platform-documentation.scrollhelp.site/developer-docs/quality-assurance-standards#QAstandards-traceability-reportsTraceabilityReports) https://coderepo.mobilehealth.va.gov/projects/VVHA/repos/vha-voice-health-assistant-service/browse/CHANGELOG.md?until=a5d22dc587bef1d2a0ccff127674cd2d7f41a92d&untilPath=CHANGELOG.md&at=refs%2Fheads%2Fdev%2F1.0 - [x] [Summary(Defects) reports](https://depo-platform-documentation.scrollhelp.site/developer-docs/quality-assurance-standards#QAstandards-traceability-reportsTraceabilityReports) see again: https://docs.google.com/spreadsheets/d/18ydxtkiP_vVG4pEu_xt98RXICOifyMsx5nmXTXMqsmc/edit#gid=0 - [x] [E2E tests](https://depo-platform-documentation.scrollhelp.site/developer-docs/quality-assurance-standards#QAstandards-e2e-test-participationE2ETestParticipation) (provide a link to the product’s code) https://docs.google.com/spreadsheets/d/1v96euIji34mY8MaiBgQ-tOUs3Txnp68w/edit#gid=1655867755 - [x] [Code coverage](https://depo-platform-documentation.scrollhelp.site/developer-docs/quality-assurance-standards#QAstandards-unit-test-coverageUnitTestCoverage) (provide a link to the product’s code) [CodeCoverageAnalysisRxBot.pdf](https://github.com/department-of-veterans-affairs/va.gov-team/files/12882909/CodeCoverageAnalysisRxBot.pdf) **Tech Notes on testing:** - We have limited test data in Staging and limited ability to test all the functions of the rx feature from the Staging env - We have been deployed in Staging for quitesome time now but we were recommending your review in Dev so you can test the entire extent of functionality of the Rx integration with the Chatbot - Here is some more details about the challenges we faced while trying to test E2E in Staging with mocked data: One of the issues that cited about our ability to test end to end is that there is no CMOP set up with the test accounts we have access to, so even if we submit a refill, we are unable to transmit it for shipment processing in SQA (Staging in VHA’s MAP env). So our track shipments can’t be tested in Staging with our current set up. - use a login flow prompted from within the chat bot, because in lower envs Dev and Staging - the Auth teams have enabled OAuth based auth, but in Prod they are still using SAML - Chatbot is still using SAML in all envs; So going through the chat prompt would be a requirement - **Here is a walk through in the dev environment that follows** [this content testing outline](https://docs.google.com/document/d/1hr6I6jktX-QEWJVAugF6fCBTP7Sok5Btv4fZWVPPAxI/edit). This was created to help when performing review. https://github.com/department-of-veterans-affairs/va.gov-team/assets/98479064/d171bd39-dd5b-4a86-a31f-1b7b403965f3 **Accessibility artifacts** - [x] [Completed accessibility testing artifact](https://github.com/department-of-veterans-affairs/va.gov-team/issues/new?assignees=briandeconinck&labels=a11y-testing&template=a11y-testing.yaml&title=Accessibility+Testing+for+%5BTeam+Name%2C+Product+Name%2C+Feature+Name%5D). For details, see [Prepare for an accessibility staging review](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/prepare-for-an-accessibility-staging-review). https://docs.google.com/spreadsheets/d/1Dqoa9U2F1iGz6f60tJpI-DoxLDYtqCgOHDYlIqh6kdw/edit#gid=0 ##### Platform actions - [x] [Slack request](https://dsva.slack.com/archives/C01KTS3F493/p1696524534061749) - [x] Meeting date/time: October 12, 2023 at 3:30pm ET #### After meeting ##### Platform actions - [x] Update this ticket with the Zoom recording - [Recording URL](https://oddball-io.zoom.us/rec/share/8_I1zuF-DbceK-EWAcmbZJq0PrhpKW-nIV9hWaCM_N0epcu8dADkiuXOQj68VLFg.l-nuvrTawYZy7Nvc) - Password: a8.1W4nd ##### VFS actions - [ ] Review the findings tickets and comment on the ticket if there are any questions or concerns - [ ] Close individual findings tickets when the issue has been resolved or validated by your Product Owner. If a team has additional questions or needs Platform help validating the issue, please comment on the issue ticket. - [ ] After launch, [request an accessibility audit from the VA 508 Office](https://depo-platform-documentation.scrollhelp.site/developer-docs/request-support-from-the-va-508-office#RequestsupportfromtheVA508office-AuditRequest). This is required even if no accessibility issues were found during the Staging Review. - [ ] Share ServiceNow ticket number here: ______ - [ ] Close ticket once Privacy, Security, Infrastructure Readiness Review has been completed, VA 508 Office audit is requested, and all other post-Staging actions are complete

Privacy, Security, Infrastructure Readiness Review

Toggle Privacy, Security, Infrastructure Readiness Review #### VFS actions - [x] Complete the [Privacy, security, infrastructure readiness review process](https://depo-platform-documentation.scrollhelp.site/collaboration-cycle/Privacy-and-security-review.1782317101.html) - [x] Link to the Readiness Review ticket below https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/issues/1301 #### Platform actions - [ ] Privacy, security, infrastructure readiness review is complete
sara-amanda commented 1 year ago

Adding link to CAIA EPIC @coforma-terry

Joy-Elizabeth commented 1 year ago

@allison0034 I just added a few more items under "Tech Notes on testing" to assist the team in review for our staging review meeting. 1. VHA Voice Health Assistant Service - Content Testing Speed Run 2. Video showing the Rx features walkthrough

shiragoodman commented 1 year ago

@Joy-Elizabeth @nathalierayter This Github query includes all of issues identified at Staging Review for the initial Chatbot release. I would recommend looking at any ticket with an accessibility defect label (a11y-defect-1, a11y-defect-2, a11y-defect-3, or a11y-defect-4) and reviewing the comments because you'll likely see Ian Santarinala's comments about opening a support ticket with Microsoft. Unfortunately, the accessibility specialist who flagged these issues is no longer on Platform, but if you have questions about any of them, please tag @rsmithadhoc and/or myself and we can help you out.

shiragoodman commented 1 year ago

Also, here is the Staging Review ticket that was opened for the initial Chatbot release: https://github.com/department-of-veterans-affairs/va.gov-team/issues/34106

All of the tickets I included in the query above should be linked as child tickets for the Staging Review ticket.