Automate process to easily ingest IAM MPI data and report on CSP counts (part 1)

[elizabethmkoch]

Issue Description

The login.gov adoption team, stakeholders, and VA leadership need to review data and analyze CSP counts and CSP method changes to track login.gov adoption.

User Story

As a Product Owner I want to know the total user counts by CSP accessing the va.gov ecosystem as we progress through the migration So that we can track adoption.

---

Tasks

pre-process script automation

• [ ] clean up (or re-write) script to pre-process IAM data exports
  • [x] Ingests CSV files
  • [x] Outputs CSV files
  • [x] Updates map file
  • [ ] Generates file tracking login methods
• [ ] ensure script has access to network drive: \vacohsm01.dva.va.gov\VACO_Workgroups\Information & Technology\OCTO Digital Experience\ETL
• [ ] enable auto-detection of IAM data exports to initiate pre-process and load into Domo
• [ ] implement a hand-off to Domo Workbench workflow

Stanzas 2-4 have been moved to another ticket.

2. Configure Domo and Domo Workbench to load new datasets

   • [ ] install Domo Workbench on shared server
   • [ ] configure Domo to detect newly exported datasets and initiate load
   • [ ] load as partitioned dataset
   • [ ] configure Domo ETL to automatically run with updated datasets

3. Update dashboard

   • [ ] update dashboard visualizations to include view of changes over time

4. Establish recurring IAM exports

   • [ ] Work with IAM team to establish weekly cadence of snapshots

Acceptance Criteria

• [ ] Process automated on regular weekly cadence

[holytoastr]

Question: Load data from where to where?

[sdickson-gt]

Questions/risks:

• where should the script live within VA.gov ecosystem?
• confirm cron job is the right approach and how to configure?
• who will script run as and will it have access to folder / how to give access to folder?
• performance related to working network drive?
• where can we install a Domo Workbench windows native exe such that the team can access?

[holytoastr]

Comments from conversation with Steve yesterday:

• concerned about risk of pii when moving files across berms
• concerned about possible mistakes made during transformation of data
• Recommend getting data files directly from data folks, in exact formatting we need for domo rather than manipulating files on our end. Would give greater confidence in the acurracy of the data

[holytoastr]

Comments from conversation with Steve today:

• Asked why we don't have access to the database with this data. (Unknown)
• Commented that it seems like we are re-inventing a database without all the efficiency of a database
• Re-assigned domo workbench ticket to devops instead of analytics where it had gone by default

[holytoastr]

Added details of what has been completed. A new card for the new spring will be created to track unfinished tasks.