Ingest for July data collection

[hobuobi]

July data collection consists of our data entry contractors entering the data from state reports, dashboards, etc. into the "XX_Data" sheets found in these folders: https://drive.google.com/drive/folders/1hhyCN5O5HBD3rl2BncHhQ1GzwV7x41Cz

Once that process is complete, we simply need to:

• Edit YAML files for any completely new metrics (in this case, CO probation commitments should be the only new thing. Will file a separate ticket for this for clarity/transparency).
• Run the ingest script (@colincadams to point)
• Resolve any minor issues that might have come up in data entry (@hobuobi to guide if so); re-run ingest script.
• Deliver new data file to nick@eleventygroup.com.

Note that this process does not require any communication with the contractors; any necessary communication there will be done by @hobuobi.

[colincadams]

Below are the relevant details for each step. They came out longer than I expected... Let me know if you have questions!

• YAML
  • The description of how to create a YAML lives in + 🔦 Justice Counts Manual Ingest, however it is a very lengthy almost comprehensive but slightly out of date reference document. The changes here should be minimal, so copy pasting may be the easiest way to go. We can also talk through it together.
• Ingest
  • instructions for running ingest live at + 🔦 Run Manual Ingest. These were a bit out of date but I just updated them. Possible I missed something so of course let me know if you run into any issues.
• Deliver data
  • The data will be refreshed automatically the next day with our normal run. I sometimes hit the export endpoint manually if I am in a rush, but that can take quite some time to run.
  • Once the data is in the bucket (recidiviz-staging-justice-counts-data), you'll download it locally.
  • At this point, I typically do a local diff against the last version of the files (see last step). This is out of an abundance of caution and shouldn't be necessary. Just because I will put my steps here:
  • for each file and version, sort and strip date published: cat $FILE.json | sort | awk -F ',"date_published":"[0-9]{4}-[0-9]{2}-[0-9]{2}"' '{print $1,$2}' > $FILE.new.json
  • Open in VSCode and use "Compare active file with..." to compare to the other version (obviously can use any tool, meld if you are cooler than me?)
  • Another test is to copy the new files into you local public dashboard repo src/data/ and run the app locally. We don't have a state switcher so you have to update the state in src/index.js. It is a bit slow so I just do it to spot check anything I was worried about or that CSG emphasized.
  • Send to Nick! The dashboard files live in this google drive folder. I upload new versions via right-click > Manage versions, but if you just drag them in it might work. Then send an email (or have humphrey or I do so) to Nick and Sarah that the updated files are in the folder.

[hobuobi]

July data collection is just about done! A few states had new data sources or metrics and will require YAML changes:

• CO: Added new data for probation admissions (Probation Admissions)
• AL: Added new data for probation revocations, which should be used instead of the existing probation revocations source. (Admissions - Parole Revocations)
• OR: Added new data for parole population (Parole Population) and probation population (Probation Population), which should both be used in the place of the existing sources.

A few states have not been completed yet, but we should still go ahead and proceed with ingest and deal with these next week:

• North Carolina (admissions data only)
• Delaware
• Texas

[colincadams]

All of this has been ingested. While I was at it I pulled the data for Delaware, finished off TX (I think it had already been done, but there were three prison population points that were missing), and ingested what had been backfilled so far for NC.

I think the only thing that leaves is ingesting any further backfill that is done for NC admissions.

[hobuobi]

Amaaaaazing! I think it's probably still valuable for us to send over what we've got. :) They also requested getting a CSV like the one we made for them last time with all the data (that went over to Jess); if that's quick enough to re-run and include when we send the new data file, great! If not, we can follow-up whenever it's convenient for us this week (and I can create a new ticket if needed).

[colincadams]

I sent all of this off to CSG. Should we open another ticket for the NC backfill? Or are we dropping that from scope?