1Password / for-open-source

Get a 1Password team account for free to support your open source initiatives!
1.59k stars 791 forks source link

Migrate existing program applications to new data format #946

Closed jodyheavener closed 2 months ago

jodyheavener commented 2 months ago

Summary

With the new data structure in place we want to move over all the existing 1Password for Open Source program applications, which exist in the form of pull requests. This PR adds all of the existing applications, massaged into our new structured JSON data format.

What's changed?

Added 720 existing applications, in our new JSON data format.

Here's what we did to get this:

  1. Query and download all the existing Pull Requests on this repo. The pull requests must be merged and have the label deprecated/approved (our old approval label).
  2. Use the data of each PR, including the body, author metadata, labels, and creation timestamp, to determine certain properties
  3. Pass the body of each PR to an AI prompt that then attempts to extract and format the correct information for more properties
  4. This got us most of the way there, but a little additional work was needed to finesse the data, so we ran additional data validation and made some manual changes as well

With this set of data we:

Testing

You may wish to spot check random entries and compare them against their original pull request. It is important to note that we are not going to get this migration 100% with the plain text format we are coming from.