Requirements

• Node v22+
• Yarn v1 ("classic")

Pipeline

The automated steps in this repo are roughly as follows:

1. Ingest
   1. Get raw data from an external resource, either by scraping an HTML page, downloading and parsing a PDF or CSV, or making a request to an API.
   2. Save raw data exactly as-is for provenance and caching.
   3. Collate most important information from raw data into common high-level output data format suited to making desired reports.
   4. Repeat previous steps in order of dependency (e.g. opportunity number -> grant numbers) until all needed info is gathered.
2. Print
   1. Run webapp (interactive dashboard that provides access to all reports).
   2. Import output data from ingest, and do some minimal final processing (e.g. combine journal info with each publication listing).
   3. Render select webapp pages (e.g. /core-project/abc123) to PDF reports.
3. Deploy webapp and PDFs to live, public web addresses.

Repo Content

• /app - Dashboard webapp made with Vue. Also used for generating PDF reports.
  • /public/pdfs - Outputted PDF reports.
• /data - All other functionality involving data (e.g. ingesting/collating/etc).
  • /api - Types and functions for getting raw data from external APIs.
  • /raw - Raw data gathered from external sources. Primarily for provenance, but also acts as ingest cache (delete files to re-fetch from external providers).
  • /ingest - Functions for scraping webpages and calling APIs, and collating that data into a common format.
  • /output - Collated data in format for making desired reports.
  • /print - Functions specific to making printed reports.
  • /util - Small-scope general purpose functions.

Technology

• TypeScript - Language used to provide type-safety from beginning to end of pipeline.
• Playwright - Tool used for scraping public web pages and rendering webapp page reports to PDFs.
• Netlify - Service used for hosting webapp (and PR previews).
• Octokit - Library used for conveniently interacting with GitHub APIs.

The ingest pipeline is optimized wherever possible and appropriate. Things like network requests and rendering are parallelized (e.g. PDF reports are printed simultaneously in separate tabs of the same Playwright browser instance). External resources are cached in their raw format to speed up subsequent runs, and to avoid being rate-limited or blocked by those providers.

Commands

Use ./run.sh with flags to run specific steps or tasks in this repo: