MLH data export

[Lordfirespeed]

Required fields:

• First name(s)
• Last name(s)
• Age
• Email
• School
• Phone number
• Country of residence
• Current level of study
• 'Checked in'

MLH data export should be an API route /applications/data-export/mlh which

• Generates a .csv file using input from Keycloak and the database (via prisma)
• Responds with the generated .csv file as an attachment

Implementation notes

• Work from the data-export branch, server/src/routes/applications/data-export, which currently implements responding with a list of UUIDs (one per line)
• would prefer output to be CSV2 (semicolon delimited) to but CSV (comma delimited) is fine
• We should make use of the Node.JS Streams API for this
  • use an async generator function which yields 'chunks' of 10 UserInfo (in an array) at a time
  • use our own Transform stream to augment each UserInfo in every chunk with data from Keycloak
  • make fetch requests to Keycloak for each UserInfo and Promise.AllSettled() the request promises
  • if any fail due to network errors, retry them (with a limit on the number of retries)
  • augment the UserInfo items with metadata from keycloak
  • push the chunk of augmented UserInfo items
  • use our own Transform stream to serialize each augmented UserInfo in every chunk to CSV
  • convert each UserInfo to a CSV representation (a single line of UTF-8 text)
  • combine the CSV lines into a single string using LF line endings (\n) to separate the lines
  • push the combined string
  • use a filesystem writable stream (fs.createWriteStream) to write the CSV rows to a temporary file location (use nanoid for filename and os.tmpdir for directory)
  • use the pipeline function from node:stream/promises module to run the stream pipeline to completion
• once the stream has been completed, respond with the file, then delete the local copy.