CDCgov / prime-reportstream

ReportStream is a public intermediary tool for delivery of data between different parts of the healthcare ecosystem.
https://reportstream.cdc.gov
Creative Commons Zero v1.0 Universal
71 stars 40 forks source link

Divide up the reports/ blobstore into new directories and subdirectories #1416

Closed jimduff-usds closed 3 years ago

jimduff-usds commented 3 years ago

We'll stop using the reports/ directory, and we'll start using three new directories, received/, internal/, and ready/. As follows:

  1. The receive step will put the originally received report into received/, exactly byte for byte as the payload was received. (Note: If the payload was empty, I believe the code doesn't create an entry in the report file table, nor a empty blob in the Blob store. Please confirm this). So, the receive step should put exactly one file in the received/ folder per successful run. Use a .csv or .hl7 suffix (or others in the future), as per the file type. Do NOT store the 'internal' version of this file any more - there's no need for it. (If you just do #1010, then you have done this step!)
  2. All other files created by the receive step should go into the internal/ folder, and should have an .internal.csv suffix. (This is a change from current .internal, to make them easier to read when we pull them out for debugging)
  3. The batch step creates files ready for download. The batch step should put all its result files into the ready/ folder,in a subdirectory named after its full receiver name (not just the org). For example, if the file is to go to receiver elr in organization vt-doh, create the directory ready/vt-doh.elr/

Acceptance criteria:

  1. Data received from a sender is stored in a new received folder in the blob store.
  2. Data received from a sender is stored with an appropriate extension based on the type of data (e.g. .csv or .hl7)
  3. Data received from a sender is NOT stored as an internal file in the blob store. See #1010
  4. Intermediate steps that generate internal versions of the file need to the stored in a new internal folder in the blob store.
  5. Internal files need to have the extension of .internal.csv, so they are easier to view/read.
  6. The batch step that sends data to the receivers need to store its files (which are ready for upload) in a new ready folder followed by the full receiver name (e.g. az-phd.elr-prod).
  7. The database need to reflect the changes to the folders and extensions
  8. The download site works as expected (the folder and extension changes does not affect it)
jimduff-usds commented 3 years ago

This bug has now happened. See #1550