Upload with websockets and integrated client side pseudonymization

[mdebic]

Desired workflow

Users logs into ADIT and next to the existing apps they have access to an upload app. The upload app is also bound to a permission manged by an admin. An admin can grant users access to the upload app. How the upload form should look like can be seen in the diagramm avobe. To process the images on the server side, the existing ADIT infrastructure for receiving images should be used, see BatchTransfer App und ADIT Core. The client side can transfer the images unpseudonymized to th server. But the user can also select an internal pseudonymization and one with a user provided batch file.

General info regarding pseudonymization:

• Happens on client side, using Java Script (dcmjs)
• Happens on-the-fly
  1. Image is received by the client browser on upload
  2. Image is read
  3. Image is pseudonymized
  4. Image is send to ADIT server
  5. Proceed with next image

General info for internal pseudonymization:

• In general the same infroamtion as stated above is also valid for this pseudonymization type
• Users can not assign pseudonyms, the complete process is handled by client side code
• Technical workflow:
  1. Take the department field from the user model of the currently uploading user
  2. Put it in front of the PatientID extracted from the uploaded image (e.g. DIR123456)
  3. Run SHA3-256 hashing function over the resulting string
  4. Use new 64 byte long string as new PatientID and PatientName
  5. Update the DICOM tags with that information
  6. Pseudonymize leftover DICOM tags normally with dcmjs
  7. Send image to server

Upload Job View

• Same general info about the upload job as in a BatchTransferJob view

• Upload tasks list also the same as in BatchTransferJob view

• But additional View Results button as in a BatchQueryJob which the user can download

  • The downloaded CSV file should contain a column with PatientID and the newly assigned pseudonym
  • If the pseudonymization is None, the two columns are basically the same
  • If the pseudonymization is Batch file, the client gets the same list as initially provided by the user
  • If the pseudonymization is Internal, the client gets a list with the generated pseudonyms and PatientIDs

• In the last to cases, no information about PatientID is transferred to the server, only pseudonym

• In the first case, the PatientID is transferred to server, since no pseudonymization is applied

• [ ] Implement frontend of upload app

• [ ] Implement connection via websockets

• [ ] Implement receiving image handling on server side

• [ ] Implement new permissions for pseudonymization type on upload (see diagramm above)

• [ ] Implement client side pseudonymization with Batch File

• [ ] Implement client side pseudonymization with Internal hashing method

[cmheidt]

Addition to the PSN process: Dept+ID Strings should be padded to ensure the same length before hashing, otherwise there's a small risk of being able to relate pseudonyms to departments, which is undesirable

[cmheidt]

Documentation of workflow as it stands:

• Client is given files to upload
• Override FileHandler to forward files to pseudonymization method
• Pseudonymize Files using Lucas dcmjs implementation (embed using Alpine.js)
• Post Files into a tempdir on server (using ajax, but could, more easily, probably be implemented by providing a temp dir to the Django FileField? ref method handle_uploaded_file on https://docs.djangoproject.com/en/4.1/topics/http/file-uploads/ )
• On the server side, create a Batch Transfer Job, using the adit node as source and desired destination
• Run the job using TransferUtils
• Clean up tempdir

@medihack I think that is how we devised it, if there are any problems with this let me know :+1:

[lmdulz]

• [ ] setup new Upload app
• registration in menu bar
• [ ] Form view
• [ ] data transfer view
• asynch
• doesn't send data to same URL as Form comes from
• checks destination of institute and uses it as transfer destination
• [ ] view safety
• User login
• Insitute access
• destination available
• [ ] check if uploaded files all belong to same patient -> only single patient uploads are allowed
• [ ] Html Template
• File search Field
• Text Field for PatientID which will be used with FixedValueAnonymizer
• embed Anonymization with Alpine.js
• [ ] Anonymization
• usage in JS File (which will be referenced in Template)
• use with seed to guarentee identical anonymization outputs
• [ ] Data Transfer
• test with Orthanc as receiving node
• check and use existing transfer implementation