Mass NIfTI conversion [NG-2023-02]

[michaelkain]

• stop storing NIfTI files by default
• open for mass conversion of nifti files
• e.g. used to harmonize big BIDS exports
• stop storing NIfTI files on the disk (storage size duplication), but propose the selection of a converter and download just-in-time converted NIfTI files, based on the DICOMs.
• Remove NIfTI converter from StudyCard and from all imports
• Attention: the constructed today available BIDS tree (symlinks to nifti files) will not exist anymore on the disk for download
• Attention: the conversion time can be long for a lot of datasets, so this will have to be managed
• Attention: we will need to manage, that NOT 10 users on the same study in parallel can run a full nifti conversion, what will be too much

[jcomedouteau]

Mass nifti conversion / Unstore nifti files specification

Objectives:

• Move nifti conversion to a safe/isolated microservice
• Be able to convert on the fly multiple niftis
• Remove nifti file “duplication” on file system
• Do not convert automatically to nifti during import

Part 1: Move nifti conversion to another microservice and allow multiple conversion (these two cannot be done separately)

A) Move nifti conversion to another microservice.

This nifti conversion should be isolated in a sandbox to avoid some risky calls. For this, use seccomp ( with docker: https://docs.docker.com/engine/security/seccomp/) → Create a new container with seccomp option → Move database logic of nifti converters → Move Docker file installation logic from import to this new container → Move all conversion code logic (better the error management)

B) Allow multiple conversion.

The massive download with a specific converter should be available:

• Solr search
• Study detail
• Study tree detail
• Examination detail
• Dataset detail
• Shanoir downloader The main idea of this feature is to convert the dicom files on the fly to get nifti files. This will change the logic of following features:
• download nifti
• display preview in papaya
• BIDS export

For the moment we propose a single dataset nifti reconversion in DatasetApiController / DatasetDownloaderServiceImpl As nifti converters are stored in import microservices, the logic is: 1) Download dicom files from the dataset in a shared folder 2) Use rabbitMQ to communicate with import MS (NIFTI_CONVERSION_QUEUE) 3) In import MS, convert data using the selected converter and return the result folder 4) Get the result folder and send it back to download

This logic is correct for a single dataset, we may need to upgrade it for multiple datasets.

PART 1 Apply conversion logic for multiple datasets

Propose a new button to download in nifti with a specific converter See the way it is done in single dataset download , when clicking on the button, the list of nifti converters is displayed. Here we need to propose the "default" (the one in dataset_acquisition) converter (in green ?) and other converters. In the back, convert the datasets either one by one, either all together (I don't know what's possible, see inDatasetsCreatorAndNIfTIConverterService) At the end of the process, delete the newly created niftis.

PART 2 Apply this conversion logic to download process and delete all existing nifti files on disk and associated dataset expressions

• Remove this new button, and apply the logic to all "download nifti" buttons.
• The error management should be bettered (to see what conversion failed)
• In papaya, load dicoms and not niftis
• Apply this logic when generating BIDS dataset -> When we generate it, we reconvert all datasets. Here we can imagine what we are going to generate the BIDS in the background, as, for some studies, the loading time will be very heavy.
• Add a warning about space usage when generating BIDS
• Propose a way to delete the BIDS structure (so that it does not take all space) -> also deleted when something is updated is the study.
• Remove all dataset expressions / files for niftis

Visualization with papaya:

• With papaya, we need to offer a choice between dicom or nifti with a specific converter.

We need to define a “default converter” in a given study. If the converter is not specified, this converter is selected.

Part 2: Remove existing niftis

• Delete all files on disk
• Remove all dataset file and dataset expression associated with niftis.

[jcomedouteau]

Also we can put bruker2dicom in this microservice I guess.

[michaelkain]

Hi Jean-Come, thank you for your efforts. I agree principally to move NIfTI conversion to another microservice, fine, but above are quite a few points, we will have to exchange on. With kind regards, Michael

[jcomedouteau]

-> The admin can decide to make a full conversion + storage + tag in the study administration (or expert ?) -> Zip the conversion result with a tag and a date if the user want ? Camille: Can be a problem if we want to share the nifti files. -> The admin can decide to convert during import too (study parameter)

[michaelkain]

developed by Jean-Come