[NG-2021-5] Study Transfer

[jcomedouteau]

Migration comptes GME:

The decision was made to make a full programatical solution to migrate data from a study to another server. The idea is tu use as much as possible the rest API to migrate the studies and their datas.

Elements that can we consider already existing

• import

• • nifticonverter

• migrations

• • migrations (the database will likely be up to date with migrations)

• preclinical

• • reference

• studies

• • coil
• • timepoint

• users

• • role

Users microservices

Users don't have to be migrated, they have to be re-created by the concerned users. Only one user (the one demanding the migration) will be added to the new study as admin

Study Microservice

Elements to be moved:

• Subject

• • reset ID

• Study

• • reset ID
• • reset SubjectStudy to update with subjects
• • Add protocol/ DUA files
• • Set StudyUser to only current user
• • reset studyExaminations

• Center

• AcquisitionEquipement

• Model

• Manufacturer Preclinical microservice

Elements to be moved:

• animal_subject -- reset ID -- Update subject ID with animalSubjectID

• subject_pathology -- reset ID -- Update subject ID with animalSubjectID

• subject_therapy -- reset ID -- Update subject ID with animalSubjectID

• anesthetics

• therapies

AFTER DATASETS

• examination_anesthetic
• • reset ID
• • reset examination ID
• examination_extradata
• • reset ID
• • reset examination ID

Dataset microservice

This is the most complicated microservice as it contains all the logic of datasets.

StudyCards

• Update study ID => StudyCardRule ==> StudyCardCondition
  ==> StudyCardAssignment

Examinations => DatasetAcquisitions

• • Update studyCard ID ==> Protocol(
• • • CT,
• • • PET
• • • MR ===> DiffusionGradient ===> MrProtocolMetadata ====> mr_protocol_metadata_mr_scanning_sequence (complicated) ====> mr_protocol_metadata_mr_sequence_variant (complicated) ) ==> datasets
• update study ID
• update imported study ID
• update subject ID ( CalibrationDataset CtDataset EegDataset ===> Channel ===> Event MegDataset MeshDataset MrDataset ===> DiffusionGradient ===> EchoTime ===> FlipAngle ===> InversionTime ===> RepetitionTime ===> MrDatasetMetadata ParameterQuantificationDataset PetDataset RegistrationDataset SegmentationDataset SpectDataset StatisticalDataset TemplateDataset ) ===> referencedDatasetForSuperimposition(ChildrenList) (?) ===> DatasetProcessing ====> input_of_dataset_processing ===> DatasetMetadata ===> DatasetExpression ====> DatasetFile => InstrumentBasedAssesment ==> VariableAssessment ===> InstrumentVariable (coded/num) ====> ScaleItem ===> Score (coded/num) ====> ScaleItem ==> Instrument ===> Scientific instrument

Related datasets (update datasetId and studyId)

Add extra-data files

Solr has to be declenched too (shanoir_metadata to be looked at ? I think it is useless but..?)

TECHNICAL DECISIONS:

To connect to a distant keycloak: 1) Declare local Shanoir as client in distant keycloak 2) Allow offline session and token settings in realm settings 3) Allow user to get offline_token for this specific client (user, Role mapping) 4) Export keycloack.json containing data about keycloak (see shanoit-uploader) -> Only name displayed in front 5) Prepare a link to be used in front to get this

The user will have to communicate its login/password/id of the distant Shanoir

To communicate between the Microservices

• RabbitMQ will be used to connect to the different mircroservice
• An offline token will be used to get access tokens and transmitted between microservices
• Start by studies that makes the first distant keycloak call, then import / preclinical / datasets ..
• Some informations will be transmitted among microservices (such as Map<oldId, newId> or token)

OLD Pre thinking. Moving data to GME, how to do this properly ? Many possibilities:

1) Copy all the database (dump) then remove one by on all the unneeded studies from interface

=> Can we bring a hard disk with all the content to the GME? / How to transfer the data ? => Can we (even temporarily) copy all the ‘private’ data of Neurinfo ? => How to copy data from the PACS ? Is it possible to make a dump too ? (or use a dcmtk tool ?) => If we delete a study, all data deleted ? (in PACS = NO I guess (this has to be corrected btw), but if the data is not here it’s not a pb)

2) Programatically, using the API/interfaces

=> Create a little program that interrogates shanoir-neurinfo on choosen study, then recreates the study in GME with all its elements.

=> Difficult to list ALL elements to recreate => Maybe use shanoir uploader? => Will we use import interface then ? What about re-anonymization ?

3) Use some kind of BIDS archive that we can export subject by subject then re-import directly into GME

=> Problem of re-anonymization as we redo an import ? => Not BIDS as we want to work with dicom essentially => Recreate manually all studies / study card / centers ? => How to do this automatically (export then re-import)

4) Create a huge SQL request / procedure to move data

=> How to move files ? => How to move PACS ? => The list of all data will be enormous

5) Mixed solution

• Move with an SQL script all references (can by applied easily) (studies, centers, etc..)
• Move programatically subjects, exams, datasets, dataset files only for the studies we want, importing data with some kind of « Shanoir uploader »

Pertinent questions: => How do we move dicom in PACS ? Can we make PACS communicate => create an alternative PACS that’ll be dupliacted from neurinfo with choosen studies ? (or use a dcmtk tool ?) Possible to make a dump too ?

That would mean opening shanoir’s/GME's PACS to the world (temporarly)?

=> How do we move files on server if not programaticaly or with hard drive ?

Lots of questions here, not a lot of answers => to be discussed !

[jcomedouteau]

EDIT: This is not relevant anymore, we won't move anything using SQL scripts. References to move with an SQL script: | datasets | center (??) | import | nifticonverter | migrations | migrations (la base sera à priori à jour) | preclinical | anesthetic
anesthetic_ingredient
contrast_agent
pathology
pathology_model reference
therapy

| studies | acquisition_equipment
center
coil
manufacturer
manufacturer_model
timepoint

| users | role

For the rest we need to move step by step all the data, that'll be complex as we need to keep a matching key between the old study and the new study.