pradeeban
commented
2 years ago Comment from Ramon:
TL:DR: We need an if statement in line 482 Explanation: Lines 467 onwards. We iterate through each metadata file. We check which column names there are and if there are nans. We create two lists one that has the number of times a tag appears in our metadata and another that counts the number of nans. Line 482 will then filter based on this. The problem is that this filtering occurs regardless of the print_only_common headers. Why does this problem occur? The potential reasoning is that given how heterogenous the Dicom metadata is, batches can have some tag variability. Therefore to ensure we can load all the metadata files into one giant file, we check which tags/columns are present in all the files. A preferred fix would be to fill out the missing columns as NAS. But this will go against the nan thresholding. This may also see tags be added that shouldn’t be reported. For example: if US and CXR images are present in the same extraction. Simply adding nan columns will result in a US sound study having nan CXR tags. This will be a point of confusion in the future. Suggestion: For now, we can add proper if statements to line 482 to change the metadata filtering. This may break the step that creates the one giant metadata file. The challenge is that CSV expects a consistent data schema, which is impossible with DICOM data from multiple modalities.
Nitesh639
We rarely get all the headers when we run lots of DICOM files through png-extraction. In contrast, if we run just a couple of files, we get more attributes. Let me give an example:
The fields we got when we ran it against June 28th 2021 all MR data:
AccessionNumber,Private Creator,InstanceNumber,InstitutionName,IssuerOfPatientID,Manufacturer,ManufacturerModelName,Modality,PatientBirthDate,PatientID,PatientName,PatientSex,Private tag data,[Unknown],ReferringPhysicianName,SOPClassUID,SOPInstanceUID,SeriesDate,SeriesDescription,SeriesInstanceUID,SeriesNumber,SeriesTime,SpecificCharacterSet,StationName,StudyDate,StudyDescription,StudyID,StudyInstanceUID,StudyTime,[Reject Image Flag],[Significant Flag],[Confidential Flag],[Assigning Authority For Patient ID],file,has_pix_array,category,BitsAllocated,BitsStored,Columns,HighBit,ImageOrientationPatient,ImagePositionPatient,ImageType,PatientPosition,PhotometricInterpretation,PixelRepresentation,PixelSpacing,Rows,SamplesPerPixel,SliceThickness
The fields we got when we ran against just 2 DICOM images from the above set:
AccessionNumber,AcquisitionDate,AcquisitionMatrix,AcquisitionNumber,AcquisitionTime,AngioFlag,BitsAllocated,BitsStored,BodyPartExamined,Columns,ContentDate,ContentTime,DeviceSerialNumber,EchoNumbers,EchoTime,EchoTrainLength,EthnicGroup,FillerOrderNumberImagingServiceRequest,FlipAngle,FrameOfReferenceUID,HighBit,ImageOrientationPatient,ImagePositionPatient,ImageType,ImagedNucleus,ImagingFrequency,InPlanePhaseEncodingDirection,InstanceCreationDate,InstanceCreationTime,InstanceNumber,InstitutionAddress,InstitutionName,InstitutionalDepartmentName,IssuerOfPatientID,LargestImagePixelValue,Laterality,MRAcquisitionType,MagneticFieldStrength,Manufacturer,ManufacturerModelName,Modality,NumberOfAverages,NumberOfPhaseEncodingSteps,OtherPatientIDs,PatientAddress,PatientAge,PatientBirthDate,PatientID,PatientName,PatientPosition,PatientSex,PatientSize,PatientWeight,PercentPhaseFieldOfView,PercentSampling,PerformedProcedureStepDescription,PerformedProcedureStepID,PerformedProcedureStepStartDate,PerformedProcedureStepStartTime,PhotometricInterpretation,PhysiciansOfRecord,PixelBandwidth,PixelRepresentation,PixelSpacing,PregnancyStatus,0_ProcedureCodeSequence_CodeMeaning,0_ProcedureCodeSequence_CodeValue,0_ProcedureCodeSequence_CodingSchemeDesignator,0_ProcedureCodeSequence_CodingSchemeVersion,ProtocolName,0_ReferencedImageSequence_ReferencedSOPClassUID,0_ReferencedImageSequence_ReferencedSOPInstanceUID,1_ReferencedImageSequence_ReferencedSOPClassUID,1_ReferencedImageSequence_ReferencedSOPInstanceUID,2_ReferencedImageSequence_ReferencedSOPClassUID,2_ReferencedImageSequence_ReferencedSOPInstanceUID,0_ReferencedPatientSequence_ReferencedSOPClassUID,0_ReferencedPatientSequence_ReferencedSOPInstanceUID,0_ReferencedStudySequence_ReferencedSOPClassUID,0_ReferencedStudySequence_ReferencedSOPInstanceUID,ReferringPhysicianName,RepetitionTime,0_RequestAttributesSequence_RequestedProcedureID,0_RequestedProcedureCodeSequence_CodeMeaning,0_RequestedProcedureCodeSequence_CodeValue,0_RequestedProcedureCodeSequence_CodingSchemeDesignator,RequestedProcedureDescription,RequestingPhysician,Rows,SAR,SOPClassUID,SOPInstanceUID,SamplesPerPixel,ScanOptions,ScanningSequence,SequenceName,SequenceVariant,SeriesDate,SeriesDescription,SeriesInstanceUID,SeriesNumber,SeriesTime,SliceLocation,SliceThickness,SmallestImagePixelValue,SoftwareVersions,SpacingBetweenSlices,SpecificCharacterSet,StationName,StudyDate,StudyDescription,StudyID,StudyInstanceUID,StudyPriorityID,StudyStatusID,StudyTime,TransmitCoilName,VariableFlipAngleFlag,WindowCenter,WindowCenterWidthExplanation,WindowWidth,dBdt,Private Creator,Private tag data,[Unknown],[CSA Image Header Type],[CSA Image Header Version ??],[SliceMeasurementDuration],[GradientMode],[FlowCompensation],[TablePositionOrigin],[ImaAbsTablePosition],[ImaRelTablePosition],[SlicePosition_PCS],[TimeAfterStart],[SliceResolution],[RealDwellTime],[CSA Image Header Version],[CSA Image Header Info],[CSA Series Header Type],[CSA Series Header Version],[CSA Series Header Info],[Series Workflow Status],[AcquisitionMatrixText],[CoilString],[PATModeText],[PositivePCSDirections],[Reject Image Flag],[Significant Flag],[Confidential Flag],[Assigning Authority For Patient ID],file,has_pix_array,category,AdmissionID,0_AnatomicRegionSequence_CodeMeaning,0_AnatomicRegionSequence_CodeValue,0_AnatomicRegionSequence_CodingSchemeDesignator,BeatRejectionFlag,CardiacNumberOfImages,HeartRate,ImagesInAcquisition,InStackPositionNumber,InversionTime,PerformedLocation,PerformedStationName,ReceiveCoilName,ReconstructionDiameter,0_ReferencedPerformedProcedureStepSequence_ReferencedSOPClassUID,0_ReferencedPerformedProcedureStepSequence_ReferencedSOPInstanceUID,StackID,TriggerWindow,[Suite id],[Product id],[Image actual date],[Service id],[Mobile location number],[Equipment UID],[Actual series data time stamp],[Horiz. Frame of ref.],[Series contrast],[Last pseq],[Series plane],[First scan ras],[First scan location],[Last scan ras],[Last scan loc],[Display field of view],[Acquisition Duration],[Second echo],[Number of echoes],[Table delta],[Contiguous],[Peak SAR],[Cardiac repetition time],[Images per cardiac cycle],[Actual receive gain analog],[Actual receive gain digital],[Delay after trigger],[Swappf],[Pause Interval],[Pause Time],[Slice offset on freq axis],[Auto Prescan Center Frequency],[Auto Prescan Transmit Gain],[Auto Prescan Analog receiver gain],[Auto Prescan Digital receiver gain],[Bitmap defining CVs],[Pulse Sequence Mode],[Pulse Sequence Name],[Pulse Sequence Date],[Internal Pulse Sequence Name],[Transmitting Coil Type],[Surface Coil Type],[Extremity Coil flag],[Raw data run number],[Calibrated Field strength],[SAT fat/water/bone],[User data 0],[User data 1],[User data 2],[User data 3],[User data 4],[User data 5],[User data 6],[User data 7],[User data 8],[User data 9],[User data 10],[User data 11],[User data 12],[User data 13],[User data 14],[User data 15],[User data 16],[User data 17],[User data 18],[User data 19],[User data 20],[User data 21],[User data 22],[Projection angle],[Saturation planes],[SAT location R],[SAT location L],[SAT location A],[SAT location P],[SAT location H],[SAT location F],[SAT thickness R/L],[SAT thickness A/P],[SAT thickness H/F],[Phase Contrast flow axis],[Velocity encoding],[Thickness disclaimer],[Prescan type],[Prescan status],[Projection Algorithm],[Fractional echo],[Cardiac phase number],[Variable echoflag],"[Concatenated SAT {# DTI Diffusion Dir., release 9.0 & below}]","[User data 23 {# DTI Diffusion Dir., release 9.0 & below}]","[User data 24 {# DTI Diffusion Dir., release 10.0 & above}]",[Velocity Encode Scale],[Fast phases],[Transmit gain],[Series from which prescribed],[Image from which prescribed],[Screen Format],[Locations in acquisition],[Graphically prescribed],[Rotation from source x rot],[Rotation from source y rot],[Rotation from source z rot],[Num 3D slabs],[Locs per 3D slab],[Overlaps],[Image Filtering 0.5/0.2T],[Diffusion direction],[Tagging Flip Angle],[Tagging Orientation],[Tag Spacing],[RTIA_timer],[Fps],[Auto window/level alpha],[Auto window/level beta],[Auto window/level window],[Auto window/level level],[Start time(secs) in first axial],[No. of updates to header],[Indicates study has complete info (DICOM/genesis)],[Last pulse sequence used],[Images in Series],[Landmark Counter],[Number of Acquisitions],[Indicates no. of updates to header],[Series Complete Flag],[Number of images archived],[Last image number used],[Primary Receiver Suite and Host],[Protocol Data Block (compressed)],[Image archive flag],[Scout Type],[Imaging Mode],[Pulse Sequence],[Imaging Options],[Plane Type],[RAS letter of image location],[Image location],[Image dimension - X],[Image dimension - Y],[Number of Excitations],[Lower range of Pixels1],[Upper range of Pixels1],[Lower range of Pixels2],[Upper range of Pixels2],[Version of the hdr struct],[Advantage comp. Overflow],[Advantage comp. Underflow],[Bitmap of prescan options],[Gradient offset in X],[Gradient offset in Y],[Gradient offset in Z],[Number of EPI shots],[Views per segment],"[Respiratory rate, bpm]",[Respiratory trigger point],[Type of receiver used],[DB/dt Peak rate of change of gradient field],[dB/dt Limits in units of percent],[PSD estimated limit],[PSD estimated limit in tesla per second],[Window value],[GE image integrity],[Level value],[Unique image iden],[Histogram tables],[User defined data],[Effective echo spacing],[Filter Mode (String slop field 1 in legacy GE MR],"[Image Type (real, imaginary, phase, magnitude)]",[Vas collapse flag],[Vas flags],[Neg_scanspacing],[Offset Frequency],[User_usage_tag],[User_fill_map_MSW],[User_fill_map_LSW],[User data 25...User data 48 {User48=Effective Resolution for spiral}],[Slop_int_6... slop_int_9],[Slop_int_10...slop_int_17],[Scanner Study Entity UID],[Scanner Study ID],[Scanner Table Entry (single gradient coil systems only)/Scanner Table Entry + Gradient Coil Selected],[Recon mode flag word],[Coil ID Data],[GE Coil Name],[System Configuration Information],[Asset R Factors],[Additional Asset Data],"[Governing Body, dB/dt, and SAR definition]",[Private In-Plane Phase Encoding Direction],[SAR Definition],[SAR value],[Prescan Reuse String],[Content Qualification],[Image Filtering Parameters],[Rx Stack Identification]
In both cases, we set as below: { "DICOMHome": "/opt/localdrive/Niffler/modules/cold-extraction/june28", "OutputDirectory": "June28_2021", "Depth": 3, "SplitIntoChunks": 1000, "PrintImages": false, "CommonHeadersOnly": false, "PublicHeadersOnly": false, "SpecificHeadersOnly": false, "UseProcesses": 0, "FlattenedToLevel": "patient", "is16Bit":true, "SendEmail": true, "YourEmail": "test@test.test" } { "DICOMHome": "/Users/pradeeban/Downloads/onedrive", "OutputDirectory": "/Users/pradeeban/Downloads/onedrive/meta1", "Depth": 0, "SplitIntoChunks": 1, "PrintImages": false, "CommonHeadersOnly": false, "PublicHeadersOnly": false, "SpecificHeadersOnly": false, "UseProcesses": 0, "FlattenedToLevel": "patient", "is16Bit":true, "SendEmail": true, "YourEmail": "test@test.test" }