Add pet2bids to ezbids

[bendhouseart]

Adding PET2BIDS to handler so that if PET dicoms are found it runs dcm2niix4pet instead of dcm2niix for those files.

[bendhouseart]

This a draft only at this time, getting failures when I drop PET dicoms into ezBIDS, but not 100% sure where things are going sideways (as is the way with working in a VM).

Currently getting this sort of output:

ezbids-handler-1  | 0|handler  | + echo 'replace file path that contains space'
ezbids-handler-1  | 0|handler  | + find /tmp/ezbids-workdir/6425a7a05f58323ab6d76491 -depth -name '*[ ()]*' -execdir rename 's/[ )]/_/g; s/\(//g' '{}' ';'
ezbids-handler-1  | 0|handler  | replace file path that contains space
ezbids-handler-1  | 0|handler  | + echo 'Making sure all NIfTI files are in .nii.gz format'
ezbids-handler-1  | 0|handler  | + touch /tmp/ezbids-workdir/6425a7a05f58323ab6d76491/nii_files
ezbids-handler-1  | 0|handler  | Making sure all NIfTI files are in .nii.gz format
ezbids-handler-1  | 0|handler  | + find /tmp/ezbids-workdir/6425a7a05f58323ab6d76491 -name '*.nii'
ezbids-handler-1  | 0|handler  | + '[' -s /tmp/ezbids-workdir/6425a7a05f58323ab6d76491/nii_files ']'
ezbids-handler-1  | 0|handler  | + echo 'processing /tmp/ezbids-workdir/6425a7a05f58323ab6d76491'
ezbids-handler-1  | 0|handler  | + echo 'finding dicom directories'
ezbids-handler-1  | 0|handler  | + ./find_dicomdir.py /tmp/ezbids-workdir/6425a7a05f58323ab6d76491
ezbids-handler-1  | 0|handler  | processing /tmp/ezbids-workdir/6425a7a05f58323ab6d76491
ezbids-handler-1  | 0|handler  | finding dicom directories
ezbids-handler-1  | 0|handler  | + cat /tmp/ezbids-workdir/6425a7a05f58323ab6d76491/dcm2niix.list
ezbids-handler-1  | 0|handler  | ./CPSInnovationsHRRT-NIMH
ezbids-handler-1  | 0|handler  | ++ which dcm2niix4pet
ezbids-handler-1  | 0|handler  | ++ echo 0
ezbids-handler-1  | 0|handler  | + PET2BIDS_INSTALLED=0
ezbids-handler-1  | 0|handler  | + '[' 0 -eq 0 ']'
ezbids-handler-1  | 0|handler  | + echo 'PET2BIDS is installed, using dcm2niix4pet for PET directories found in root dir /tmp/ezbids-workdir/6425a7a05f58323ab6d76491'
ezbids-handler-1  | 0|handler  | + export -f rundcm2niix4pet
ezbids-handler-1  | 0|handler  | + ./find_petdir.py /tmp/ezbids-workdir/6425a7a05f58323ab6d76491
ezbids-handler-1  | 0|handler  | PET2BIDS is installed, using dcm2niix4pet for PET directories found in root dir /tmp/ezbids-workdir/6425a7a05f58323ab6d76491
ezbids-api-1      | 0|ezbids  | Mongoose: sessions.findOne({ _id: new ObjectId("6425a7a05f58323ab6d76491") }, { projection: {} })
ezbids-handler-1  | 0|handler  | checking dcm2niix progress--------------------------
ezbids-handler-1  | 0|handler  | Mongoose: sessions.updateOne({ _id: new ObjectId("6425a7a05f58323ab6d76491") }, { '$set': { status_msg: 'PET2BIDS is installed, using dcm2niix4pet for PET directories found in root dir /tmp/ezbids-workdir/6425a7a05f58323ab6d76491', dicomCount: 2, update_date: new Date("Thu, 30 Mar 2023 15:15:53 GMT") }}, { session: null })
ezbids-api-1      | 0|ezbids  | Mongoose: sessions.findOne({ _id: new ObjectId("6425a7a05f58323ab6d76491") }, { projection: {} })
ezbids-mongodb-1  | {"t":{"$date":"2023-03-30T15:15:54.619+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:42082","connectionId":145,"connectionCount":10}}
ezbids-mongodb-1  | {"t":{"$date":"2023-03-30T15:15:54.623+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn145","msg":"client metadata","attr":{"remote":"127.0.0.1:42082","client":"conn145","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.15"},"os":{"type":"Linux","name":"Ubuntu","architecture":"x86_64","version":"20.04"}}}}
ezbids-mongodb-1  | {"t":{"$date":"2023-03-30T15:15:54.714+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn145","msg":"Connection ended","attr":{"remote":"127.0.0.1:42082","connectionId":145,"connectionCount":9}}
ezbids-handler-1  | 0|handler  | + echo 'Found PET directories:'
ezbids-handler-1  | 0|handler  | + cat /tmp/ezbids-workdir/6425a7a05f58323ab6d76491/pet2bids.list
ezbids-handler-1  | 0|handler  | Found PET directories:
ezbids-handler-1  | 0|handler  | ./CPSInnovationsHRRT-NIMH
ezbids-handler-1  | 0|handler  | + echo 'Removing PET directories from dcm2niix list'
ezbids-handler-1  | 0|handler  | + grep -v -x -f /dev/fd/63 /dev/fd/62
ezbids-handler-1  | 0|handler  | Removing PET directories from dcm2niix list

Ehhhh....

[bendhouseart]

Oh, this is new:

Line 1566 in ezBIDS_core.py:

uploaded_json_list = pd.read_csv("list", header=None, sep="\n").to_numpy().flatten().tolist()

### is throwing this error:
Traceback (most recent call last):
  File "./ezBIDS_core/ezBIDS_core.py", line 1566, in <module>
    uploaded_json_list = pd.read_csv("list", header=None, sep="\n").to_numpy().flatten().tolist()
  File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 211, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pandas/util/_decorators.py", line 331, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pandas/io/parsers/readers.py", line 935, in read_csv
    kwds_defaults = _refine_defaults_read(
  File "/usr/local/lib/python3.8/dist-packages/pandas/io/parsers/readers.py", line 2063, in _refine_defaults_read
    raise ValueError(
ValueError: Specified \n as separator or delimiter. This forces the python engine which does not accept a line terminator. Hence it is not allowed to use the line terminator as separator.

attached is list file in question.

list.txt

[dlevitas]

@bendhouseart is your python error occurring on your dev instance? I'm unable to reproduce the error on my dev instance or production itself, though I can produce the error on my local python installation. Regardless, using lineterminator will fix your issue: uploaded_json_list = pd.read_csv("list", header=None, lineterminator="\n").to_numpy().flatten().tolist()

[bendhouseart]

@bendhouseart is your python error occurring on your dev instance? I'm unable to reproduce the error on my dev instance or production itself, though I can produce the error on my local python installation. Regardless, using lineterminator will fix your issue: uploaded_json_list = pd.read_csv("list", header=None, lineterminator="\n").to_numpy().flatten().tolist()

It's occurring on my branch https://github.com/bendhouseart/ezbids/tree/pet with any kind of dicom.

Seems like a pandas thing as I rebuild the containers earlier this week. What version of pandas are you using? Mine is 1.5.3 in the container:

root@5ca947864733:/app/handler# python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> pandas.__version__
'1.5.3'

[dlevitas]

ezBIDS's pandas version is admittingly a bit dated (version 1.0.1). The lineterminator="\n" should work for pandas versions 1.5.3 and 1.0.1 though

[bendhouseart]

Okay, I think it's time to move onto Vue/JS stuff??

Take a look at the below screenshot and let me know what is next:

Seems like I need to make sure to:

• not exclude PET
• include some fields PHI fields we drop when we run PET2BIDS, I'm thinking of adding an ezBIDS flag to ensure our json output matches with what you expect. e.g. PatientName, PatientID, PatientBirthDate, AcquisitionDateTime etc.

[dlevitas]

Looks like things are coming along nicely! Be aware that the message in your screenshot is being triggered because ezBIDS doesn't like the data array's dtype, thinking it's something other than imaging data (see here).

Here's the Vue file for what information is displayed on the Series Mapping page. Importantly, we'll need to display acceptable entity labels for the specific datatype & suffix combination, and alert them if required labels aren't specified. The PatientName, PatientID, PatientBirthDate, AcquisitionDateTime and other information isn't necessary.

Let us know if you need help with the Vue/JS integration.

[bendhouseart]

Be aware that the message in your screenshot is being triggered because ezBIDS doesn't like the data array's dtype, thinking it's something other than imaging data (see here).

Hmm, yeah, will update the warning so it doesn't get so mad at PET.

Let us know if you need help with the Vue/JS integration.

That link should suffice for now; I'll reach out if/when I I run into trouble.

Thanks and have a great weekend!

[bendhouseart]

Updated w/ changes discussed earlier, removed draft from PR name to indicated this PR is ready to merge.