search

muscbridge / PyDesigner

A hands-free DTI, DKI, FBI and FBWM preprocessing pipeline. Information on algorithms and preprocessing steps are available at https://www.biorxiv.org/content/10.1101/2021.10.20.465189v1 A video tutorial on PyDesigner and its usage is now available at https://www.youtube.com/watch?v=mChQFuQqX3k
https://pydesigner.readthedocs.io/en/latest/
Other
23 stars 7 forks source link

rpe_pair option #194

Closed jchen33344 closed 4 years ago

jchen33344 commented 4 years ago

Hello, I've been running pydesigner this way: python3.6 pydesigner.py \ -o $outpath \ --force --denoise --extent 5,5,5 --degibbs --rician \ --undistort --smooth \ --wmti --mask --verbose \ $dwi1,$dwi2

However, I realize that the default rpe option is rpe_header. It's hardcoded in the script, so I'm wondering if we'll eventually be able to use the rpe_pair option where we can input image of the opposite phase encoding.

Thank you!

TheJaeger commented 4 years ago

Hello!

You may introduce your phase encoding dataset as another input such that your inputs are $dwi1,$dwi2,$reverse_pe.

We omitted all possibilities of having users specify rpe_type because it over-complicated coding, while exposing the pipeline to user errors. As long as you have a .JSON file, PyDesigner should automatically process your DWI with the proper corrections.

jchen33344 commented 4 years ago

Ok. I see. I tried running with my rpe image, but I'm getting this error when pydesigner is converting it to .mif: [ERROR] bvecs and bvals files must have same number of diffusion directions as DW-image (gradients: 1, image: 81)

it looks like it is trying to run: mrconvert -stride 1,2,3,4 -force -fslgrade bvec bval -json_import json dwi2.mif

jchen33344 commented 4 years ago

mrconvert -stride 1,2,3,4 -force -fslgrade bvec bval -json_import json rpe_img dwi2.mif

TheJaeger commented 4 years ago

Could you share your entire console output so I could trace the error?

jchen33344 commented 4 years ago

mrconvert: [WARNING] existing output files will be overwritten mrconvert: [100%] copying from "/mnt/labsp...JS/M0065/json/DIFF_meso.nii" to "/home/jc83...k/M0065_pydesigner/dwi0.mif" mrconvert: [WARNING] existing output files will be overwritten mrconvert: [100%] copying from "/mnt/labsp...n/DIFF_meso_research_s6.nii" to "/home/jc83...k/M0065_pydesigner/dwi1.mif" mrconvert: [WARNING] existing output files will be overwritten mrconvert: [ERROR] bvecs and bvals files must have same number of diffusion directions as DW-image (gradients: 1, image: 81) mrconvert: [ERROR] error importing diffusion gradient table for image "/mnt/labspace/Projects/MESO/ALLSUBJS/M0065/json/DIFF_meso_PA_s7.nii" Traceback (most recent call last): File "/home/jc8329/PyDesigner/pydesigner.py", line 823, in main() File "/home/jc8329/PyDesigner/pydesigner.py", line 238, in main resume=args.resume) File "/home/jc8329/PyDesigner/designer/preprocessing/util.py", line 495, in cat raise Exception('Please use the "--force" flag to ' Exception: Please use the "--force" flag to overwrite existing outputs, or clear the output directory

TheJaeger commented 4 years ago

Remove everything from your output directory and run this subject again. The error seems to indicate concatenation issue because of pre-existing files.

jchen33344 commented 4 years ago

I checked and my output directory is empty. I wrote the bash script so it copies the folder to another drive and the output directory is deleted.

TheJaeger commented 4 years ago

The other possible source of error I see are in BVEC/BVCAL pair files DIFF_meso_PA_s7. It appears that this DWI has 81 volumes, but only one gradient. Every 3D volume needs a corresponding gradient. If this is a DWI of 81 B0 volumes, you need to have 81 0, 0, 0 gradient triplets in your .bvec file. Likewise for .bval

EDIT: If this is a volume composed of just B0s, you may remove both .bval and .bvec files. PyDesigner assumes that files accompanied without BVAL/BVEC pairs are B0s, and will create its own properly formatted gradient files and use those.

jchen33344 commented 4 years ago

The DWI image is only one volume. I've tried removing .bval and .bvec files, but the newly created .bval and .bvec files using json2fslgrad are still giving the same error.

jchen33344 commented 4 years ago

I tried to convert DIFF_meso_PA using: mrconvert -stride 1,2,3,4 -force -json_import $pa_json $pa_nii $pa.mif So, without the bvec and bval files and it created the mif file, but of course, I get the same error (1 gradient, 81 images) when including the gradient files.

TheJaeger commented 4 years ago

Would you mind anonymizing and sharing this dataset on Box? I can't pinpoint the exact issue. Another dataset I have from NYU for one of our collaborative studies containing similar protocols DIFF_meso_2, DIFF_meso_PA, and MICSI_BRAIN_MAPPING_meso is working fine on my end.

jchen33344 commented 4 years ago

Yes, I don't use Box, but can share the dataset via google drive. I'll send the link to your email. Thanks!

TheJaeger commented 4 years ago

Thanks for the data. I was able to process all three DWIs to completion with no problems at all, using both v0.2 and developmental branch. You may view the attached log file to see the command I gave it.

processing.log

I'd recommend upgrading to v0.2 if you're not already using it because it fixes that json2grad function. We've had issues with it in the past.

jchen33344 commented 4 years ago

I cloned the PyDesigner repository recently (within 2 weeks ago). Not sure why it's not running for me. I'm still unable to run it completely. I've been calling PyDesigner like this: python3.6 /home/jc8329/PyDesigner/pydesigner.py Followed by the arguments instead of installing and configuring so I can just call it with 'pydesigner', but I'm not sure that would be why. One other thing I did was move the pydesigner.py file out of the 'designer' folder so it is just in the 'PyDesigner' folder because I was getting this error: Traceback (most recent call last): File "/home/jc8329/PyDesigner/designer/pydesigner.py", line 18, in from designer.preprocessing import util, preparation, snrplot, mrinfoutil, mrpreproc ModuleNotFoundError: No module named 'designer'

Sorry, I'm not sure if I'm making sense, but let me know if I should elaborate.

TheJaeger commented 4 years ago

I'm surprised you got it to work without installation :O, it shouldn't be possible. I'd suggest installing it so both our configuration is on the same page. You may install this in a conda environment so you don't mess up your OS's Python configuration. Refer to our documentation for doing this here: https://pydesigner.readthedocs.io/en/develop/index.html

So do this:

  1. Remove your copy of PyDesigner
  2. Clone the 'develop' branch of PyDesigner
  3. Open CLI and cd into your new PyDesigner location
  4. Run pip install -e . to install PyDesigner in editable mode (so you may make changes to the code without reinstalling)

Try processing again and let me know if you get any further.

jchen33344 commented 4 years ago

Ok. I see. I've been running these on a server that has FSL and mrtrix installed, so I thought that would be enough to run successfully. I've been running Designer through this server, so I've been trying to do the same with PyDesigner. It doesn't have Anaconda installed and I don't have permission to install or pip install. I'm using a Window's laptop and I see we can install Docker, so I'll try that.

TheJaeger commented 4 years ago

It's going to be hard to test without installing PyD. Roll with Docker then, which will install FSL, MRTrix3, PyDesigner and all other dependencies into an image. To pull a pre-built develop branch, run the command docker pull dmri/neurodock:dev.

Running with the Docker container is quite similar to running natively, you'd just have to adjust your input paths and add a few prefix commands. Let me know if you need help running this.

jchen33344 commented 4 years ago

It looks like I need Window's 10 pro to install Docker, but I don't have that. I'll have to check if I can get permission to install on the server.

jchen33344 commented 4 years ago

Hello, we were able to install miniconda and created an environment to install pydesigner successfully: pip install -e . Obtaining file:///cbi05data/data1/Hamster/Projects/PyDesigner Requirement already satisfied: numpy in /usr/local/miniconda3/lib/python3.7/site-packages (from PyDesigner==0.2) (1.18.2) Requirement already satisfied: scipy in /usr/local/miniconda3/lib/python3.7/site-packages (from PyDesigner==0.2) (1.4.1) Requirement already satisfied: matplotlib in /usr/local/miniconda3/lib/python3.7/site-packages (from PyDesigner==0.2) (3.2.1) Requirement already satisfied: py-cpuinfo in /usr/local/miniconda3/lib/python3.7/site-packages (from PyDesigner==0.2) (5.0.0) Requirement already satisfied: joblib in /usr/local/miniconda3/lib/python3.7/site-packages (from PyDesigner==0.2) (0.14.1) Requirement already satisfied: tqdm in /usr/local/miniconda3/lib/python3.7/site-packages (from PyDesigner==0.2) (4.44.1) Requirement already satisfied: multiprocess in /usr/local/miniconda3/lib/python3.7/site-packages (from PyDesigner==0.2) (0.70.9) Requirement already satisfied: nibabel in /usr/local/miniconda3/lib/python3.7/site-packages (from PyDesigner==0.2) (3.0.2) Requirement already satisfied: cvxpy in /usr/local/miniconda3/lib/python3.7/site-packages (from PyDesigner==0.2) (1.0.29) Requirement already satisfied: python-dateutil>=2.1 in /usr/local/miniconda3/lib/python3.7/site-packages (from matplotlib->PyDesigner==0.2) (2.8.1) Requirement already satisfied: cycler>=0.10 in /usr/local/miniconda3/lib/python3.7/site-packages (from matplotlib->PyDesigner==0.2) (0.10.0) Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/miniconda3/lib/python3.7/site-packages (from matplotlib->PyDesigner==0.2) (1.2.0) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/miniconda3/lib/python3.7/site-packages (from matplotlib->PyDesigner==0.2) (2.4.7) Requirement already satisfied: dill>=0.3.1 in /usr/local/miniconda3/lib/python3.7/site-packages (from multiprocess->PyDesigner==0.2) (0.3.1.1) Requirement already satisfied: ecos>=2 in /usr/local/miniconda3/lib/python3.7/site-packages (from cvxpy->PyDesigner==0.2) (2.0.7.post1) Requirement already satisfied: osqp>=0.4.1 in /usr/local/miniconda3/lib/python3.7/site-packages (from cvxpy->PyDesigner==0.2) (0.6.1) Requirement already satisfied: scs>=1.1.3 in /usr/local/miniconda3/lib/python3.7/site-packages (from cvxpy->PyDesigner==0.2) (2.1.2) Requirement already satisfied: six>=1.5 in /usr/local/miniconda3/lib/python3.7/site-packages (from python-dateutil>=2.1->matplotlib->PyDesigner==0.2) (1.14.0) Requirement already satisfied: future in /usr/local/miniconda3/lib/python3.7/site-packages (from osqp>=0.4.1->cvxpy->PyDesigner==0.2) (0.18.2) Installing collected packages: PyDesigner Attempting uninstall: PyDesigner Found existing installation: PyDesigner 0.2 Uninstalling PyDesigner-0.2: Successfully uninstalled PyDesigner-0.2 Running setup.py develop for PyDesigner Successfully installed PyDesigner

However, I'm still getting this error when I run pydesigner in this conda env: mrconvert: [WARNING] existing output files will be overwritten mrconvert: [100%] copying from "/mnt/labsp...JS/M0065/json/DIFF_meso.nii" to "/home/jc83...k/M0065_pydesigner/dwi0.mif" mrconvert: [WARNING] existing output files will be overwritten mrconvert: [100%] copying from "/mnt/labsp...n/DIFF_meso_research_s6.nii" to "/home/jc83...k/M0065_pydesigner/dwi1.mif" mrconvert: [WARNING] existing output files will be overwritten mrconvert: [ERROR] bvecs and bvals files must have same number of diffusion directions as DW-image (gradients: 1, image: 81) mrconvert: [ERROR] error importing diffusion gradient table for image "/mnt/labspace/Projects/MESO/ALLSUBJS/M0065/json/DIFF_meso_PA_s7.nii" Traceback (most recent call last): File "/usr/local/miniconda3/bin/pydesigner", line 11, in load_entry_point('PyDesigner', 'console_scripts', 'pydesigner')() File "/cbi05data/data1/Hamster/Projects/PyDesigner/designer/pydesigner.py", line 238, in main resume=args.resume) File "/cbi05data/data1/Hamster/Projects/PyDesigner/designer/preprocessing/util.py", line 495, in cat raise Exception('Please use the "--force" flag to ' Exception: Please use the "--force" flag to overwrite existing outputs, or clear the output directory

TheJaeger commented 4 years ago

Clear your output directory, then delete the following two files and try again:

/mnt/labspace/Projects/MESO/ALLSUBJS/M0065/json/DIFF_meso_PA_s7.bvec /mnt/labspace/Projects/MESO/ALLSUBJS/M0065/json/DIFF_meso_PA_s7.bval

jchen33344 commented 4 years ago

I checked my output directory before running and removed meso PA bval and bvec files, but it's still getting the same error with the bval and bvec files generated by PyDesigner.

TheJaeger commented 4 years ago

I think I know what's going on here. The BVAL/BVEC pair being produced by PyD needs to be transposed. Let me push an update to address this.

TheJaeger commented 4 years ago

@jchen33344 PyDesigner v0.32 found at https://github.com/m-ama/PyDesigner/releases/tag/v0.32 should fix this issue. Let me know whether it works. I also noticed that you are extracting WMTI parameters, which are inaccurate in v0.2. The update will resolve not just your BVEC/BVAL issue, but also produce better WMTI metrics.

EDIT: Otherwise, the only other possible source of error is mrtrix3 itself. You can try updating yours. I cannot replicate this issue on my end at all - this problematic dataset processes perfectly on my end.

jchen33344 commented 4 years ago

Hello, thanks for the new version. I tried it and it still gave the same error, so it could be mrtrix3. It seems the problem is that while converting to .mif, it detects 81 images although it's only 1.

TheJaeger commented 4 years ago

Try running mrinfo [path to nifti] on the problematic nifti and share the console output here.

jchen33344 commented 4 years ago

Image: "DIFF_meso_PA_s7.nii"

Dimensions: 130 x 130 x 50 Voxel size: 1.69231 x 1.69231 x 3 Data strides: [ -1 -2 3 ] Format: NIfTI-1.1 Data type: unsigned 16 bit integer (little endian) Intensity scaling: offset = 0, multiplier = 1 Transform: 0.9962 -9.582e-10 0.08727 -107.6 0.0188 0.9765 -0.2147 -103 -0.08522 0.2155 0.9728 -45.55 comments: untitled mrtrix_version: 3.0_RC3-159-gdfda38fd

TheJaeger commented 4 years ago

This looks fine, even by MRtrix3's perspective. It's reading it correctly as a single-volume 3D file. One last thing you can do to isolate this issue to MRtrix3 goes back to your previous comment on manually converting this to a mif, which also gave you an error. This time though, use the BVAL/BVEC pair that PyDesigner creates:

mrconvert -stride 1,2,3,4 -force -json_import -fslgrad $bvec $bval $pa_json $pa_nii $pa.mif

Your problem is definitely MRtrix3 if this creates an error.

jchen33344 commented 4 years ago

Yes, I've tried running mrconvert that way with the bvec/bval files both the original I had and the ones generated from PyDesigner and I get the same error where it's saying bvec/bval does not have the same diffusion directions as dwi image (gradients: 1, image: 81).

We'll look into updating mrtrix3 and reporting the problem there.

TheJaeger commented 4 years ago

I discovered another issues lately that may be related to this. The JSON sidecar may be incomplete in some rare instances, which confuses MRtrix3. This is usually a missing } bracket at the end of the file.

Try running the same mrconvert command without JSON import, like:

mrconvert -stride 1,2,3,4 -force -fslgrad $bvec $bval $pa_nii $pa.mif

Let me know if this command converts to .mif successfully.

jchen33344 commented 4 years ago

It is still giving the same error. mrconvert: [ERROR] bvecs and bvals files must have same number of diffusion directions as DW-image (gradients: 1, image: 81) mrconvert: [ERROR] error importing diffusion gradient table for image "/mnt/labspace/Projects/MESO/ALLSUBJS/M0065/json/DIFF_meso_PA_s7.nii"

Sorry. We haven't had the chance to try by updating mrtrix3 yet, but I will let you know when we do.

TheJaeger commented 4 years ago

Yeah looks like you'll have to update MRtrix3. Sorry I can't replicate the issue.

TheJaeger commented 4 years ago

Issue is stale. Closing until user responds.