search

PennLINC / qsiprep

Preprocessing of diffusion MRI
http://qsiprep.readthedocs.io
BSD 3-Clause "New" or "Revised" License
138 stars 56 forks source link

"Intel MKL FATAL ERROR" `cannot load libmkl_intel_lp64.so` #478

Closed dyhan316 closed 1 year ago

dyhan316 commented 1 year ago

Hello! I was using qsiprep on TACC, and encountered into the following error :

221019-09:15:33,902 nipype.workflow INFO:
     Running with omp_nthreads=8, nthreads=272
221019-09:15:34,3 nipype.workflow IMPORTANT:

    Running qsiprep version 0.14.3:
      * BIDS dataset path: /data.
      * Participant list: ['200139'].
      * Run identifier: 20221019-091451_243262a1-c605-40db-9d41-2bbb35c16816.

221019-09:15:36,279 nipype.utils WARNING:
     A newer version (1.8.4) of nipy/nipype is available. You are using 1.6.1
221019-09:16:35,630 nipype.workflow INFO:
     Combining all 1 dwis within the single available session
221019-09:16:37,407 nipype.workflow INFO:
     [{'dwi_series': ['/data/sub-200139/dwi/sub-200139_dwi.nii.gz'], 'fieldmap_info': {'suffix': None}, 'dwi_series_pedir': 'j', 'concatenated_bids_name': 'sub-200139'}]
221019-09:16:40,585 nipype.workflow IMPORTANT:
     Creating dwi processing workflow "dwi_preproc_wf" to produce output sub-200139 (1.03 GB / 36 DWIs). Memory resampled/largemem=1.10/1.13 GB.
221019-09:16:40,873 nipype.workflow INFO:
     Automatically using 5, 5, 5 window for dwidenoise
221019-09:16:44,594 nipype.workflow INFO:
     Using 8 threads in eddy
221019-09:18:12,90 nipype.workflow IMPORTANT:
     Works derived from this qsiprep execution should include the following boilerplate:

Preprocessing was performed using *QSIPrep* 0.14.3,
which is based on *Nipype* 1.6.1
(@nipype1; @nipype2; RRID:SCR_002502).

Anatomical data preprocessing

: The T1-weighted (T1w) image was corrected for intensity non-uniformity (INU)
using `N4BiasFieldCorrection` [@n4, ANTs 2.3.1],
and used as T1w-reference throughout the workflow.
The T1w-reference was then skull-stripped using `antsBrainExtraction.sh`
(ANTs 2.3.1), using OASIS as target template.
Brain surfaces were reconstructed using `recon-all` [FreeSurfer 6.0.1,
RRID:SCR_001847, @fs_reconall], and the brain mask estimated
previously was refined with a custom variation of the method to reconcile
ANTs-derived and FreeSurfer-derived segmentations of the cortical
gray-matter of Mindboggle [RRID:SCR_002438, @mindboggle].
Spatial normalization to the ICBM 152 Nonlinear Asymmetrical
template version 2009c [@mni, RRID:SCR_008796] was performed
through nonlinear registration with `antsRegistration`
[ANTs 2.3.1, RRID:SCR_004757, @ants], using
brain-extracted versions of both T1w volume and template.
Brain tissue segmentation of cerebrospinal fluid (CSF),
white-matter (WM) and gray-matter (GM) was performed on
the brain-extracted T1w using `FAST` [FSL 6.0.3:b862cdd5, RRID:SCR_002823,
@fsl_fast].

Diffusion data preprocessing

: Any images with a b-value less than 100 s/mm^2 were treated as a *b*=0 image. MP-PCA denoising as implemented in MRtrix3's `dwidenoise`[@dwidenoise1] was applied with a 5-voxel window. After MP-PCA, Gibbs unringing was performed using MRtrix3's `mrdegibbs` [@mrdegibbs]. Following unringing, B1 field inhomogeneity was corrected using `dwibiascorrect` from MRtrix3 with the N4 algorithm [@n4]. After B1 bias correction, the mean intensity of the DWI series was adjusted so all the mean intensity of the b=0 images matched across eachseparate DWI scanning sequence.

FSL (version 6.0.3:b862cdd5)'s eddy was used for head motion correction and Eddy current correction [@anderssoneddy]. Eddy was configured with a $q$-space smoothing factor of 10, a total of 5 iterations, and 1000 voxels used to estimate hyperparameters. A linear first level model and a linear second level model were used to characterize Eddy current-related spatial distortion. $q$-space coordinates were forcefully assigned to shells. Field offset was attempted to be separated from subject movement. Shells were aligned post-eddy. Eddy's outlier replacement was run [@eddyrepol]. Data were grouped by slice, only including values from slices determined to contain at least 250 intracerebral voxels. Groups deviating by more than 4 standard deviations from the prediction had their data replaced with imputed values. Final interpolation was performed using the `jac` method.

Several confounding time-series were calculated based on the
preprocessed DWI: framewise displacement (FD) using the
implementation in *Nipype* [following the definitions by @power_fd_dvars].
The head-motion estimates calculated in the correction step were also
placed within the corresponding confounds file. Slicewise cross correlation
was also calculated.
The DWI time-series were resampled to ACPC,
generating a *preprocessed DWI run in ACPC space* with 1.2mm isotropic voxels.

Many internal operations of *QSIPrep* use
*Nilearn* 0.8.1 [@nilearn, RRID:SCR_001362] and
*Dipy* [@dipy].
For more details of the pipeline, see [the section corresponding
to workflows in *QSIPrep*'s documentation](https://qsiprep.readthedocs.io/en/latest/workflows.html "QSIPrep's documentation").

### References

221019-09:18:22,577 nipype.workflow WARNING:
     Could not generate CITATION.html file:
pandoc -s --bibliography /usr/local/miniconda/lib/python3.7/site-packages/qsiprep/data/boilerplate.bib --filter pandoc-citeproc /out/qsiprep/logs/CITATION.md -o /out/qsiprep/logs/CITATION.html
2022-10-19 09:25:21,393 [WARNING] DueCredit internal failure while running <function _get_active_due at 0x2b85149ce378>: MemoryError(). Please report to developers at https://github.com/duecredit/duecredit/issues (utils.py:194)
INTEL MKL ERROR: /usr/local/miniconda/lib/python3.7/site-packages/mkl/../../../libmkl_core.so: cannot make segment writable for relocation: Cannot allocate memory.
Intel MKL FATAL ERROR: Cannot load libmkl_core.so.
INTEL MKL ERROR: /usr/local/miniconda/lib/python3.7/site-packages/mkl/../../../libmkl_core.so: cannot make segment writable for relocation: Cannot allocate memory.
Intel MKL FATAL ERROR: Cannot load libmkl_core.so.
INTEL MKL ERROR: /usr/local/miniconda/lib/python3.7/site-packages/mkl/../../../libmkl_core.so: failed to map segment from shared object.
Intel MKL FATAL ERROR: Cannot load libmkl_core.so.
INTEL MKL ERROR: /usr/local/miniconda/lib/python3.7/site-packages/mkl/../../../libmkl_core.so: failed to map segment from shared object.
Intel MKL FATAL ERROR: Cannot load libmkl_core.so.
INTEL MKL ERROR: /usr/local/miniconda/lib/python3.7/site-packages/mkl/../../../libmkl_core.so: cannot make segment writable for relocation: Cannot allocate memory.
Intel MKL FATAL ERROR: Cannot load libmkl_core.so.
2022-10-19 09:25:27,871 [WARNING] DueCredit internal failure while running <function _get_active_due at 0x2b85149ce400>: MemoryError(). Please report to developers at https://github.com/duecredit/duecredit/issues (utils.py:194)
INTEL MKL ERROR: /usr/local/miniconda/lib/python3.7/site-packages/mkl/../../../libmkl_core.so: cannot make segment writable for relocation: Cannot allocate memory.
Intel MKL FATAL ERROR: Cannot load libmkl_core.so.
INTEL MKL ERROR: /usr/local/miniconda/lib/python3.7/site-packages/mkl/../../../libmkl_intel_lp64.so: failed to map segment from shared object.
I2022-10-19 09:25:33,349 [WARNING] DueCredit internal failure while running <function _get_active_due at 0x2b85149cd400>: ImportError('/usr/local/miniconda/lib/python3.7/lib-dynload/_csv.cpython-37m-x86_64-linux-gnu.so: failed to map segment from shared object'). Please report to developers at https://github.com/duecredit/duecredit/issues (utils.py:194)
ntel MKL FATAL ERROR: Cannot load libmkl_intel_lp64.so.
INTEL MKL ERROR: /usr/local/miniconda/lib/python3.7/site-packages/mkl/../../../libmkl_core.so: failed to map segment from shared object.
Intel MKL FATAL ERROR: Cannot load libmkl_core.so.
INTEL MKL ERROR: /usr/local/miniconda/lib/python3.7/site-packages/mkl/../../../libmkl_intel_lp64.so: failed to map segment from shared object.
Intel MKL FATAL ERROR: Cannot load libmkl_intel_lp64.so.
INTEL MKL ERROR: /usr/local/miniconda/lib/python3.7/site-packages/mkl/../../../libmkl_intel_lp64.so: failed to map segment from shared object.
Intel MKL FATAL ERROR: Cannot load libmkl_intel_lp64.so.
2022-10-19 09:25:33,387 [WARNING] DueCredit internal failure while running <function _get_active_due at 0x2b85149ce400>: MemoryError(). Please report to developers at https://github.com/duecredit/duecredit/issues (utils.py:194)

At first I thought that "Cannot allocate memory" refers to the fact that there is not enough memory, so I made it so that each subject had about 6GB of RAM but I still got the same error... Is there something I can do to solve this issue? Thank you in advance for any help/comment!

mattcieslak commented 1 year ago

It looks like you're trying to use an extremely large number of threads

Running with omp_nthreads=8, nthreads=272

Can you try --nthreads 32 or something considerably smaller than 272?

dyhan316 commented 1 year ago

Thank you! Somehow that solved the problem:)