search

tbepler / topaz

Pipeline for particle picking in cryo-electron microscopy images using convolutional neural networks trained from positive and unlabeled examples. Also featuring micrograph and tomogram denoising with DNNs.
GNU General Public License v3.0
179 stars 63 forks source link

cryosparc import error #206

Closed gnye8 closed 2 months ago

gnye8 commented 2 months ago

I am having some issues with running topaz in cryoSPARC. It is generating an error that I do not get when I run topaz on its own, using the same command.

I am attempting to install topaz on a cluster which uses a batch submission system. I have installed topaz version 0.2.5a in a conda environment. We have cryosparc v4.5.3 installed.

Here is the batch submission script I use to run topaz on its own (which runs successfully): 

#!/bin/sh -l
#SBATCH --account=cryoem:ca105
#SBATCH --partition=turing
#SBATCH --job-name=topaz
#SBATCH --mem-per-cpu=3800m
#SBATCH --time=0-24:00:00
#SBATCH --ntasks-per-node=4
#SBATCH --nodes=1
#SBATCH --gpus=4
#SBATCH --error=/sdf/group/cryoem/g/CA105/GPN_CS/CS-fse/imports/J194_denoised/topaz.err
#SBATCH --output=/sdf/group/cryoem/g/CA105/GPN_CS/CS-fse/imports/J194_denoised/topaz.out

source activate topaz
topaz denoise /sdf/group/cryoem/g/CA105/GPN_CS/CS-fse/exports/jobs/J194_exposure_sets/J194_split_0/*/*/*patch_aligned_doseweighted.mrc --output /sdf/group/cryoem/g/CA105/GPN_CS/CS-fse/J200/denoised_micrographs

To use topaz with cryosparc, I use the following wrapper script: 

#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
    conda deactivate > /dev/null 2>&1 || true  # ignore any errors
    conda deactivate > /dev/null 2>&1 || true  # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH

if [[ $@ == '--version' ]]; then
        echo '0.2.5'
        exit 0
fi

source /sdf/group/cryoem/sw/conda/etc/profile.d/conda.sh
conda activate topaz

topaz $@

conda deactivate

Finally, when I run denoising in cryosparc, here is the output: 

 [CPU:  235.8 MB  Avail: 113.53 GB]

Starting Topaz process using version 0.2.5...
[CPU:  235.8 MB  Avail: 113.53 GB]

Using Topaz provided pretrained model.
[CPU:  235.8 MB  Avail: 113.53 GB]

Beginning Topaz denoising command by running command /sdf/home/g/gnye88/topaz.sh denoise [300 MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY] --device 0 --format mrc --patch-size 1024 --patch-padding 500 --output /sdf/group/cryoem/g/CA105/GPN_CS/CS-fse/J205/denoised_micrographs --lowpass 1 --gaussian 0 --inv-gaussian 0 --deconv-patch 1 --pixel-cutoff 0 --model unet

[CPU:  235.8 MB  Avail: 113.53 GB]

Distributing over 2 processes...
[CPU:  235.8 MB  Avail: 113.49 GB]

/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/numpy/__init__.py:138: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
[CPU:  235.8 MB  Avail: 113.49 GB]

/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/numpy/__init__.py:138: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
[CPU:  235.8 MB  Avail: 113.49 GB]

from . import _distributor_init
[CPU:  235.8 MB  Avail: 113.49 GB]

from . import _distributor_init
[CPU:  235.8 MB  Avail: 113.49 GB]

Traceback (most recent call last):
[CPU:  235.8 MB  Avail: 113.49 GB]

Traceback (most recent call last):
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/bin/topaz", line 33, in <module>
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/bin/topaz", line 33, in <module>
[CPU:  235.8 MB  Avail: 113.49 GB]

sys.exit(load_entry_point('topaz-em==0.2.5', 'console_scripts', 'topaz')())
[CPU:  235.8 MB  Avail: 113.49 GB]

sys.exit(load_entry_point('topaz-em==0.2.5', 'console_scripts', 'topaz')())
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 60, in main
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 60, in main
[CPU:  235.8 MB  Avail: 113.49 GB]

import topaz.commands.train
[CPU:  235.8 MB  Avail: 113.49 GB]

import topaz.commands.train
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/topaz/commands/train.py", line 8, in <module>
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/topaz/commands/train.py", line 8, in <module>
[CPU:  235.8 MB  Avail: 113.49 GB]

import numpy as np
[CPU:  235.8 MB  Avail: 113.49 GB]

import numpy as np
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/numpy/__init__.py", line 230, in <module>
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/numpy/__init__.py", line 230, in <module>
[CPU:  235.8 MB  Avail: 113.51 GB]

import mkl
[CPU:  235.8 MB  Avail: 113.53 GB]

import mkl
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/mkl/__init__.py", line 48, in <module>
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/mkl/__init__.py", line 48, in <module>
[CPU:  235.8 MB  Avail: 113.53 GB]

with RTLD_for_MKL():
[CPU:  235.8 MB  Avail: 113.53 GB]

with RTLD_for_MKL():
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/mkl/__init__.py", line 33, in __enter__
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/mkl/__init__.py", line 33, in __enter__
[CPU:  235.8 MB  Avail: 113.53 GB]

import ctypes
[CPU:  235.8 MB  Avail: 113.53 GB]

import ctypes
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/ctypes/__init__.py", line 7, in <module>
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/ctypes/__init__.py", line 7, in <module>
[CPU:  235.8 MB  Avail: 113.53 GB]

from _ctypes import Union, Structure, Array
[CPU:  235.8 MB  Avail: 113.53 GB]

from _ctypes import Union, Structure, Array
[CPU:  235.8 MB  Avail: 113.53 GB]

ImportError: libffi.so.6: cannot open shared object file: No such file or directory
[CPU:  235.8 MB  Avail: 113.53 GB]

ImportError: libffi.so.6: cannot open shared object file: No such file or directory
[CPU:  236.0 MB  Avail: 113.52 GB]

Topaz denoising command complete in 2.766s.
tbepler commented 2 months ago

When these issues have come up before it has usually be related to an environment problem or versioning problem and that appears to be the case here, since it looks like this is an import error with a system library. You'll might get better help on the cryosparc forums.

gnye8 commented 2 months ago

Okay, thank you! I will do that.