LBEM-CH / focus

High Throughput Electron Microscopy Image Processing Software
http://www.focus-em.org
GNU General Public License v2.0
35 stars 14 forks source link

problems with particle extraction from 2d crystal #242

Open EthanJUCB opened 4 years ago

EthanJUCB commented 4 years ago

Hi, I have some problems during particle extraction. In the beginning, I was able to extract particles from several images with the error" No handlers could be found for logger "mrcz.test_mrcz" . I successfully had phase flipped stacks for crystal averages.

Then I tried to include more images using 8 threads. Somehow, only 1 thread was running according to system monitor and scripts stopped after extracting particles from up to 4 images no matter how many image I selected.

After that, I had the worst one, script completely stopped extracting even for a single image. It now shows similar error"No handlers could be found for logger "mrcz.test_mrcz" but with additional message " then: then/endif not found."

I have no idea how to fix it now. Please see attached log. Many thanks.

Best Xi

###########################PickExportParticleStack############################

############################################

Parameters requested by the Script

############################################ thread_number = 8 scriptname = spr_sample_pixel = 0.5568 spr_doresample = n spr_resmin = 100.0 spr_resmax = 10.0 spr_phaseshift = 0.0,180.0 spr_boxsize = 200 spr_sigcc = 0.3 spr_invertcontrast = y spr_normalizebox = y spr_calculatedefocustilted = y spr_savephaseflipped = y spr_savectfmultiplied = n spr_savewienerfiltered = n spr_wienerconstant = 0.4 spr_sigmanorm = 1.0 spr_radnorm = 0.90 spr_whichtiltgeometry = Defocus spr_whichctf = Particle spr_whichhalfset = particle-based spr_savepickfig = y cs = 2.6 kv = 300 sample_pixel = 0.5568 phacon = 0.995 magnification = 89798.9 use_masked_image = n spr_ignore_lat2 = y spr_ignore_c1 = y spr_createstarfile = y spr_dir = SPR spr_imgs_dir = spr_mergefile = spr_stacks_dir = spr_picking_dir = spr_stack_rootname =

############################################

Parameters from the Preferences settings

############################################ app_python = /usr/bin/python2.7 app_cadaver = /usr/bin/cadaver app_anaconda = /usr/bin/python2.7 dir_ccp4 = /usr/local/ccp4-7.0 app_gawk = /usr/bin/gawk cuda_library = /usr/local/cuda/lib64 app_unblur = /usr/local/cina/unblur/unblur_1.0.2/bin/unblur_openmp_7_17_15.exe app_motioncor2 = /home/xijiang/Downloads/Motioncor2/MotionCor2-01-30-2017 app_gctf = /home/xijiang/Downloads/Gctf_v1.06/bin/Gctf-v1.06_sm_30_cu8.0_x86_64 app_ctffind = /home/xijiang/Downloads/CTFFIND4/bin/ctffind app_gautomatch = /home/xijiang/Downloads/Gautomatch_v0.53/bin/Gautomatch-v0.53_sm_20_cu8.0_x86_64 dir_eman2 = /usr/local/EMAN2/bin dir_imod = /usr/local/IMOD dir_frealign = /home/xijiang/Downloads/frealign-2dx-master/bin dir_relion = /usr/local/relion3 Microscope_Name = CryoEM Default_CS = 2.6 Default_KV = 300.0 Default_phase_shift_doit = n status_folder_update = n status_folder = none status_microscope = Titan status_refresh_rate = 0 status_webdav = http://my.server.com/files website_blur_doit = y website_blurkernel = 7 SCRATCH_DISK = . FAST_DISK = . GPU_how_many = 1 GPU_to_use = 0 GPU_cycle = y MotionCor2_alone = y notify_email_admin = y notify_email_admin_adress = name@provider.com notify_last_minutes = 15 notify_delay_minutes = 60 notify_lastnumber = 50 notify_limit_mild = 30 notify_limit_severe = 15 flag_isdark_threshold = 0.1 flag_hasstripe_threshold = 38.0 flag_hastoohighdefocus_threshold = 7.0 flag_hastoolowdefocus_threshold = 0.2 flag_icinesstoohigh_threshold = 1.6 flag_drifttoohigh_threshold = 50.0 flag_astigmatismtoohigh_threshold = 0.03 flag_ctfresolutiontoobad_threshold = 8.0

#############################################

Now the remainder of the original script:

#############################################

++++++++++++++++++++++++++++++++ Initializing script environment ++++++++++++++++++++++++++++++++

Running the following shell: version tcsh 6.18.01 (Astron) 2012-02-14 (x86_64-unknown-linux) options wide,nls,dl,al,kan,rh,color,filec

date = Thu Sep 19 20:01:59 PDT 2019 system = Linux bin_2dx = /opt/focus/kernel/mrc/bin proc_2dx = /opt/focus/scripts/proc ==== dir_eman2 = /usr/local/EMAN2/bin ================================================================================================================================================================= ==== app_python = /usr/bin/python2.7 ================================================================================================================================================================== ==== app_anaconda = /usr/bin/python2.7 ================================================================================================================================================================ ==== dir_frealign = /home/xijiang/Downloads/frealign-2dx-master/bin =================================================================================================================================== ==== dir_relion = /usr/local/relion3 ================================================================================================================================================================== ==== dir_ccp4 = /usr/local/ccp4-7.0 =================================================================================================================================================================== Finding CSH setup script from CCP4 installation.. Sourcing /usr/local/ccp4-7.0/bin/ccp4.setup-csh Finished with /usr/local/ccp4-7.0/bin/ccp4.setup-csh Will use 1 threads for optimal load balancing. [1] 18154

No handlers could be found for logger "mrcz.test_mrcz"

Job 1/1 picking particles from micrographs 1 to 1...

[1] Done /usr/bin/python2.7 /opt/focus/scripts/proc/SPR_ExtractParticles.py ../ SPR/2dx_merge_dirfile-unique.dat SPR/picking/ SPR/stacks/ particles 200 0.0,180.0 0.5568 300 2.6 0.995 89798.9 0.3 y y y y n n 0.4 1.0 0.90 Defocus Particle y n n 1 1 Now organizing and converting partial stack(s) to single .mrcs and .par files...

then: then/endif not found. ::######################PickExportParticleStack finished.#######################
rdrighetto commented 4 years ago

Dear Ethan,

The "No handlers could be found for logger "mrcz.test_mrcz" message is just a warning, not an error. It can be safely ignored. But this "then: then/endif not found." error is indeed something we have to look into. Some questions:

1) Did you update FOCUS in between the different runs of particle picking? If yes, some bug could have been introduced. 2) How many "more images" did you include in your second run with 8 threads? And were you outputting the new picked particles into the same directory as before (the default is SPR/) \, or a different one? If the same, that could lead to some unexpected errors. 3) Are you splitting your particle stacks per-crystal or per-particle? There's an option for that. If you are picking particles from a single 2D crystal image you cannot split per-crystal, because you have only one.

Hope this helps and thanks for using FOCUS!

-- Ricardo Diogo Righetto

Em sex, 20 de set de 2019 às 05:09, EthanJUCB notifications@github.com escreveu:

Hi, I have some problems during particle extraction. In the beginning, I was able to extract particles from several images with the error" No handlers could be found for logger "mrcz.test_mrcz" . I successfully had phase flipped stacks for crystal averages.

Then I tried to include more images using 8 threads. Somehow, only 1 thread was running according to system monitor and scripts stopped after extracting particles from up to 4 images no matter how many image I selected.

After that, I had the worst one, script completely stopped extracting even for a single image. It now shows similar error"No handlers could be found for logger "mrcz.test_mrcz" but with additional message " then: then/endif not found."

I have no idea how to fix it now. Please see attached log. Many thanks.

Best Xi

###########################PickExportParticleStack############################

############################################ Parameters requested by the Script

############################################ thread_number = 8 scriptname = spr_sample_pixel = 0.5568 spr_doresample = n spr_resmin = 100.0 spr_resmax = 10.0 spr_phaseshift = 0.0,180.0 spr_boxsize = 200 spr_sigcc = 0.3 spr_invertcontrast = y spr_normalizebox = y spr_calculatedefocustilted = y spr_savephaseflipped = y spr_savectfmultiplied = n spr_savewienerfiltered = n spr_wienerconstant = 0.4 spr_sigmanorm = 1.0 spr_radnorm = 0.90 spr_whichtiltgeometry = Defocus spr_whichctf = Particle spr_whichhalfset = particle-based spr_savepickfig = y cs = 2.6 kv = 300 sample_pixel = 0.5568 phacon = 0.995 magnification = 89798.9 use_masked_image = n spr_ignore_lat2 = y spr_ignore_c1 = y spr_createstarfile = y spr_dir = SPR spr_imgs_dir = spr_mergefile = spr_stacks_dir = spr_picking_dir = spr_stack_rootname =

############################################ Parameters from the Preferences settings

############################################ app_python = /usr/bin/python2.7 app_cadaver = /usr/bin/cadaver app_anaconda = /usr/bin/python2.7 dir_ccp4 = /usr/local/ccp4-7.0 app_gawk = /usr/bin/gawk cuda_library = /usr/local/cuda/lib64 app_unblur = /usr/local/cina/unblur/unblur_1.0.2/bin/unblur_openmp_7_17_15.exe app_motioncor2 = /home/xijiang/Downloads/Motioncor2/MotionCor2-01-30-2017 app_gctf = /home/xijiang/Downloads/Gctf_v1.06/bin/Gctf-v1.06_sm_30_cu8.0_x86_64 app_ctffind = /home/xijiang/Downloads/CTFFIND4/bin/ctffind app_gautomatch = /home/xijiang/Downloads/Gautomatch_v0.53/bin/Gautomatch-v0.53_sm_20_cu8.0_x86_64 dir_eman2 = /usr/local/EMAN2/bin dir_imod = /usr/local/IMOD dir_frealign = /home/xijiang/Downloads/frealign-2dx-master/bin dir_relion = /usr/local/relion3 Microscope_Name = CryoEM Default_CS = 2.6 Default_KV = 300.0 Default_phase_shift_doit = n status_folder_update = n status_folder = none status_microscope = Titan status_refresh_rate = 0 status_webdav = http://my.server.com/files website_blur_doit = y website_blurkernel = 7 SCRATCH_DISK = . FAST_DISK = . GPU_how_many = 1 GPU_to_use = 0 GPU_cycle = y MotionCor2_alone = y notify_email_admin = y notify_email_admin_adress = name@provider.com notify_last_minutes = 15 notify_delay_minutes = 60 notify_lastnumber = 50 notify_limit_mild = 30 notify_limit_severe = 15 flag_isdark_threshold = 0.1 flag_hasstripe_threshold = 38.0 flag_hastoohighdefocus_threshold = 7.0 flag_hastoolowdefocus_threshold = 0.2 flag_icinesstoohigh_threshold = 1.6 flag_drifttoohigh_threshold = 50.0 flag_astigmatismtoohigh_threshold = 0.03 flag_ctfresolutiontoobad_threshold = 8.0

############################################# Now the remainder of the original script:

#############################################

++++++++++++++++++++++++++++++++ Initializing script environment ++++++++++++++++++++++++++++++++

Running the following shell: version tcsh 6.18.01 (Astron) 2012-02-14 (x86_64-unknown-linux) options wide,nls,dl,al,kan,rh,color,filec

date = Thu Sep 19 20:01:59 PDT 2019 system = Linux bin_2dx = /opt/focus/kernel/mrc/bin proc_2dx = /opt/focus/scripts/proc ==== dir_eman2 = /usr/local/EMAN2/bin

==== app_python = /usr/bin/python2.7

==== app_anaconda = /usr/bin/python2.7

==== dir_frealign = /home/xijiang/Downloads/frealign-2dx-master/bin

==== dir_relion = /usr/local/relion3

==== dir_ccp4 = /usr/local/ccp4-7.0

Finding CSH setup script from CCP4 installation.. Sourcing /usr/local/ccp4-7.0/bin/ccp4.setup-csh Finished with /usr/local/ccp4-7.0/bin/ccp4.setup-csh Will use 1 threads for optimal load balancing. [1] 18154 No handlers could be found for logger "mrcz.test_mrcz"

Job 1/1 picking particles from micrographs 1 to 1...

[1] Done /usr/bin/python2.7 /opt/focus/scripts/proc/SPR_ExtractParticles.py ../ SPR/2dx_merge_dirfile-unique.dat SPR/picking/ SPR/stacks/ particles 200 0.0,180.0 0.5568 300 2.6 0.995 89798.9 0.3 y y y y n n 0.4 1.0 0.90 Defocus Particle y n n 1 1 Now organizing and converting partial stack(s) to single .mrcs and .par files... then: then/endif not found. ::######################PickExportParticleStack finished.#######################

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/C-CINA/focus/issues/242?email_source=notifications&email_token=AB2W673EQMA7CPR7CVVRJXLQKQ5G5A5CNFSM4IYSFZDKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HMSGHCQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2W674SX2BBXPANDJKUTXDQKQ5G5ANCNFSM4IYSFZDA .

EthanJUCB commented 4 years ago

Hi Ricardo,

Many thanks.

I didn’t update FOCUS during the different runs. Then I uninstalled and installed again, it didn’t solve the problem.

I included 328 images but it stopped after 4 images. Yes, I used the default SPR folder. It looked like the current Par file will be overwrote or renamed with “old" if I started a new extraction. I tried to use different folder, It didn’t work either.

I split particles per crystal since I had two images in the beginning and it worked well.

Do you have any suggestions that how to solve this strange issue? Is it a good idea to install a fresh centos7 with only focus and its dependencies.

PS. I read the Nature Comm. paper about 2DX SPA processing. I wonder the 2D averaged images of crystals were obtained by averaging all boxes in a stack of one micrograph or they were obtained by using 2D classification method such like maximum likelihood in Relion.

Best Xi

On Sep 20, 2019, at 1:36 AM, Ricardo Righetto notifications@github.com wrote:

Dear Ethan,

The "No handlers could be found for logger "mrcz.test_mrcz" message is just a warning, not an error. It can be safely ignored. But this "then: then/endif not found." error is indeed something we have to look into. Some questions:

1) Did you update FOCUS in between the different runs of particle picking? If yes, some bug could have been introduced. 2) How many "more images" did you include in your second run with 8 threads? And were you outputting the new picked particles into the same directory as before (the default is SPR/) \, or a different one? If the same, that could lead to some unexpected errors. 3) Are you splitting your particle stacks per-crystal or per-particle? There's an option for that. If you are picking particles from a single 2D crystal image you cannot split per-crystal, because you have only one.

Hope this helps and thanks for using FOCUS!

-- Ricardo Diogo Righetto

Em sex, 20 de set de 2019 às 05:09, EthanJUCB notifications@github.com escreveu:

Hi, I have some problems during particle extraction. In the beginning, I was able to extract particles from several images with the error" No handlers could be found for logger "mrcz.test_mrcz" . I successfully had phase flipped stacks for crystal averages.

Then I tried to include more images using 8 threads. Somehow, only 1 thread was running according to system monitor and scripts stopped after extracting particles from up to 4 images no matter how many image I selected.

After that, I had the worst one, script completely stopped extracting even for a single image. It now shows similar error"No handlers could be found for logger "mrcz.test_mrcz" but with additional message " then: then/endif not found."

I have no idea how to fix it now. Please see attached log. Many thanks.

Best Xi

###########################PickExportParticleStack############################

############################################ Parameters requested by the Script

############################################ thread_number = 8 scriptname = spr_sample_pixel = 0.5568 spr_doresample = n spr_resmin = 100.0 spr_resmax = 10.0 spr_phaseshift = 0.0,180.0 spr_boxsize = 200 spr_sigcc = 0.3 spr_invertcontrast = y spr_normalizebox = y spr_calculatedefocustilted = y spr_savephaseflipped = y spr_savectfmultiplied = n spr_savewienerfiltered = n spr_wienerconstant = 0.4 spr_sigmanorm = 1.0 spr_radnorm = 0.90 spr_whichtiltgeometry = Defocus spr_whichctf = Particle spr_whichhalfset = particle-based spr_savepickfig = y cs = 2.6 kv = 300 sample_pixel = 0.5568 phacon = 0.995 magnification = 89798.9 use_masked_image = n spr_ignore_lat2 = y spr_ignore_c1 = y spr_createstarfile = y spr_dir = SPR spr_imgs_dir = spr_mergefile = spr_stacks_dir = spr_picking_dir = spr_stack_rootname =

############################################ Parameters from the Preferences settings

############################################ app_python = /usr/bin/python2.7 app_cadaver = /usr/bin/cadaver app_anaconda = /usr/bin/python2.7 dir_ccp4 = /usr/local/ccp4-7.0 app_gawk = /usr/bin/gawk cuda_library = /usr/local/cuda/lib64 app_unblur = /usr/local/cina/unblur/unblur_1.0.2/bin/unblur_openmp_7_17_15.exe app_motioncor2 = /home/xijiang/Downloads/Motioncor2/MotionCor2-01-30-2017 app_gctf = /home/xijiang/Downloads/Gctf_v1.06/bin/Gctf-v1.06_sm_30_cu8.0_x86_64 app_ctffind = /home/xijiang/Downloads/CTFFIND4/bin/ctffind app_gautomatch = /home/xijiang/Downloads/Gautomatch_v0.53/bin/Gautomatch-v0.53_sm_20_cu8.0_x86_64 dir_eman2 = /usr/local/EMAN2/bin dir_imod = /usr/local/IMOD dir_frealign = /home/xijiang/Downloads/frealign-2dx-master/bin dir_relion = /usr/local/relion3 Microscope_Name = CryoEM Default_CS = 2.6 Default_KV = 300.0 Default_phase_shift_doit = n status_folder_update = n status_folder = none status_microscope = Titan status_refresh_rate = 0 status_webdav = http://my.server.com/files website_blur_doit = y website_blurkernel = 7 SCRATCH_DISK = . FAST_DISK = . GPU_how_many = 1 GPU_to_use = 0 GPU_cycle = y MotionCor2_alone = y notify_email_admin = y notify_email_admin_adress = name@provider.com notify_last_minutes = 15 notify_delay_minutes = 60 notify_lastnumber = 50 notify_limit_mild = 30 notify_limit_severe = 15 flag_isdark_threshold = 0.1 flag_hasstripe_threshold = 38.0 flag_hastoohighdefocus_threshold = 7.0 flag_hastoolowdefocus_threshold = 0.2 flag_icinesstoohigh_threshold = 1.6 flag_drifttoohigh_threshold = 50.0 flag_astigmatismtoohigh_threshold = 0.03 flag_ctfresolutiontoobad_threshold = 8.0

############################################# Now the remainder of the original script:

#############################################

++++++++++++++++++++++++++++++++ Initializing script environment ++++++++++++++++++++++++++++++++

Running the following shell: version tcsh 6.18.01 (Astron) 2012-02-14 (x86_64-unknown-linux) options wide,nls,dl,al,kan,rh,color,filec

date = Thu Sep 19 20:01:59 PDT 2019 system = Linux bin_2dx = /opt/focus/kernel/mrc/bin proc_2dx = /opt/focus/scripts/proc ==== dir_eman2 = /usr/local/EMAN2/bin

==== app_python = /usr/bin/python2.7

==== app_anaconda = /usr/bin/python2.7

==== dir_frealign = /home/xijiang/Downloads/frealign-2dx-master/bin

==== dir_relion = /usr/local/relion3

==== dir_ccp4 = /usr/local/ccp4-7.0

Finding CSH setup script from CCP4 installation.. Sourcing /usr/local/ccp4-7.0/bin/ccp4.setup-csh Finished with /usr/local/ccp4-7.0/bin/ccp4.setup-csh Will use 1 threads for optimal load balancing. [1] 18154 No handlers could be found for logger "mrcz.test_mrcz"

Job 1/1 picking particles from micrographs 1 to 1...

[1] Done /usr/bin/python2.7 /opt/focus/scripts/proc/SPR_ExtractParticles.py ../ SPR/2dx_merge_dirfile-unique.dat SPR/picking/ SPR/stacks/ particles 200 0.0,180.0 0.5568 300 2.6 0.995 89798.9 0.3 y y y y n n 0.4 1.0 0.90 Defocus Particle y n n 1 1 Now organizing and converting partial stack(s) to single .mrcs and .par files... then: then/endif not found. ::######################PickExportParticleStack finished.#######################

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/C-CINA/focus/issues/242?email_source=notifications&email_token=AB2W673EQMA7CPR7CVVRJXLQKQ5G5A5CNFSM4IYSFZDKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HMSGHCQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2W674SX2BBXPANDJKUTXDQKQ5G5ANCNFSM4IYSFZDA .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/C-CINA/focus/issues/242?email_source=notifications&email_token=AEVLNHGCK5WANBW6FPSBOB3QKSDQTA5CNFSM4IYSFZDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7GADCA#issuecomment-533463432, or mute the thread https://github.com/notifications/unsubscribe-auth/AEVLNHBEFGFXBUSFI7EL7UTQKSDQTANCNFSM4IYSFZDA.

EthanJUCB commented 4 years ago

Hi Richardo,

I tested extraction again on a fresh installed computer. Extraction worked for 1 image perfectly, once I included more images it stopped working and show the same error message as I posed.

I looked into and run the extraction script in terminal, it seemed like python was not able to import ioMrc, focus_utility and focus_ctf properly. I have no idea whey it didn’t import them properly as long as it worked in the beginning for 1 but not for many images later.

Best Xi

On Sep 20, 2019, at 1:36 AM, Ricardo Righetto notifications@github.com wrote:

Dear Ethan,

The "No handlers could be found for logger "mrcz.test_mrcz" message is just a warning, not an error. It can be safely ignored. But this "then: then/endif not found." error is indeed something we have to look into. Some questions:

1) Did you update FOCUS in between the different runs of particle picking? If yes, some bug could have been introduced. 2) How many "more images" did you include in your second run with 8 threads? And were you outputting the new picked particles into the same directory as before (the default is SPR/) \, or a different one? If the same, that could lead to some unexpected errors. 3) Are you splitting your particle stacks per-crystal or per-particle? There's an option for that. If you are picking particles from a single 2D crystal image you cannot split per-crystal, because you have only one.

Hope this helps and thanks for using FOCUS!

-- Ricardo Diogo Righetto

Em sex, 20 de set de 2019 às 05:09, EthanJUCB notifications@github.com escreveu:

Hi, I have some problems during particle extraction. In the beginning, I was able to extract particles from several images with the error" No handlers could be found for logger "mrcz.test_mrcz" . I successfully had phase flipped stacks for crystal averages.

Then I tried to include more images using 8 threads. Somehow, only 1 thread was running according to system monitor and scripts stopped after extracting particles from up to 4 images no matter how many image I selected.

After that, I had the worst one, script completely stopped extracting even for a single image. It now shows similar error"No handlers could be found for logger "mrcz.test_mrcz" but with additional message " then: then/endif not found."

I have no idea how to fix it now. Please see attached log. Many thanks.

Best Xi

###########################PickExportParticleStack############################

############################################ Parameters requested by the Script

############################################ thread_number = 8 scriptname = spr_sample_pixel = 0.5568 spr_doresample = n spr_resmin = 100.0 spr_resmax = 10.0 spr_phaseshift = 0.0,180.0 spr_boxsize = 200 spr_sigcc = 0.3 spr_invertcontrast = y spr_normalizebox = y spr_calculatedefocustilted = y spr_savephaseflipped = y spr_savectfmultiplied = n spr_savewienerfiltered = n spr_wienerconstant = 0.4 spr_sigmanorm = 1.0 spr_radnorm = 0.90 spr_whichtiltgeometry = Defocus spr_whichctf = Particle spr_whichhalfset = particle-based spr_savepickfig = y cs = 2.6 kv = 300 sample_pixel = 0.5568 phacon = 0.995 magnification = 89798.9 use_masked_image = n spr_ignore_lat2 = y spr_ignore_c1 = y spr_createstarfile = y spr_dir = SPR spr_imgs_dir = spr_mergefile = spr_stacks_dir = spr_picking_dir = spr_stack_rootname =

############################################ Parameters from the Preferences settings

############################################ app_python = /usr/bin/python2.7 app_cadaver = /usr/bin/cadaver app_anaconda = /usr/bin/python2.7 dir_ccp4 = /usr/local/ccp4-7.0 app_gawk = /usr/bin/gawk cuda_library = /usr/local/cuda/lib64 app_unblur = /usr/local/cina/unblur/unblur_1.0.2/bin/unblur_openmp_7_17_15.exe app_motioncor2 = /home/xijiang/Downloads/Motioncor2/MotionCor2-01-30-2017 app_gctf = /home/xijiang/Downloads/Gctf_v1.06/bin/Gctf-v1.06_sm_30_cu8.0_x86_64 app_ctffind = /home/xijiang/Downloads/CTFFIND4/bin/ctffind app_gautomatch = /home/xijiang/Downloads/Gautomatch_v0.53/bin/Gautomatch-v0.53_sm_20_cu8.0_x86_64 dir_eman2 = /usr/local/EMAN2/bin dir_imod = /usr/local/IMOD dir_frealign = /home/xijiang/Downloads/frealign-2dx-master/bin dir_relion = /usr/local/relion3 Microscope_Name = CryoEM Default_CS = 2.6 Default_KV = 300.0 Default_phase_shift_doit = n status_folder_update = n status_folder = none status_microscope = Titan status_refresh_rate = 0 status_webdav = http://my.server.com/files website_blur_doit = y website_blurkernel = 7 SCRATCH_DISK = . FAST_DISK = . GPU_how_many = 1 GPU_to_use = 0 GPU_cycle = y MotionCor2_alone = y notify_email_admin = y notify_email_admin_adress = name@provider.com notify_last_minutes = 15 notify_delay_minutes = 60 notify_lastnumber = 50 notify_limit_mild = 30 notify_limit_severe = 15 flag_isdark_threshold = 0.1 flag_hasstripe_threshold = 38.0 flag_hastoohighdefocus_threshold = 7.0 flag_hastoolowdefocus_threshold = 0.2 flag_icinesstoohigh_threshold = 1.6 flag_drifttoohigh_threshold = 50.0 flag_astigmatismtoohigh_threshold = 0.03 flag_ctfresolutiontoobad_threshold = 8.0

############################################# Now the remainder of the original script:

#############################################

++++++++++++++++++++++++++++++++ Initializing script environment ++++++++++++++++++++++++++++++++

Running the following shell: version tcsh 6.18.01 (Astron) 2012-02-14 (x86_64-unknown-linux) options wide,nls,dl,al,kan,rh,color,filec

date = Thu Sep 19 20:01:59 PDT 2019 system = Linux bin_2dx = /opt/focus/kernel/mrc/bin proc_2dx = /opt/focus/scripts/proc ==== dir_eman2 = /usr/local/EMAN2/bin

==== app_python = /usr/bin/python2.7

==== app_anaconda = /usr/bin/python2.7

==== dir_frealign = /home/xijiang/Downloads/frealign-2dx-master/bin

==== dir_relion = /usr/local/relion3

==== dir_ccp4 = /usr/local/ccp4-7.0

Finding CSH setup script from CCP4 installation.. Sourcing /usr/local/ccp4-7.0/bin/ccp4.setup-csh Finished with /usr/local/ccp4-7.0/bin/ccp4.setup-csh Will use 1 threads for optimal load balancing. [1] 18154 No handlers could be found for logger "mrcz.test_mrcz"

Job 1/1 picking particles from micrographs 1 to 1...

[1] Done /usr/bin/python2.7 /opt/focus/scripts/proc/SPR_ExtractParticles.py ../ SPR/2dx_merge_dirfile-unique.dat SPR/picking/ SPR/stacks/ particles 200 0.0,180.0 0.5568 300 2.6 0.995 89798.9 0.3 y y y y n n 0.4 1.0 0.90 Defocus Particle y n n 1 1 Now organizing and converting partial stack(s) to single .mrcs and .par files... then: then/endif not found. ::######################PickExportParticleStack finished.#######################

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/C-CINA/focus/issues/242?email_source=notifications&email_token=AB2W673EQMA7CPR7CVVRJXLQKQ5G5A5CNFSM4IYSFZDKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HMSGHCQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2W674SX2BBXPANDJKUTXDQKQ5G5ANCNFSM4IYSFZDA .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/C-CINA/focus/issues/242?email_source=notifications&email_token=AEVLNHGCK5WANBW6FPSBOB3QKSDQTA5CNFSM4IYSFZDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7GADCA#issuecomment-533463432, or mute the thread https://github.com/notifications/unsubscribe-auth/AEVLNHBEFGFXBUSFI7EL7UTQKSDQTANCNFSM4IYSFZDA.

rdrighetto commented 4 years ago

OK, so it seems there is a bug in the script if you try to run it more than once. I will look into that and get back to you soon!

Regarding your question about the 2D crystal averages, they are simply the average of all boxes extracted from a given crystal. No 2D classification or alignment were performed. This is the next script in the FOCUS pipeline after the "Pick & Extract Particles".

Best wishes,

-- Ricardo Diogo Righetto

Em dom, 22 de set de 2019 às 01:15, EthanJUCB notifications@github.com escreveu:

Hi Richardo,

I tested extraction again on a fresh installed computer. Extraction worked for 1 image perfectly, once I included more images it stopped working and show the same error message as I posed.

I looked into and run the extraction script in terminal, it seemed like python was not able to import ioMrc, focus_utility and focus_ctf properly. I have no idea whey it didn’t import them properly as long as it worked in the beginning for 1 but not for many images later.

Best Xi

On Sep 20, 2019, at 1:36 AM, Ricardo Righetto notifications@github.com wrote:

Dear Ethan,

The "No handlers could be found for logger "mrcz.test_mrcz" message is just a warning, not an error. It can be safely ignored. But this "then: then/endif not found." error is indeed something we have to look into. Some questions:

1) Did you update FOCUS in between the different runs of particle picking? If yes, some bug could have been introduced. 2) How many "more images" did you include in your second run with 8 threads? And were you outputting the new picked particles into the same directory as before (the default is SPR/) \, or a different one? If the same, that could lead to some unexpected errors. 3) Are you splitting your particle stacks per-crystal or per-particle? There's an option for that. If you are picking particles from a single 2D crystal image you cannot split per-crystal, because you have only one.

Hope this helps and thanks for using FOCUS!

-- Ricardo Diogo Righetto

Em sex, 20 de set de 2019 às 05:09, EthanJUCB notifications@github.com escreveu:

Hi, I have some problems during particle extraction. In the beginning, I was able to extract particles from several images with the error" No handlers could be found for logger "mrcz.test_mrcz" . I successfully had phase flipped stacks for crystal averages.

Then I tried to include more images using 8 threads. Somehow, only 1 thread was running according to system monitor and scripts stopped after extracting particles from up to 4 images no matter how many image I selected.

After that, I had the worst one, script completely stopped extracting even for a single image. It now shows similar error"No handlers could be found for logger "mrcz.test_mrcz" but with additional message " then: then/endif not found."

I have no idea how to fix it now. Please see attached log. Many thanks.

Best Xi

###########################PickExportParticleStack############################

############################################ Parameters requested by the Script

############################################ thread_number = 8 scriptname = spr_sample_pixel = 0.5568 spr_doresample = n spr_resmin = 100.0 spr_resmax = 10.0 spr_phaseshift = 0.0,180.0 spr_boxsize = 200 spr_sigcc = 0.3 spr_invertcontrast = y spr_normalizebox = y spr_calculatedefocustilted = y spr_savephaseflipped = y spr_savectfmultiplied = n spr_savewienerfiltered = n spr_wienerconstant = 0.4 spr_sigmanorm = 1.0 spr_radnorm = 0.90 spr_whichtiltgeometry = Defocus spr_whichctf = Particle spr_whichhalfset = particle-based spr_savepickfig = y cs = 2.6 kv = 300 sample_pixel = 0.5568 phacon = 0.995 magnification = 89798.9 use_masked_image = n spr_ignore_lat2 = y spr_ignore_c1 = y spr_createstarfile = y spr_dir = SPR spr_imgs_dir = spr_mergefile = spr_stacks_dir = spr_picking_dir = spr_stack_rootname =

############################################ Parameters from the Preferences settings

############################################ app_python = /usr/bin/python2.7 app_cadaver = /usr/bin/cadaver app_anaconda = /usr/bin/python2.7 dir_ccp4 = /usr/local/ccp4-7.0 app_gawk = /usr/bin/gawk cuda_library = /usr/local/cuda/lib64 app_unblur = /usr/local/cina/unblur/unblur_1.0.2/bin/unblur_openmp_7_17_15.exe app_motioncor2 = /home/xijiang/Downloads/Motioncor2/MotionCor2-01-30-2017 app_gctf = /home/xijiang/Downloads/Gctf_v1.06/bin/Gctf-v1.06_sm_30_cu8.0_x86_64 app_ctffind = /home/xijiang/Downloads/CTFFIND4/bin/ctffind app_gautomatch =

/home/xijiang/Downloads/Gautomatch_v0.53/bin/Gautomatch-v0.53_sm_20_cu8.0_x86_64 dir_eman2 = /usr/local/EMAN2/bin dir_imod = /usr/local/IMOD dir_frealign = /home/xijiang/Downloads/frealign-2dx-master/bin dir_relion = /usr/local/relion3 Microscope_Name = CryoEM Default_CS = 2.6 Default_KV = 300.0 Default_phase_shift_doit = n status_folder_update = n status_folder = none status_microscope = Titan status_refresh_rate = 0 status_webdav = http://my.server.com/files website_blur_doit = y website_blurkernel = 7 SCRATCH_DISK = . FAST_DISK = . GPU_how_many = 1 GPU_to_use = 0 GPU_cycle = y MotionCor2_alone = y notify_email_admin = y notify_email_admin_adress = name@provider.com notify_last_minutes = 15 notify_delay_minutes = 60 notify_lastnumber = 50 notify_limit_mild = 30 notify_limit_severe = 15 flag_isdark_threshold = 0.1 flag_hasstripe_threshold = 38.0 flag_hastoohighdefocus_threshold = 7.0 flag_hastoolowdefocus_threshold = 0.2 flag_icinesstoohigh_threshold = 1.6 flag_drifttoohigh_threshold = 50.0 flag_astigmatismtoohigh_threshold = 0.03 flag_ctfresolutiontoobad_threshold = 8.0

############################################# Now the remainder of the original script:

#############################################

++++++++++++++++++++++++++++++++ Initializing script environment ++++++++++++++++++++++++++++++++

Running the following shell: version tcsh 6.18.01 (Astron) 2012-02-14 (x86_64-unknown-linux) options wide,nls,dl,al,kan,rh,color,filec

date = Thu Sep 19 20:01:59 PDT 2019 system = Linux bin_2dx = /opt/focus/kernel/mrc/bin proc_2dx = /opt/focus/scripts/proc ==== dir_eman2 = /usr/local/EMAN2/bin

================================================================================================================================================================= ==== app_python = /usr/bin/python2.7

================================================================================================================================================================== ==== app_anaconda = /usr/bin/python2.7

================================================================================================================================================================ ==== dir_frealign = /home/xijiang/Downloads/frealign-2dx-master/bin

=================================================================================================================================== ==== dir_relion = /usr/local/relion3

================================================================================================================================================================== ==== dir_ccp4 = /usr/local/ccp4-7.0

=================================================================================================================================================================== Finding CSH setup script from CCP4 installation.. Sourcing /usr/local/ccp4-7.0/bin/ccp4.setup-csh Finished with /usr/local/ccp4-7.0/bin/ccp4.setup-csh Will use 1 threads for optimal load balancing. [1] 18154 No handlers could be found for logger "mrcz.test_mrcz"

Job 1/1 picking particles from micrographs 1 to 1...

[1] Done /usr/bin/python2.7 /opt/focus/scripts/proc/SPR_ExtractParticles.py ../ SPR/2dx_merge_dirfile-unique.dat SPR/picking/ SPR/stacks/ particles 200 0.0,180.0 0.5568 300 2.6 0.995 89798.9 0.3 y y y y n n 0.4 1.0 0.90 Defocus Particle y n n 1 1 Now organizing and converting partial stack(s) to single .mrcs and .par files... then: then/endif not found. ::######################PickExportParticleStack finished.#######################

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < https://github.com/C-CINA/focus/issues/242?email_source=notifications&email_token=AB2W673EQMA7CPR7CVVRJXLQKQ5G5A5CNFSM4IYSFZDKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HMSGHCQ , or mute the thread < https://github.com/notifications/unsubscribe-auth/AB2W674SX2BBXPANDJKUTXDQKQ5G5ANCNFSM4IYSFZDA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/C-CINA/focus/issues/242?email_source=notifications&email_token=AEVLNHGCK5WANBW6FPSBOB3QKSDQTA5CNFSM4IYSFZDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7GADCA#issuecomment-533463432>, or mute the thread < https://github.com/notifications/unsubscribe-auth/AEVLNHBEFGFXBUSFI7EL7UTQKSDQTANCNFSM4IYSFZDA .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/C-CINA/focus/issues/242?email_source=notifications&email_token=AB2W67YFHOI6ATSPZHB6S4LQK2TJVA5CNFSM4IYSFZDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7I3G6Q#issuecomment-533836666, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2W674WPT7CS2IMHJPPKXLQK2TJVANCNFSM4IYSFZDA .

rdrighetto commented 4 years ago

Dear Xi,

I have found a bug in the splitting of particles into per-crystal half-sets, that yielded the following error message:

sed: invalid usage of line address 0

Have you ever encountered that? This bugfix is now available in the latest version of the source code and should be in the snapshot binaries from our website within the hour (https://focus.c-cina.unibas.ch/download.php). Please make sure you are using this latest version of the source code, either from Github or the daily binary snapshots.

Other than that, I was not able to reproduce the issue. Extracting particles gives no error no matter how many images are included or how many times I run it, even on the same target directory for storing the stacks (e.g. SPR/).

EthanJUCB commented 4 years ago

Hi Ricardo,

No, I never had this issue.

Seems like the problem is related to the libraries in my computer rather than bugs in 2dx.

I wonder which OS you use for testing and developing.

Best Xi

On Oct 8, 2019, at 9:22 AM, Ricardo Righetto notifications@github.com wrote:

Dear Xi,

I have found a bug in the splitting of particles into per-crystal half-sets, that yielded the following error message:

sed: invalid usage of line address 0

Have you ever encountered that? This bugfix is now available in the latest version of the source code and should be in the snapshot binaries from our website within the hour (https://focus.c-cina.unibas.ch/download.php https://focus.c-cina.unibas.ch/download.php). Please make sure you are using this latest version of the source code, either from Github or the daily binary snapshots.

Other than that, I was not able to reproduce the issue. Extracting particles gives no error no matter how many images are included or how many times I run it, even on the same target directory for storing the stacks (e.g. SPR/).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/C-CINA/focus/issues/242?email_source=notifications&email_token=AEVLNHF22KXYJHWHLH4WA7LQNSXUPA5CNFSM4IYSFZDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAUYNTQ#issuecomment-539592398, or mute the thread https://github.com/notifications/unsubscribe-auth/AEVLNHBZDDUPAAWPKCCI5M3QNSXUPANCNFSM4IYSFZDA.

rdrighetto commented 4 years ago

Dear Xi,

I use Ubuntu 18.04.3 LTS for running and developing FOCUS. But I used CentOS 7 until recently so it should work.

Best wishes, Ricardo

acdrennan commented 4 years ago

I am having a related issue... I've processed drift-corrected images as 2d crystals and now I would like to pick particles from one of the images. I selected the image in the library and then attempted to run "Pick & export particles", and it appears to work but I don't get a window at the right of the GUI like I do with the 2d crystal scripts, so I think something is wrong... I also get some errors (see output below). I would appreciate any help/guidance- Best, Amanda

###########################PickExportParticleStack############################

############################################

Parameters requested by the Script

############################################ thread_number = 12 scriptname = spr_sample_pixel = 1.37 spr_doresample = y spr_resmin = 100.0 spr_resmax = 10.0 spr_phaseshift = 0.0,180.0 spr_boxsize = 500 spr_sigcc = 0.0 spr_invertcontrast = y spr_normalizebox = y spr_calculatedefocustilted = y spr_savephaseflipped = n spr_savectfmultiplied = y spr_savewienerfiltered = n spr_wienerconstant = 0.4 spr_sigmanorm = 1.0 spr_radnorm = -1 spr_whichtiltgeometry = Merge spr_whichctf = Particle spr_whichhalfset = particle-based spr_savepickfig = n cs = 2.7 kv = 300 sample_pixel = 1.0 phacon = 0.995 magnification = 38540 use_masked_image = y spr_ignore_lat2 = y spr_ignore_c1 = y spr_createstarfile = y spr_dir = SPR spr_imgs_dir = spr_mergefile = spr_stacks_dir = spr_picking_dir = spr_stack_rootname =

############################################

Parameters from the Preferences settings

############################################ app_python = /home/drennan@ad.wisc.edu/.conda/envs/focusenv/bin/python app_cadaver = /usr/bin/cadaver app_anaconda = /home/drennan@ad.wisc.edu/.conda/envs/focusenv/bin/python dir_ccp4 = /opt/ccp4/ccp4-7.0 app_gawk = /usr/bin/gawk cuda_library = /usr/local/cuda/lib64 app_unblur = /opt/unblur_1.0.2/bin/unblur_openmp_7_17_15.exe app_motioncor2 = /opt/MotionCor2/MotionCor2_Cuda9.1_v1.0.5 app_gctf = /opt/gctf/Gctf-v1.18_sm_30_cu8.0_x86_64 app_ctffind = /opt/ctffind-4.1.10/ctffind app_gautomatch = /programs/x86_64-linux/gautomatch/gautomatch.rc dir_eman2 = /opt/EMAN2 dir_imod = /opt/imod/imod_4.9.8 dir_frealign = /programs/x86_64-linux/frealign dir_relion = /opt/relion-2.1.stable Default_CS = 2.7 Default_KV = 300 Default_phase_shift_doit = n status_folder_update = n status_folder = none status_microscope = Titan status_refresh_rate = 0 status_webdav = http://my.server.com/files website_blur_doit = y website_blurkernel = 7 SCRATCH_DISK = . FAST_DISK = . GPU_how_many = 4 GPU_to_use = 0 1 2 3 GPU_cycle = n MotionCor2_alone = y

#############################################

Now the remainder of the original script:

#############################################

++++++++++++++++++++++++++++++++ Initializing script environment ++++++++++++++++++++++++++++++++ date = Tue Jan 7 12:22:35 CST 2020 system = Linux bin_2dx = /opt/focus/kernel/mrc/bin proc_2dx = /opt/focus/scripts/proc ==== dir_eman2 = /opt/EMAN2 =========================================================================================================================================================================== ==== app_python = /home/drennan@ad.wisc.edu/.conda/envs/focusenv/bin/python =========================================================================================================================== ==== app_anaconda = /home/drennan@ad.wisc.edu/.conda/envs/focusenv/bin/python ========================================================================================================================= ==== dir_frealign = /programs/x86_64-linux/frealign =================================================================================================================================================== ==== dir_relion = /opt/relion-2.1.stable ============================================================================================================================================================== ==== dir_ccp4 = /opt/ccp4/ccp4-7.0 ==================================================================================================================================================================== Finding CSH setup script from CCP4 installation.. Sourcing /opt/ccp4/ccp4-7.0/bin/ccp4.setup-csh Finished with /opt/ccp4/ccp4-7.0/bin/ccp4.setup-csh Will use 1 threads for optimal load balancing. [1] 8341

Job 1/1 picking particles from micrographs 1 to 1...

Now boxing unit cells of micrograph 1/1.

../auto/0000000019/mimage_2dx.mrc [1] Exit 1 /home/drennan@ad.wisc.edu/.conda/envs/focusenv/bin/python /opt/focus/scripts/proc/SPR_ExtractParticles.py ../ SPR/2dx_merge_dirfile-unique.dat SPR/picking/ SPR/stacks/ particles 500 0.0,180.0 1.37 300 2.7 0.995 38540 0.0 y y y n y n 0.4 1.0 -1 Merge Particle n y y 1 1 Now organizing and converting partial stack(s) to single .mrcs and .par files... ######################PickExportParticleStack finished.####################### No handlers could be found for logger "mrcz.test_mrcz" Traceback (most recent call last): File "/opt/focus/scripts/proc/SPR_ExtractParticles.py", line 1187, in main() File "/opt/focus/scripts/proc/SPR_ExtractParticles.py", line 308, in main img = -1.0 * img TypeError: can't multiply sequence by non-int of type 'float' then: then/endif not found.

rdrighetto commented 4 years ago

Dear Amanda,

You are getting two errors:

Thanks for using FOCUS, Ricardo

acdrennan commented 4 years ago

Hi Ricardo, FOCUS is set up to see python 2.7.17. I updated the source code and then ran "Pick and export particles" with a single image again, but I got the same errors. Below is the logfile you requested. Thank you for your help! I really appreciate it. Best, Amanda

(focusenv) [drennan@ad.wisc.edu@BIOCWK-01037L data]$ cat /mnt/data/top100tilt0/merge/LOGS/PickExportParticleStack.log ::###########################PickExportParticleStack############################

############################################

Parameters requested by the Script

############################################ thread_number = 12 scriptname = spr_sample_pixel = 1.37 spr_doresample = y spr_resmin = 100.0 spr_resmax = 10.0 spr_phaseshift = 0.0,180.0 spr_boxsize = 500 spr_sigcc = 0.0 spr_invertcontrast = y spr_normalizebox = y spr_calculatedefocustilted = y spr_savephaseflipped = n spr_savectfmultiplied = y spr_savewienerfiltered = n spr_wienerconstant = 0.4 spr_sigmanorm = 1.0 spr_radnorm = -1 spr_whichtiltgeometry = Merge spr_whichctf = Particle spr_whichhalfset = particle-based spr_savepickfig = n cs = 2.7 kv = 300 sample_pixel = 1.0 phacon = 0.995 magnification = 38540 use_masked_image = y spr_ignore_lat2 = y spr_ignore_c1 = y spr_createstarfile = y spr_dir = SPR spr_imgs_dir = spr_mergefile = spr_stacks_dir = spr_picking_dir = spr_stack_rootname =

############################################

Parameters from the Preferences settings

############################################ app_python = /home/drennan@ad.wisc.edu/.conda/envs/focusenv/bin/python app_cadaver = /usr/bin/cadaver app_anaconda = /home/drennan@ad.wisc.edu/.conda/envs/focusenv/bin/python dir_ccp4 = /opt/ccp4/ccp4-7.0 app_gawk = /usr/bin/gawk cuda_library = /usr/local/cuda/lib64 app_unblur = /opt/unblur_1.0.2/bin/unblur_openmp_7_17_15.exe app_motioncor2 = /opt/MotionCor2/MotionCor2_Cuda9.1_v1.0.5 app_gctf = /opt/gctf/Gctf-v1.18_sm_30_cu8.0_x86_64 app_ctffind = /opt/ctffind-4.1.10/ctffind app_gautomatch = /programs/x86_64-linux/gautomatch/gautomatch.rc dir_eman2 = /opt/EMAN2 dir_imod = /opt/imod/imod_4.9.8 dir_frealign = /programs/x86_64-linux/frealign dir_relion = /opt/relion-2.1.stable Default_CS = 2.7 Default_KV = 300 Default_phase_shift_doit = n status_folder_update = n status_folder = none status_microscope = Titan status_refresh_rate = 0 status_webdav = http://my.server.com/files website_blur_doit = y website_blurkernel = 7 SCRATCH_DISK = . FAST_DISK = . GPU_how_many = 4 GPU_to_use = 0 1 2 3 GPU_cycle = n MotionCor2_alone = y

#############################################

Now the remainder of the original script:

#############################################

:++++++++++++++++++++++++++++++++ :Initializing script environment :++++++++++++++++++++++++++++++++ date = Wed Jan 8 14:42:12 CST 2020 system = Linux bin_2dx = /opt/focus/kernel/mrc/bin proc_2dx = /opt/focus/scripts/proc :==== dir_eman2 = /opt/EMAN2 =========================================================================================================================================================================== :==== app_python = /home/drennan@ad.wisc.edu/.conda/envs/focusenv/bin/python =========================================================================================================================== :==== app_anaconda = /home/drennan@ad.wisc.edu/.conda/envs/focusenv/bin/python ========================================================================================================================= :==== dir_frealign = /programs/x86_64-linux/frealign =================================================================================================================================================== :==== dir_relion = /opt/relion-2.1.stable ============================================================================================================================================================== :==== dir_ccp4 = /opt/ccp4/ccp4-7.0 ==================================================================================================================================================================== Finding CSH setup script from CCP4 installation.. : Sourcing /opt/ccp4/ccp4-7.0/bin/ccp4.setup-csh : Finished with /opt/ccp4/ccp4-7.0/bin/ccp4.setup-csh ::Will use 1 threads for optimal load balancing. [1] 25436

No handlers could be found for logger "mrcz.test_mrcz"

Job 1/1 picking particles from micrographs 1 to 1...

:: Now boxing unit cells of micrograph 1/1.

../auto/0000000019/mimage_2dx.mrc

Traceback (most recent call last): File "/opt/focus/scripts/proc/SPR_ExtractParticles.py", line 1187, in main() File "/opt/focus/scripts/proc/SPR_ExtractParticles.py", line 308, in main img = -1.0 * img TypeError: can't multiply sequence by non-int of type 'float' [1] Exit 1 /home/drennan@ad.wisc.edu/.conda/envs/focusenv/bin/python /opt/focus/scripts/proc/SPR_ExtractParticles.py ../ SPR/2dx_merge_dirfile-unique.dat SPR/picking/ SPR/stacks/ particles 500 0.0,180.0 1.37 300 2.7 0.995 38540 0.0 y y y n y n 0.4 1.0 -1 Merge Particle n y y 1 1 ::Now organizing and converting partial stack(s) to single .mrcs and .par files... then: then/endif not found. ::######################PickExportParticleStack finished.#######################
rdrighetto commented 4 years ago

Dear Amanda,

Thanks for sending the logfile. Does it work if you use a Python 3.X version? Make sure that mrcz is available also for Python 3, e.g. by running pip3 install mrcz.

Best wishes, Ricardo

acdrennan commented 4 years ago

Hi Ricardo, I made a new conda environment with python3 and then ran pip install mrcz. Then I opened focus and ran "pick and export particles" on that same single image; the output gave different errors this time. Best, Amanda

Successfully installed mrcz-0.5.4 numpy-1.18.1 (focusenv3) [drennan@ad.wisc.edu@BIOCWK-01037L data]$ focus & [1] 4210 (focusenv3) [drennan@ad.wisc.edu@BIOCWK-01037L data]$ cat /mnt/data/top100tilt0/merge/LOGS/PickExportParticleStack.log ::###########################PickExportParticleStack############################

############################################

Parameters requested by the Script

############################################ thread_number = 12 scriptname = spr_sample_pixel = 1.37 spr_doresample = y spr_resmin = 100.0 spr_resmax = 10.0 spr_phaseshift = 0.0,180.0 spr_boxsize = 500 spr_sigcc = 0.0 spr_invertcontrast = y spr_normalizebox = y spr_calculatedefocustilted = y spr_savephaseflipped = n spr_savectfmultiplied = y spr_savewienerfiltered = n spr_wienerconstant = 0.4 spr_sigmanorm = 1.0 spr_radnorm = -1 spr_whichtiltgeometry = Merge spr_whichctf = Particle spr_whichhalfset = particle-based spr_savepickfig = n cs = 2.7 kv = 300 sample_pixel = 1.0 phacon = 0.995 magnification = 38540 use_masked_image = y spr_ignore_lat2 = y spr_ignore_c1 = y spr_createstarfile = y spr_dir = SPR spr_imgs_dir = spr_mergefile = spr_stacks_dir = spr_picking_dir = spr_stack_rootname =

############################################

Parameters from the Preferences settings

############################################ app_python = /home/drennan@ad.wisc.edu/.conda/envs/focusenv3/bin/python3 app_cadaver = /usr/bin/cadaver app_anaconda = /home/drennan@ad.wisc.edu/.conda/envs/focusenv3/bin/python3 dir_ccp4 = /opt/ccp4/ccp4-7.0 app_gawk = /usr/bin/gawk cuda_library = /usr/local/cuda/lib64 app_unblur = /opt/unblur_1.0.2/bin/unblur_openmp_7_17_15.exe app_motioncor2 = /opt/MotionCor2/MotionCor2_Cuda9.1_v1.0.5 app_gctf = /opt/gctf/Gctf-v1.18_sm_30_cu8.0_x86_64 app_ctffind = /opt/ctffind-4.1.10/ctffind app_gautomatch = /programs/x86_64-linux/gautomatch/gautomatch.rc dir_eman2 = /opt/EMAN2 dir_imod = /opt/imod/imod_4.9.8 dir_frealign = /programs/x86_64-linux/frealign dir_relion = /opt/relion-2.1.stable Default_CS = 2.7 Default_KV = 300 Default_phase_shift_doit = n status_folder_update = n status_folder = none status_microscope = Titan status_refresh_rate = 0 status_webdav = http://my.server.com/files website_blur_doit = y website_blurkernel = 7 SCRATCH_DISK = . FAST_DISK = . GPU_how_many = 4 GPU_to_use = 0 1 2 3 GPU_cycle = n MotionCor2_alone = y

#############################################

Now the remainder of the original script:

#############################################

:++++++++++++++++++++++++++++++++ :Initializing script environment :++++++++++++++++++++++++++++++++ date = Thu Jan 9 08:24:40 CST 2020 system = Linux bin_2dx = /opt/focus/kernel/mrc/bin proc_2dx = /opt/focus/scripts/proc :==== dir_eman2 = /opt/EMAN2 =========================================================================================================================================================================== :==== app_python = /home/drennan@ad.wisc.edu/.conda/envs/focusenv3/bin/python3 ========================================================================================================================= :==== app_anaconda = /home/drennan@ad.wisc.edu/.conda/envs/focusenv3/bin/python3 ======================================================================================================================= :==== dir_frealign = /programs/x86_64-linux/frealign =================================================================================================================================================== :==== dir_relion = /opt/relion-2.1.stable ============================================================================================================================================================== :==== dir_ccp4 = /opt/ccp4/ccp4-7.0 ==================================================================================================================================================================== Finding CSH setup script from CCP4 installation.. : Sourcing /opt/ccp4/ccp4-7.0/bin/ccp4.setup-csh : Finished with /opt/ccp4/ccp4-7.0/bin/ccp4.setup-csh ::Will use 1 threads for optimal load balancing. [1] 4351

File "/opt/focus/scripts/proc/SPR_ExtractParticles.py", line 138 print '\nJob %d/%d picking particles from micrographs %d to %d...\n' % (this_thread, n_threads, n, last_img) ^ SyntaxError: invalid syntax [1] + Exit 1 /home/drennan@ad.wisc.edu/.conda/envs/focusenv3/bin/python3 /opt/focus/scripts/proc/SPR_ExtractParticles.py ../ SPR/2dx_merge_dirfile-unique.dat SPR/picking/ SPR/stacks/ particles 500 0.0,180.0 1.37 300 2.7 0.995 38540 0.0 y y y n y n 0.4 1.0 -1 Merge Particle n y y 1 1 ::Now organizing and converting partial stack(s) to single .mrcs and .par files... then: then/endif not found. ::######################PickExportParticleStack finished.#######################
rdrighetto commented 4 years ago

OK, from this new error message I can confirm that somehow you are not running the latest version of the source code:

File "/opt/focus/scripts/proc/SPR_ExtractParticles.py", line 138
print '\nJob %d/%d picking particles from micrographs %d to %d...\n' % (this_thread, n_threads, n, last_img)

This line, as written above, is not compatible with Python3, however the code has been updated a few months ago to be Python3 compliant... If you are updating from the source code (git pull), you can try cleaning up your build/ directory and running ./build_all [...] again to ensure the updated files will be copied to the proper location... worst case, you can try a fresh installation. Please feel free to contact me via email if you need more in depth assistance, we can set up a Skype meeting if needed. I am very sorry for the inconvenience.

Best wishes, Ricardo

acdrennan commented 4 years ago

Ok, thank you so much!!!

acdrennan commented 4 years ago

Pick & export particles works after a fresh installation.

arndtbill1 commented 2 years ago

Hi Focus team,

I am experiencing the same error that Xi described above. I was wondering if a solution was found?

Thanks! Bill

::###########################PickExportParticleStack############################

############################################

Parameters requested by the Script

############################################ thread_number = 12 scriptname = spr_sample_pixel = 1.078 spr_doresample = y spr_resmin = 100.0 spr_resmax = 10.0 spr_phaseshift = 0.0,0.0 spr_boxsize = 256 spr_sigcc = 0.0 spr_invertcontrast = y spr_normalizebox = y spr_calculatedefocustilted = y spr_savephaseflipped = n spr_savectfmultiplied = y spr_savewienerfiltered = n spr_wienerconstant = 0.4 spr_sigmanorm = 1.0 spr_radnorm = 0.90 spr_whichtiltgeometry = Defocus spr_whichctf = Particle spr_whichhalfset = crystal-based spr_savepickfig = y cs = 2.7 kv = 300 sample_pixel = 1.078 phacon = 0.995 magnification = 46382.2 use_masked_image = n spr_ignore_lat2 = y spr_ignore_c1 = y spr_createstarfile = y spr_dir = SPR spr_imgs_dir = spr_mergefile = spr_stacks_dir = spr_picking_dir = spr_stack_rootname =

############################################

Parameters from the Preferences settings

############################################ app_python = /usr/bin/python3 app_cadaver = /usr/bin/cadaver app_anaconda = /usr/bin/python3 dir_ccp4 = /garnet5/arndt/ccp4-7.1 app_gawk = /programs/x86_64-linux/system/gawk/4.1.4/bin/gawk cuda_library = /programs/x86_64-linux/cuda/9.2.88/cuda/lib64/libcublas.so app_unblur = /programs/x86_64-linux/unblur/1.0.2/bin/unblur_openmp_7_17_15.exe app_motioncor2 = /programs/x86_64-linux/motioncor2/1.2.1/motioncor2 app_gctf = /usr/local/Gctf_v1.06/bin/Gctf-v1.06_sm_30_cu8.0_x86_64 app_ctffind = /programs/x86_64-linux/ctffind4/4.1.14-c7/bin/ctffind4 app_gautomatch = /usr/local/Gautomatch_v0.56_sm61_cu8.0 dir_eman2 = /programs/x86_64-linux/eman2 dir_imod = /usr/local/IMOD dir_frealign = /usr/local/frealign_v9.11 dir_relion = /programs/x86_64-linux/relion/3.1.2_cu9.2 Microscope_Name = CryoEM Default_CS = 2.7 Default_KV = 300 Default_phase_shift_doit = n status_folder_update = n status_folder = none status_microscope = Titan status_refresh_rate = 0 status_webdav = http://my.server.com/files website_blur_doit = y website_blurkernel = 7 SCRATCH_DISK = . FAST_DISK = /garnet2-ssd/arndt/Focus_scratch GPU_how_many = 4 GPU_to_use = 0 1 2 3 GPU_cycle = n MotionCor2_alone = y notify_email_admin = y notify_email_admin_adress = bugs@sbgrid.org notify_last_minutes = 15 notify_delay_minutes = 60 notify_lastnumber = 50 notify_limit_mild = 30 notify_limit_severe = 15 flag_isdark_threshold = 0.1 flag_hasstripe_threshold = 38.0 flag_hastoohighdefocus_threshold = 7.0 flag_hastoolowdefocus_threshold = 0.2 flag_icinesstoohigh_threshold = 1.6 flag_drifttoohigh_threshold = 50.0 flag_astigmatismtoohigh_threshold = 0.03 flag_ctfresolutiontoobad_threshold = 8.0

#############################################

Now the remainder of the original script:

#############################################

:++++++++++++++++++++++++++++++++ :Initializing script environment :++++++++++++++++++++++++++++++++ : : Running the following shell: : version tcsh 6.21.00 (Astron) 2019-05-08 (x86_64-unknown-linux) options wide,nls,dl,al,kan,sm,rh,nd,color,filec : date = Thu 21 Oct 2021 01:44:16 PM CDT system = Linux bin_2dx = /opt/sbgrid/x86_64-linux/focus/20201022/kernel/mrc/bin proc_2dx = /opt/sbgrid/x86_64-linux/focus/20201022/scripts/proc :==== dir_eman2 = /programs/x86_64-linux/eman2 ========================================================================================================================================================= :==== app_python = /usr/bin/python3 ==================================================================================================================================================================== :==== app_anaconda = /usr/bin/python3 ================================================================================================================================================================== :==== dir_frealign = /usr/local/frealign_v9.11 ========================================================================================================================================================= :==== dir_relion = /programs/x86_64-linux/relion/3.1.2_cu9.2 =========================================================================================================================================== :==== dir_ccp4 = /garnet5/arndt/ccp4-7.1 =============================================================================================================================================================== Finding CSH setup script from CCP4 installation.. : Sourcing /garnet5/arndt/ccp4-7.1/bin/ccp4.setup-csh : Finished with /garnet5/arndt/ccp4-7.1/bin/ccp4.setup-csh ::Will use 1 threads for optimal load balancing. [1] 7989

Job 1/1 picking particles from micrographs 1 to 1...

[1] Done /usr/bin/python3 /opt/sbgrid/x86_64-linux/focus/20201022/scripts/proc/SPR_ExtractParticles.py ../ SPR/2dx_merge_dirfile-unique.dat SPR/picking/ SPR/stacks/ particles 256 0.0,0.0 1.078 300 2.7 0.995 46382.2 0.0 y y y n y n 0.4 1.0 0.90 Defocus Particle y n y 1 1 ::Now organizing and converting partial stack(s) to single .mrcs and .par files... then: then/endif not found. ::######################PickExportParticleStack finished.#######################
rdrighetto commented 2 years ago

Hi Bill,

Apologies for not replying to your earlier requests! This looks like there was some error when extracting the particles.

1) Can you confirm whether there are any files (and which ones) inside the folder /SPR/stacks/ ?

2) Can you please send a more detailed output of this script? (please choose maximum verbose level on the Focus GUI)

Thanks! Best wishes, Ricardo

arndtbill1 commented 2 years ago

Hi Ricardo,

Thank you for getting back to me.

A "particles_1_r1.par" is being generated in the /SPR/stacks folder, but the par file is empty.

However, I have now been able extraction to work! I reinstalled all the needed dependencies, which might have done the trick (numexpr, pillow, matplotlib). Example: python3 -m pip install matplotlib

Best, Bill