Unsuppored stack-tomoman_tiltctf.param

[yitliao]

Hi, I was trying to follow the tutorail for getting familiar using TOMOMAN. However, I got some problems when I trying to run the tomoman_tiltctf.param and I am hoping you could give me some advice. Here's the error message:

TOMOMAN > tomoman(pwd,'tomoman_tiltctf.param'); TOMOMAN: Initializing!!! TOMOMAN: Old tomolist found... Loading tomolist!!! MATLAB error: Error using tm_tiltctf_ctffind4 (line 69) TOMOMAN: ACTHUNG!!! Unsuppored stack!!! Only "unfiltered" and "dose-filtered" supported!!!

here are the folders: MotionCor2, AreTomo and Files: TS_01_dose-filt.rawtlt TS_01.rawtlt none.rawtlt TS_01_dose-filt.st TS_01.st

As I am using the standalone, I have no idea how to open the tomolist.mat...

Thank you!

[williamnwan]

Hi,

Can you post your tomoman_tiltctf.param file?

Thanks, Will

[yitliao]

Hi Will,

Thanks for your reply.

Here's the tomoman_tiltctf.param file

tomoman_tiltctf

Directory Parameters

root_dir = /store0/yliao/Stopgap/HIV_dataset/tomo/ts1/ # Root folder for dataset stack directories will be generated here. tomolist_name = tomolist.mat # Relative to root_dir log_name = tomoman.log # Relative to root_dir

Tilctf Parameters

force_tiltctf = 1 # 1 = yes, 0 = no ps_size = 512 # Size of power-spectrum in pixels def_tol = 0.05 # Tilted defocus tolerance in microns. This determines sampling rate along the x-axis of the tilt-series. fscaling = 2 # Scaling in Fourier space. calc_ps = 1 # 1 = to run ps calculation from scratch, 0 = to only run ctffind4 on already calculated ps def_range = 0.5 # Defocus fitting search tolerance in microns. TOMOMAN will use this and the "target_defocus" values in the tomolist to calculate the CTFFIND4 min_res and max_res parameters. use_xtilt = 1 # Check for xtilt value and use if available. handedness = 1 # Determines the CTF handedness.

CTFFIND parameters

cs = 2.7 # Spherical aberration famp = 0.07 # Ampltidue contrast min_res = 30 # Minimum resolution to fit max_res = 5 # Maximum resolution to fit def_step = 0.01 # Defocus search step in microns. Default is 0.01. known_astig = 0 # Do you know what astigmatism is present? (0 = no, 1 = yes). Default is 0 slower = 0 # Slower, more exhaustive search (0 = no, 1 = yes). Default is 0 astig = 0 # Known astigmatism. astig_angle = 0 # Known astigmatism angle. rest_astig = 1 # Restrict astigmatism (0 = no, 1 = yes). Default = 1 exp_astig = 200 # Expected (tolerated) astigmatism. Default is 200. det_pshift = 0 # Determine phase shift (0 = no, 1 = yes). pshift_min = 0 # Minimum phase shift (rad). Default = 0.0. pshift_max = 3.15 # Maximum phase shift (rad). Default = 3.15. pshift_step = 0.1 # Phase shift search step. Default = 0.1. expert = 1 # Do you want to set expert options? (0 = no, 1 = yes) Default is 0 resample = 1 # Resample micrograph if pixel size too small? (0 = no, 1 = yes) known_defocus = 0 # Do you already know the defocus? (0 = no, 1 = yes) Default is 0 known_defocus_1 = 0.0 # Known defocus 1 . Default is 0 known_defocus_2 = 0.0 # Known defocus 2 . Default is 0 ctffinf.known_defocus_astig = 0 # Known defocus astigmatism. Default is 0 ctffinf.known_defocus_pshift = 0 # Known defocus phase shift in radians. Default is 0 nthreads = 14 # Desired number of parallel threads.

Thank you!

[williamnwan]

Did you already perform tilt-series alignment with AreTomo? If so, can you also include that .param file. It might also be helpful if you attached your tomolist file.

Thanks.

[yitliao]

Hi Will, Yes, I already have the folder called bin8_aretomo which includes file "TS_01_dose-filt_bin8.mrc" here's the tomoman_aretomo.param

tomoman_aretomo

Directory Parameters

root_dir = /store0/yliao/Stopgap/HIV_dataset/tomo/ts1/ # Root folder for dataset stack directories will be generated here. tomolist_name = tomolist.mat # Relative to root_dir log_name = tomoman.log # Relative to root_dir

AreTomo Files and Folders

out_dir = bin8_aretomo/ # Output directory for reconstructed tomograms, relative to root_dir process_stack = dose-filtered # Stack for processing. Either 'unfiltered' or 'dose-filtered' aretomo_list = none # List of per-tilt series parameters. By column, tomo_num, AlignZ, and VolZ; AlignZ and VolZ are optional, but if used, must have all entries. Tomograms in the tomolist but not in the aretomo_list will not be processed. Set to 'none' for no list. ovZ = none # Override thickness in IMOD-format tilt.com. TOMOMAN will normally take the VolZ value, but this allows for overriding this. Units are in bin1 pixels. Set to 'none' for no override.

AreTomo Parameters

InBin = 4 # TOMOMAN parameter for binning of input tilt series
AlignZ = 1400 # Approximate thickness of specimen in unbinned pixels VolZ = 1800 # Thickness for tomogram reconstruction in unbinned pixels OutBin = 8 # Output binning (Total binning, independent of the InBin value) TiltCor = 1 # Correct for tilt angle offset. See AreTomo manual for details FlipVol = 1 # Flip into standard xy-plane FlipInt = 0 # Flip output intensities Sart = none # Parameters for SART reconstruction. Default parameters are 20,5. Set to none to skip. Wbp = 1 # WBP reconstruction. (1 = yes, 0 = no) Roi = none # ROI for focused alignment. Set to none to skip. Patch = none # Parameters for patch-based local motion correction. Set to none to skip. OutImod = 1 # Write out IMOD formatted files for Relion4 inputs. (1 = yes, 0 = no) OutXF = 1 # Write out IMOD formatted .xf file. (1 = yes, 0 = no) DarkTol = 0.001 # Tolerance for detecting and removing dark images. Default = 0.70. Gpu = 0 1 2 # ID of GPU to use for processing force_align = 0 # Force AreTomo to re-align stacks. (1 = yes, 0 = no)

tomolist.mat.zip

[williamnwan]

I'm not sure why, but the tomolist doesn't have a record of you running AreTomo.

To check yourself, you can load the tomolist in the standalone console and print out the first line like this:

load('tomolist.mat'); tomolist(1)

You should get this output: ans =

struct with fields:

                   root_dir: '/store0/yliao/Stopgap/HIV_dataset/tomo/ts1/'
                  stack_dir: '/store0/yliao/Stopgap/HIV_dataset/tomo/ts1/TS_01/'
                  frame_dir: '/store0/yliao/Stopgap/HIV_dataset/tomo/ts1/TS_01/frames/'
                  mdoc_name: 'TS_01.mrc.mdoc'
                   tomo_num: 1
            collected_tilts: [41x1 double]
                frame_names: {41x1 cell}
                   n_frames: 8
                  pixelsize: 1.3500
                 image_size: [3712 3712]
                    voltage: 300
   cumulative_exposure_time: [41x1 double]
                       dose: [41x1 double]
             target_defocus: -4
                    gainref: []
               defects_file: []
                rotate_gain: 0
                  flip_gain: 0
             raw_stack_name: 'none'
               mirror_stack: []
                       skip: 0
             frames_aligned: 1
  frame_alignment_algorithm: 'MotionCor2'
                 stack_name: 'TS_01.st'
                clean_stack: 1
              removed_tilts: []
                     rawtlt: [41x1 double]
                   max_tilt: 60.0006
                   min_tilt: -59.9986
            tilt_axis_angle: 85.3000
              dose_filtered: 1
   dose_filtered_stack_name: 'TS_01_dose-filt.st'
      dose_filter_algorithm: 'TOMOMAN-images'
          imod_preprocessed: 0
              stack_aligned: 0
            alignment_stack: 'none'
         alignment_software: 'none'
             ctf_determined: 0
             ctf_parameters: []
ctf_determination_algorithm: 'none'
         determined_defocii: []
                tomo_recons: 0
      tomo_recons_algorithm: []
                   metadata: []

Looking at your AreTomo param file, it should have stack_aligned: 1 alignment_stack: 'dose-filtered' alignment_software: 'aretomo'

but instead all these fields are empty.

Try running AreTomo again and see if they get updated. Maybe there's write permissions issue?

Best, Will

[yitliao]

Hi Will,

Thanks for your kind reply. I noticed that the file "tomoman_aretomo.param" I have is missing the following lines:

"# Aligned Stack Output write_ali_stack = 0 # Write out an aligned stack using AreTomo parameters ali_stack_binning = 8 # Binning of output aligned stack ali_stack_dir = bin8_aretomo_ali/ # Output directory for aligned stacks, relative to root_dir

After adding these lines back to my "tomoman_aretomo.param" file and rerunning AreTomo, I received the following error message:

Done! TOMOMAN: Coverting AreTomo files to IMOD for stack: TS_01.st MATLAB error: Reference to non-existent field 'write_ali_stack'.

I guess it is probably the reason... Is there anything else I should check or do to resolve this issue?

Thank you!

[williamnwan]

These aligned stack output fields were added on July 15, and there have also been a few bug fixes since then. I think the latest were on August 28, so you might want to double check.

If you have an old TOMOMAN, adding those lines will cause such errors.

If you have a new TOMOMAN without those lines, I think you'll also generate those errors.

[yitliao]

Hi Will,

Thank you! I'll contact SBGrid to upgrade TOMOMAN to the latest version (currently, the version is 0.9 on Linux 64, Just updates in the end of July).

And, I also noticed that the error message "Reference to non-existent field 'write_ali_stack'" appears whenever I add those lines. I was wondering if you think this is the reason why alignment_stack and alignment_software fields remain empty in my tomolist.

Thank you so much!

[williamnwan]

This error is because the older version doesn't know what to do with those inputs.

For now, if you remove those lines and just re-run AreTomo, do the alignment_stack and alignment_software fields remain empty? You can also check the edit date on the tomolist before and after you run AreTomo to see if it's being edited.

[yitliao]

Hi Will, Unfortunately, the fields remain empty and I still get the error message "Reference to non-existent field 'write_ali_stack'" after I remove those lines and re-run Aretomo. And the tomolist seems not to be edited after running Aretomo... Besides, I was wondering if there is a way to specify the MotionCor2 version in tomoman_motioncor2.param (e.g. MotionCor2_1.6.4_cuda11X)

Thank you

[williamnwan]

Let's see if SBGrid updates fixes this problem. I can also help you setup your own installation outside of SBGrid if you would like.

For using different versioning, you should be able to just source different versions using the .sbgrid.conf file, since they all have the same executable name. Otherwise, you can override defaults by adding an edited tm_dependencies.param file to your processing folder (rootdir).

GitHub won't let me upload the file with the .param ending, so I've attached it as a .txt; just update the name.

tm_dependencies.txt

[yitliao]

Hi Will, Thank you so much for your kind help! It works now :) We’d definitely like to set up TOMOMAN/STOPGAP on our workstation. We can purchase a MATLAB license if you recommend using MATLAB, or do you think the standalone version is sufficient? Could you provide some guidance on this? Our lab currently uses SBGrid for all of our software installations. Thank you!!

[yitliao]

Hi Will,

I hope you're doing well. I've gone through the entire tutorial for TOMOMAN/STOPGAP, and it works for me. However, I recently encountered different errors when using "tomoman_relion_motioncorr.param" for an EER dataset, and I was wondering if you could give me some advice.

The error occurred after running tomoman(pwd, 'tomoman_relion_motioncorr.param'); with the message:

"MATLAB error: Reference to non-existent field 'bin_factor'."

As a test, I tried using Relion 5 GUI for motion correction on the same dataset, and it worked. I also used the chmod 777 command for the folder to make sure its not a permission issue...

Thank you!

Here are param files I used before:

“tomoman_relion_motioncorr”:

tomoman_relion_motioncorr

Tomolist parameters

root_dir = /store0/yliao/Stopgap/chal2/test_1016/ # Root folder for dataset; stack directories will be generated here. tomolist_name = tomolist.mat # Relative to root_dir log_name = tomoman.log # Relative to root_dir

Parallelization

n_cores = 24 # Number of CPU cores to use.

Align frames / generate stack

force_realign = 0 # 1 = yes, 0 = no; image_size = 4096,4096 # Crops or expands to image_size. Useful to produce image dimensions with integer binning factors (e.g. K2)

Relion's motioncorr parameters

input_format = eer # 'tiff' or 'mrc' or 'eer' patch = 1 1 # Number of patches to be used for patch based alignment, Default 0 0 corresponding full frame alignment. bin_factor = 2 # Binning factor for input stack. bfactor = 150 # B-Factor for alignment, default 150. save_OddEven = 1 # Save odd/even frame sums.

EER-specific parameters

eer_dosefractions = 10 # Number of dose fractions. TOMOMAN will parse the number of frames from the .eer header and set the appropriate grouping size. eer_upsampling = 1 # EER upsampling (1 = 4K or 2 = 8K)

“tm_dependencis.param”: cp = cp # Linux copy mv = mv # Linux move ln = ln # Linux link relionmc = relion_run_motioncorr # Relion MotionCorr2 relion_import = relion_import # Relion Import motioncor2 = MotionCor2_1.6.4_Cuda113_Mar312023 # MotionCor2 imod_3dmod = 3dmod # IMOD 3dmod newstack = newstack # IMOD newstack clip = clip # IMOD clip aretomo = AreTomo # AreTomo ctffind4 = ctffind # CTFFIND4 etomo = etomo # IMOD etomo; assume presence of etomo inclues all other functions novactf = novaCTF # NovaCTF fourier3d = Fourier3D # Fourier3D
mpiexec = mpiexec # Command to run MPI cryoCARE_extract_train_data = cryoCARE_extract_train_data.py # CryoCARE Extract cryoCARE_train = cryoCARE_train.py # CryoCARE Train cryoCARE_predict = cryoCARE_predict.py # CryoCARE Predict

“tomoman_import.param”:

tomoman_import

Directory parameters

root_dir = /store0/yliao/Stopgap/chal2/test_1016/ # Root folder for dataset; stack directories will be generated here. raw_stack_dir = mdoc/ # Folder containing raw stacks; relative to root_dir. raw_frame_dir = frames/ # Folder containing unsorted frames; relative to root_dir.

Tomolist

tomolist_name = tomolist.mat # Name of tomolist. Saved relative to root_dir. log_name = tomoman.log # Name of tomoman log file. Saved relative to root_dir

Filename parameters

raw_stack_ext = none # File extension of raw stacks. Set to 'none' to skip

Data collection parameters

gainref = gain/gainref/EER_GainReference_reciprocal.mrc

For no gainref, set to 'none'

defects_file = none # For no defects_file, set to 'none' rotate_gain = 0 # Gain ref rotation flip_gain = 0 # Gain ref flip; 0 = none, 1 = up/down, 2 = left/right os = windows # Operating system for data collection. Options are 'windows' and 'linux'. This is usually 'windows'. mirror_stack = y # Mirror images, MAKE SURE YOU KNOW YOUR STUFF!!; voltage = 300 # Voltage used for data collection in kV.

Override .mdoc values (set to none for no override)

tilt_axis_angle = -85 # Tilt axis angle in degrees dose_rate = 6.2 # e/pixel/s pixelsize = 1.96 # Pixel size in Angstroms target_defocus = none # Target defocus in Microns

Sorting parameters (0 = false, 1 = true)

ignore_raw_stacks = 1 # Generate tilt-series folder even if raw stack is missing ignore_missing_frames = 0 # Generate tilt-series folder even if frames are missing

[williamnwan]

Hi Yitao,

This task was written for relion4 and we haven't had a chance to try it on relion5. Please give it a try with relion4 and see if it works.

Best, Will

[yitliao]

Hi Will, Thank you for getting back to me. I have added the paths to the tm_dependencies.param file as follows: relionmc=/XXX/path/relion/4.0.1.cu12.2/bin/relion_motion_correction and relion_import=/XXX/path/relion/4.0.1.cu12.2/bin/relion_import. However, it still doesn't work for me. I noticed in line 152 of the "tm_relion_motioncorr_batch_wrapper" script the following code: relion_import_cmd = [dep.relion_import,' --i "', '.',relionmc.input_format,'" --ofile movies.star --angpix ', num2str(tomolist.pixelsize,'%f'), ' --odir ./ --do_movies']; %disp(relion_import_cmd); system(relion_import_cmd); I was wondering if the placement of the double and single quotes between --i and relionmc.input_format (i.e., dep.relion_import,' --i "', '.',relionmc.input_format,'") could be affecting how the script runs.

Besides, I was wondering if it's possible to convert .mod and .motl files (manually particle picking from 3dmod) or .star files (from cube/Warp) into the StopGap STAR file format.

Thank you.

[williamnwan]

The executable name for Relion's motioncor is "relion_run_motioncorr", so you can try updating the dependency. Alternatively, if you're using SBGrid, you can also set your motioncor version using the ~/.sbgrid.conf file.

I believe that the relion_import command should be fine.

Also, what does the failure look like, i.e. what output do you get?

For converting particle lists to stopgap, I can have a look at writing some conversion scripts if you can send me the documentation for the different formats. For IMOD .mod files, my recollection is that there is only a 3D coordinate but no angular information. Also, my understanding is that they're on a per-tomogram basis, so there would be some type of iteration required to pick up a full dataset. I'm not familiar with the .motl convention.

For Warp, are their star files the same as relion?

In general, particle lists should be relatively easy to adapt. The wedgelists are more complicated due to all the different bits of metadata that need to be put together, which was part of the motivation for starting the TOMOMAN project.