Making a STAR file for simulated particles

anoy-2020codereivew commented 3 years ago

Describe your problem

If I want to start 3D initialization directly in simulation data (includes 100000 2D images stack: mrcfile), how can I write the star file?

The example I write is:'rlnImageBoxSize', 'rlnOpticsGroupName', 'rlnOpticsGroup', 'rlnVoltage', 'rlnSphericalAberration', 'rlnAmplitudeContrast'. But It appears a error: It can't find a particle. Could you help me? Thanks Hanlin

Environment:

OS: [e.g. Ubuntu 16.04 LTS]
MPI runtime: [e.g. OpenMPI 2.0.1]
RELION version [e.g. RELION-3.1-devel-commit-6ba935 (please see the title bar of the GUI)]
Memory: [e.g. 128 GB]
GPU: [e.g. GTX 1080Ti]

Dataset:

Box size: [e.g. 256 px]
Pixel size: [e.g. 0.9 Å/px]
Number of particles: [e.g. 150,000]
Description: [e.g. A tetrameric protein of about 400 kDa in total]

Job options:

Type of job: [e.g. Refine3D]
Number of MPI processes: [e.g. 4]
Number of threads: [e.g. 6]

Full command (see note.txt in the job directory):

`which relion_refine_mpi` --o Refine3D/job019/run --auto_refine --split_random_halves --i CtfRefine/job018/particles_ctf_refine.star --ref PostProcess/job001/postprocess.mrc --firstiter_cc --ini_high 12 --dont_combine_weights_via_disc --scratch_dir /ssd --pool 3 --pad 2  --ctf --ctf_corrected_ref --particle_diameter 142 --flatten_solvent --zero_mask --solvent_mask Result-by-Rado/run_class001_mask_th0.01_ns3_ngs7_box400.mrc --solvent_correct_fsc  --oversampling 1 --healpix_order 3 --auto_local_healpix_order 4 --offset_range 5 --offset_step 2 --sym O --low_resol_join_halves 40 --norm --scale  --j 8 --gpu "" --keep_scratch --pipeline_control Refine3D/job019/

Error message:

Please cite the full error message as the example below.

A line in the STAR file contains fewer columns than the number of labels. Expected = 3 Found = 2
Error in line: 0 0.0
in: /prog/relion-devel-lmb/src/metadata_table.cpp, line 966
=== Backtrace  ===
/prog/relion-devel-lmb/bin/relion_motion_refine_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x42e981]
/prog/relion-devel-lmb/bin/relion_motion_refine_mpi(_ZN13MetaDataTable12readStarLoopERSt14basic_ifstreamIcSt11char_traitsIcEEPSt6vectorI8EMDLabelSaIS6_EESsb+0xedd) [0x4361ad]
/prog/relion-devel-lmb/bin/relion_motion_refine_mpi(_ZN13MetaDataTable8readStarERSt14basic_ifstreamIcSt11char_traitsIcEERKSsPSt6vectorI8EMDLabelSaIS8_EESsb+0x580) [0x436f10]
/prog/relion-devel-lmb/bin/relion_motion_refine_mpi(_ZN10Micrograph4readE8FileNameb+0x5a3) [0x454bb3]
/prog/relion-devel-lmb/bin/relion_motion_refine_mpi(_ZN10MicrographC2E8FileNameS0_d+0x2e3) [0x4568b3]
/prog/relion-devel-lmb/bin/relion_motion_refine_mpi(_ZN17MicrographHandler14isMoviePresentERK13MetaDataTableb+0x180) [0x568280]
/prog/relion-devel-lmb/bin/relion_motion_refine_mpi(_ZN17MicrographHandler17cullMissingMoviesERKSt6vectorI13MetaDataTableSaIS1_EEi+0xe6) [0x568dc6]
/prog/relion-devel-lmb/bin/relion_motion_refine_mpi(_ZN13MotionRefiner4initEv+0x56f) [0x49e1ff]
/prog/relion-devel-lmb/bin/relion_motion_refine_mpi(main+0x31) [0x42a5e1]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x2b7ac026e495]
/prog/relion-devel-lmb/bin/relion_motion_refine_mpi() [0x42b3cf]
==================

biochem-fan commented 3 years ago

First, I edited your question. Please use a concise title and write details in the message body. You didn't write how you produced the synthetic dataset, how you ran Refine3D, how you generated your STAR file.

I recommend you to first go through RELION tutorial and then mimic particles.star from the Extract job there.

biochem-fan commented 3 years ago

At least you need the following.

data_optics

loop_ 
_rlnOpticsGroupName #1 
_rlnOpticsGroup #2 
_rlnMicrographOriginalPixelSize #3
_rlnVoltage #4
_rlnSphericalAberration #5 
_rlnAmplitudeContrast #6 
_rlnImagePixelSize #7 
_rlnImageSize #8 
_rlnImageDimensionality #9 
opticsGroup1            1    0.885000   200.000000     1.400000     0.100000     3.540000           64        2 

data_particles

loop_ 
_rlnImageName #1
_rlnMicrographName #2 
_rlnOpticsGroup #3 
_rlnDefocusU #4 
_rlnDefocusV #5 
_rlnDefocusAngle #6 

000001@particles.mrcs mic001.mrc    1  10863.857422 10575.721680    77.967194
000002@particles.mrcs mic001.mrc    1  10863.857422 10575.721680    77.967194
...

anoy-2020codereivew commented 3 years ago

Thanks for answering and sorry for the unclear part in question, I want to reconstruct the 3D volume based on clean 2D simulation images. (These are projections of one known structure).

Thus I don't have the micrograph, only have the image stack(128*128: image size) and some parameters. Could I do this in Relion starting from the 3D initialization step? And do I need _rlnMicrographName or _rlnMicrographOriginalPixelSize? Does Relion have some functions to generate the related star file? As I see your answer to getboxsize question, Should I add "boxsize" information in the mrc file instead of the star file?

My parameter are the following: rlnImageSize :128 'rlnImageBoxSize': 128, 'rlnImagePixelSize': 1.6, 'rlnOpticsGroupName': /home/hguaf/mixed.mrcs, rlnOpticsGroup': 1 'rlnVoltage': 300, 'rlnSphericalAberration': 2.26 'rlnAmplitudeContrast': 0.1

biochem-fan commented 3 years ago

And do I need _rlnMicrographName or _rlnMicrographOriginalPixelSize?

You can omit rlnMicrographOriginalPixelSize but not _rlnMicrographName. _rlnMicrographName is used internally to group particles for noise estimation. Just assign dummy names (the files do not have to exist) to every 100-200 particles. Putting all particles into one group may cause memory problems.

Does Relion have some functions to generate the related star file?

No.

As I see your answer to getboxsize question, Should I add "boxsize" information in the mrc file instead of the star file?

An MRC file always has the box size in the header. Nonetheless you have to specify _rlnImageSize in the STAR file.

Try to adapt my example above; if it still fails, show me the full error message and your STAR file.

anoy-2020codereivew commented 3 years ago

My star file looks like the following: data_optics

loop_ _rlnOpticsGroupName #1 _rlnOpticsGroup #2 _rlnVoltage #3 _rlnSphericalAberration #4 _rlnAmplitudeContrast #5 _rlnPixelSize #6 _rlnImagePixelSize #7 _rlnImageSize #8 _rlnImageDimensionality #9 opticsGroup1 1 300 2.26 0.1 1.6 1.6 128 2

data_particles

loop_ _rlnImageName #1 _rlnMicrographName #2 _rlnOpticsGroup #3 _rlnDefocusU #4 _rlnDefocusV #5 _rlnDefocusAngle #6 000@particle.mrcs /home/share/xuhuihuang/three_conformation_image/mix_data_mrc_200/mix000.mrc 1 3000 4000 20 001@particle.mrcs /home/share/xuhuihuang/three_conformation_image/mix_data_mrc_200/mix000.mrc 1 3000 4000 20

But If I try to start 3D initial model with the input of this star file, it appeals an error:

WARNING: ../../share/xuhuihuang/three_conformation_image/mixed_three_conf_new.star seems to be from a previous version of Relion. Attempting conversion... You should make sure metadata in the optics group table after conversion is correct. in: /home/hguaf/relion/src/jaz/io/star_converter.cpp, line 185 ERROR: BUG: cannot find name for particle... === Backtrace === /home/hguaf/relion/build/bin/relion_refine(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x446451] /home/hguaf/relion/build/bin/relion_refine(_ZN13StarConverter27convert_3p0_particlesTo_3p1ERK13MetaDataTableRS0_S3_Ssb+0x197a) [0x5185fa] /home/hguaf/relion/build/bin/relion_refine(_ZN16ObservationModel10loadSafelyESsRS_R13MetaDataTableSsib+0x4d7) [0x4fe807] /home/hguaf/relion/build/bin/relion_refine(_ZN10Experiment4readE8FileNamebbbbi+0x4b7) [0x543227] /home/hguaf/relion/build/bin/relion_refine(_ZN11MlOptimiser17initialiseGeneralEi+0x423) [0x4977f3] /home/hguaf/relion/build/bin/relion_refine(_ZN11MlOptimiser10initialiseEv+0x484) [0x49a7a4] /home/hguaf/relion/build/bin/relion_refine(main+0x33) [0x431293] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f51d5f42505] /home/hguaf/relion/build/bin/relion_refine() [0x434f4f] ERROR: BUG: cannot find name for particle...

But I am sure my link of mrc file is right and if I use relion_conver_star to transfer my file, the information of data particles and rln imagesize are lost

biochem-fan commented 3 years ago

Don't you have extra spaces? The new line code must be LF only, not CR+LF as in WIndows.

anoy-2020codereivew commented 3 years ago

Do you mean the star file has extra spaces? I don't do CR in star file. I make a mistake with rlnImageName and_rlnMicrographName.

The rlnImageName is the image stack link instead of _rlnMicrographName, the latter term is some fixed values. But it still exits the same error.

The star file is in the attachment, I transfer to the txt file to upload.

starfile.txt

anoy-2020codereivew commented 3 years ago

Here each 00000@particles.mrc is one image file(128*128)

biochem-fan commented 3 years ago

I don't do CR in star file

Wrong. Your attachment uses CR+LF as a new line code. See https://unix.stackexchange.com/questions/79702/how-to-test-whether-a-file-uses-crlf-or-lf-without-modifying-it.

biochem-fan commented 3 years ago

Note that stack numbers start from 1, not 0.

You should also avoid using absolute paths inside the STAR file. See https://www3.mrc-lmb.cam.ac.uk/relion/index.php/FAQs#What_is_the_right_way_to_import_files_outside_the_project_directory.3F.

I confirmed that the following STAR file is accepted.

data_optics

loop_
_rlnOpticsGroupName #1
_rlnOpticsGroup #2
_rlnVoltage #3
_rlnSphericalAberration #4
_rlnAmplitudeContrast #5
_rlnPixelSize #6
_rlnImagePixelSize #7
_rlnImageSize #8
_rlnImageDimensionality #9
opticsGroup1    1               300             2.26            0.1             1.6             1.6             128             2               

data_particles

loop_
_rlnImageName #1
_rlnMicrographName #2
_rlnOpticsGroup #3
_rlnDefocusU #4
_rlnDefocusV #5
_rlnDefocusAngle #6
01@particles.mrcs   mix000.mrc  1   3000    4000    20
02@particles.mrcs   mix000.mrc  1   3000    4000    20

anoy-2020codereivew commented 3 years ago

Oh, It works after I remove CR, thanks a lot.

Hanlin

anoy-2020codereivew commented 3 years ago

I encounter another warning in 3D classification:

Expectation iteration 1 of 25 000/??? sec ~~(,_,"> [oo] WARNING: norm_correction= 10.1035 for particle 62697 in group 314; Are your groups large enough? Or is the reference on the correct greyscale? WARNING: norm_correction= 10.2465 for particle 58846 in group 295; Are your groups large enough? Or is the reference on the correct greyscale?

Actually, I have 500 group micrographs and each group contains 200 images as you suggest, need I increase the group? Or preprocess the data before 3D reconstruction. ( I did relion_preprocess to normalize the image)

biochem-fan commented 3 years ago

I did relion_preprocess to normalize the image

Are your background diameter for preprocess and the particle diameter in refine consistent? Your grouping is OK.

If you have this warning only for a few particles, you can just remove them or disable normalization correction by --no_norm in relion_refine.

anoy-2020codereivew commented 3 years ago

My image size is 128, so I choose 110 as the diameter. Also, the particle diameter is consistent between preprocessing and refinement. Actually, one iteration about 240 particles exist warning. Does it influence the final results largely?

biochem-fan commented 3 years ago

Does it influence the final results largely?

Probably not but you can just remove them or use --no_norm.

ghl1995 commented 3 years ago

Oh, I ignore these images and finish the 3D reconstruction and auto refinement. Thanks for your patient help.

Best regards Hanlin

3dem / relion

Making a STAR file for simulated particles #707