warpem / warp

https://warpem.github.io/warp/
GNU General Public License v3.0
36 stars 7 forks source link

rlnTomoProjX/Y/Z/W are wrong but internally consistent in RELION 2D stack metadata #227

Closed rkjensen closed 6 days ago

rkjensen commented 1 week ago

I've tried to import an old-style RELION star file (RELION <3.1) to extract particles (ribosomes) with WarpTools (if needed see context below). If I extract with the --3d flag, and run relion_reconstruct, I get a perfectly fine looking ribosome, that I can then use for a round if local refinment, which for around 1000 particles goes to 18 A. If I do the same with the --2d flag, the angles don't work and local alignment just gives me noise. Global refinement is still running, but is currently at 27 A, looking like a ribosome, so I think the subtilt stacks as such are fine, and that it is rather an issue with the euler angle conversion. Let me know if you are interested in any of the star files.

EDIT: The 3d subtomograms are aligned in RELION4 and the 2D subtilts are aligned in RELION5

Below is my two extraction commands, in case I forgot a 2D specific parameter or similar. 2D: WarpTools ts_export_particles --2d --settings warp_tiltseries.settings --input_star ${input_starfile} --coords_angpix 1.7005 --output_star relion/particles.star --output_angpix 6.802 --box 64 --diameter 250 --perdevice 2 3D:WarpTools ts_export_particles --3d --settings warp_tiltseries.settings --input_star ${input_starfile} --coords_angpix 1.7005 --output_star relion3d/particles.star --output_angpix 6.802 --box 64 --diameter 250 --perdevice 2

For context: I'm converting a larger dataset from Warp 1.0.7 to WarpTools and everything to the point of subtomogram extraction has worked perfectly. I imported averaged frames, redid ctf estimation, imported mask files from old boxnet, imported old tilt alignments, redid 3D CTF (+ tomogram extraction for reference), and everything looks perfectly well.

alisterburt commented 1 week ago

Hi @rkjensen - this is really concerning, thanks for the report and for doing all of the correct controls. I can confirm that up until now we only made sure that RELION could refine stacks and that we could import and further refine results in M.

It's very much possible that our RELION output is off in a way that is internally consistent (tilts are properly related in Fourier space) but actually wrong (product of tilt matrix and particle matrix is wrong in the same way for all particles)

I'll have to dig into this properly - #21 will need to be done properly which is a bit of a can of worms...

cc @dtegunov this is a big blocker for release, stops people using priors on orientation

dtegunov commented 6 days ago

@rkjensen could you please share the first particle line from the 2D and 3D STAR files?

rkjensen commented 6 days ago

Here you go: 3D: `data_

loop_ _rlnCoordinateX #1 _rlnCoordinateY #2 _rlnCoordinateZ #3 _rlnAngleRot #4 _rlnAngleTilt #5 _rlnAnglePsi #6 _rlnMicrographName #7 _rlnMagnification #8 _rlnDetectorPixelSize #9 _rlnCtfMaxResolution #10 _rlnImageName #11 _rlnCtfImage #12 _rlnPixelSize #13 _rlnVoltage #14 _rlnSphericalAberration #15 33.112 159.362 233.916 -19.83566 75.29693 -173.58894 00035.tomostar 10000.0 6.80200 6.0 warp_tiltseries/subtomo/00035/00035_0000000_6.80A.mrc warp_tiltseries/subtomo/00035/00035_0000000_ctf_6.80A.mrc 6.80200 300.000 2.700`

2D: `data_general

_rlnTomoSubTomosAre2DStacks 1

data_optics

loop_ _rlnOpticsGroup #1 _rlnOpticsGroupName #2 _rlnSphericalAberration #3 _rlnVoltage #4 _rlnTomoTiltSeriesPixelSize #5 _rlnCtfDataAreCtfPremultiplied #6 _rlnImageDimensionality #7 _rlnTomoSubtomogramBinning #8 _rlnImagePixelSize #9 _rlnImageSize #10 _rlnAmplitudeContrast #11 1 opticsGroup1 2.700 300.000 1.70050 1 2 4.00000 6.80200 64 0.070

data_particles

loop_ _rlnTomoName #1 _rlnTomoParticleId #2 _rlnCoordinateX #3 _rlnCoordinateY #4 _rlnCoordinateZ #5 _rlnAngleRot #6 _rlnAngleTilt #7 _rlnAnglePsi #8 _rlnTomoParticleName #9 _rlnOpticsGroup #10 _rlnImageName #11 _rlnOriginXAngst #12 _rlnOriginYAngst #13 _rlnOriginZAngst #14 _rlnTomoVisibleFrames #15 00051.tomostar 1 282.945 195.945 95.641 7100.961 5800.936 -159.281 00051/1 1 ../../../../../scratch/kjeldsen/warp_runs/untreated_full/series_0/warp_tiltseries/particleseries/00051/00051_6.80A_000001.mrcs 0.0 0.0 0.0 [1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]`

alisterburt commented 6 days ago

thanks @rkjensen ! Could you please reexport the 2D stacks on v2.0.0dev26 and let us know if things now run as expected in RELION? There was a double application of converting radians to degrees which Dimitry fixed in 65cd2ab50aba285d05e0e375e97b9b8b9dcd2b09 🙂

rkjensen commented 6 days ago

It is working! I directly ran a local refinement job after extraction (reference=None) and half maps after iteration 1 looks like ribosomes.

alisterburt commented 6 days ago

@rkjensen glad to hear it! Thanks for the quick turnarounds, much appreciated 🙂