removing non interpolated from the output

[Vilax]

The non-interpolated output cannot be properly used in later processin steps

In a conversation with the developer, he thinks the interpolated output has enoght quality to work with it, and should be used.

The reason to remove the non-interpolated is the following one:

• The alignment is not correct, the in plane rotation per image is not provided by aretomo. Aretomo provides the same in-plane rotation for all tilt images. This in-plane rotation is the tilt axis orientation.

More modifications should be addressed in the future, perhaps in aretomo3. The current PR does not fix them, but enhance the user experience of aretomo:

• Currently, the CTF correction cannot be carried out. CTF should be estimated with the non-interpolated, but applied after the alignment. If the output is aligned the astigmatism is not rotated. If the CTF is corrected before aligning, the problem is fixed, but then the output will show +ali, !interp. This is because, the CTF correction applies the alignment. So this situation is misleading for the users.

• Perhaps, alignment, CTF and reconstruction should be independent protocols. This would be very versatile and more user friendly

[azazellochg]

Hi @Vilax, please see the comments below.

The non-interpolated output cannot be properly used in later processin steps

This is a serious claim, going against a few years of workflow testing. You would have to explain a bit more. If you could share the failing workflow it would be helpful.

In a conversation with the developer, he thinks the interpolated output has enoght quality to work with it, and should be used.

This goes against everyone else in the field including devs who wrote wrappers for aretomo for other packages. However, currently, nobody forbids users from using output interpolated TS.

The reason to remove the non-interpolated is the following one:

• The alignment is not correct, the in plane rotation per image is not provided by aretomo. Aretomo provides the same in-plane rotation for all tilt images. This in-plane rotation is the tilt axis orientation.

This is true, but alignment is correct and equivalent to etomo fine alignment with a single rotation solution.

More modifications should be addressed in the future, perhaps in aretomo3. The current PR does not fix them, but enhance the user experience of aretomo:

• Currently, the CTF correction cannot be carried out. CTF should be estimated with the non-interpolated, but applied after the alignment. If the output is aligned the astigmatism is not rotated. If the CTF is corrected before aligning, the problem is fixed, but then the output will show +ali, !interp. This is because, the CTF correction applies the alignment. So this situation is misleading for the users.

I'm not sure what CTF correction has to do with aretomo, it only (optionally) estimates CTF. Astigmatism rotation was discussed a few years ago and was fixed in the imod plugin.

• Perhaps, alignment, CTF and reconstruction should be independent protocols. This would be very versatile and more user friendly

This will give you 3x overhead of scipion + aretomo instead of a single command.

[Vilax]

Hi @azazellochg,

I know this is a serious claim, and I am happy you comment on this because it is a specially important topic.

I will reply step by step:

The non-interpolated output cannot be properly used in later processin steps

This is a serious claim, going against a few years of workflow testing. You would have to explain a bit more. If you could share the failing workflow it would be helpful.

If you look at the output of aretomo, the .aln file (or the .xf file if IMOD output is enable) you will see that all tilt images have associated the same rotation, but different shift. Using the non-interpolated output, if this alignment is applied with IMOD apply alignment the result will be misaligned. This is because the in-plane rotation is the same for all tilt images. This means that aretomo metadata of the non-interpolated is not interoperable with other Scipion protocols.

In fact, the users close to me use the interpolated output of aretomo.

In a conversation with the developer, he thinks the interpolated output has enoght quality to work with it, and should be used.

This goes against everyone else in the field including devs who wrote wrappers for aretomo for other packages. However, currently, nobody forbids users from using output interpolated TS.

This is correct, however, if you use the non-interpolated, later reconstructed tomograms will be artefactual due to the misalignment. Or if you enter in RELION/EMAN and you return to the tilt series, the predicted coordinates will fall out of the correct position. If your protein is small this incorrect position are significative.

The reason to remove the non-interpolated is the following one:

• The alignment is not correct, the in plane rotation per image is not provided by aretomo. Aretomo provides the same in-plane rotation for all tilt images. This in-plane rotation is the tilt axis orientation.

This is true, but alignment is correct and equivalent to etomo fine alignment with a single rotation solution.

I've not used etomo fine alignment in a long time, what happens with the interoperability of this alignment?

More modifications should be addressed in the future, perhaps in aretomo3. The current PR does not fix them, but enhance the user experience of aretomo:

• Currently, the CTF correction cannot be carried out. CTF should be estimated with the non-interpolated, but applied after the alignment. If the output is aligned the astigmatism is not rotated. If the CTF is corrected before aligning, the problem is fixed, but then the output will show +ali, !interp. This is because, the CTF correction applies the alignment. So this situation is misleading for the users.

I'm not sure what CTF correction has to do with aretomo, it only (optionally) estimates CTF. Astigmatism rotation was discussed a few years ago and was fixed in the imod plugin.

• Perhaps, alignment, CTF and reconstruction should be independent protocols. This would be very versatile and more user friendly

This will give you 3x overhead of scipion + aretomo instead of a single command.

The aretomo CTF estimation is really good, I use to run aretomo to take this estimation and continue withother reconstruction algorithm. In my opinion, to have all features together is an overcost of the protocol. On the other hand, I use to run aretomo in two steps, first I align, and later I reconstruct. I do this to ensure that my alignment is correct. I think this is more versatile, (just my opinon)

Anyway, this is an important PR/modification, I like to discuss it in detail

[pconesa]

Well, just to clarify... All works inside scipion. Here we are discussing about the quality or aretomo in a relion sta approach.

What we should do always is: prevent user from using "wrong data and metadata"

Now the question to decide is if aretomo alignment metadata alone is useful in at least some cases. Our conversation with Shawn made us think that it was not useful but, if @azazellochg has any case that works then we can keep the output.

[azazellochg]

That's getting interesting :) Has Shawn explained why he keeps adding options for output compatible with imod/warp/relion if such uninterpolated output is useless and "wrong"? He does not plan to fix "wrong" metadata instead?

[pconesa]

It is not that is wrong, but less optimal. He can't summarize in such a simple format local deformations/alignment that the interpolated tilt series suffers and is reflected in the tomogram reconstruction.

I think he recognize not to have success using the non interpolated ones. Maybe it is acquisition/dataset dependent

[azazellochg]

@pconesa @Vilax Thank you for your comments guys. We all know that even if aretomo could output local deformations we could not store them. So, I don't see a problem here - if you prefer the interpolated TS, use them. Or even better, use the reconstructed tomogram from aretomo. Remember that PPPT methods are incompatible with this approach - you always need the raw data.

@Vilax

IMOD apply alignment the result will be misaligned. This is because the in-plane rotation is the same for all tilt images.

I used empiar 10164 TS and have run:

• aretomo alignment + imod apply alignment + imod reconstruct
• aretomo align & reconstruct
• etomo manual reconstruction, solving for a single rotation

I don't see any problem, tomograms look good in all cases. If I solve for all rotations in etomo, the rotation angle varies by +/- 0.5 degrees. Of course, you could say that these TS are of very good quality.

In fact, the users close to me use the interpolated output of aretomo.

Great! So, if your users know that aretomo alignment is not good enough and interpolated locally-aligned TS are better, then keep using them :) Why would you want to remove the other functionality from the plugin?

Currently, the CTF correction cannot be carried out.

CTF correction is always complicated. Novactf does ctf correction -> apply alignment -> filtering -> reconstruction. Imod ctf correction protocol does rotate astigmatism since -xform is provided. I would assume novactf does the same. If the input TS are already aligned, then indeed astigmatism is not rotated when using either method. You could use tomo - astigmatism rotation protocol but since you say one rotation is not good enough, this will not be good enough either.
Maybe you could try the following workflow:

1. run aretomo alignment with ctf estimation and local refinement (but still get 1 rotation angle)
2. use ctf estimation from (1) and run novactf on raw TS and get ctf-corrected TS - no astigmatism rotation needed
3. run aretomo alignment again using ctf-corrected TS from (2) and aln file from (1). Here aretomo protocol allows you to specify protocol (1) as input.
4. use output interp TS from (3) for your own reconstruction method

If you enter in RELION/EMAN and you return to the tilt series, the predicted coordinates will fall out of the correct position. If your protein is small this incorrect position are significative.

I cannot comment on this as I have no experience with relion/eman tomo pipelines.

To summarize, have you discussed this with e.g. Relion devs? E.g. Alister Burt or others, since relion 5 tomo with aretomo wrapper has been released for quite a while. Have they seen this problem with one rotation in the downstream workflow?

[Vilax]

@azazellochg , @pconesa , @JorMaister

Sorry, for the delay in my response.

@pconesa @Vilax Thank you for your comments guys. We all know that even if aretomo could output local deformations we could not store them. So, I don't see a problem here - if you prefer the interpolated TS, use them. Or even better, use the reconstructed tomogram from aretomo. Remember that PPT methods are incompatible with this approach - you always need the raw data.

I agree about, we should. In the conversations we had about aretomo the recommendation was to use the interpolated one. The interpolation seems to be quite good. I recognize I have no experience in this regard about interpolation.

@Vilax

IMOD apply alignment the result will be misaligned. This is because the in-plane rotation is the same for all tilt images.

I used empiar 10164 TS and have run:

* aretomo alignment + imod apply alignment + imod reconstruct

* aretomo align & reconstruct

* etomo manual reconstruction, solving for a single rotation

I don't see any problem, tomograms look good in all cases. If I solve for all rotations in etomo, the rotation angle varies by +/- 0.5 degrees. Of course, you could say that these TS are of very good quality.

I think this is a particular case of this dataset with a good alignment or in-plane rotations. In some tests I've carried out the tomogram looks misaligned when this approach (* aretomo alignment + imod apply alignment + imod reconstruct) is carried out.

In fact, the users close to me use the interpolated output of aretomo.

Great! So, if your users know that aretomo alignment is not good enough and interpolated locally-aligned TS are better, then keep using them :) Why would you want to remove the other functionality from the plugin?

That is my position, I would keep the interpolated to avoid the non-interpolated case.

Currently, the CTF correction cannot be carried out.

CTF correction is always complicated. Novactf does ctf correction -> apply alignment -> filtering -> reconstruction. Imod ctf correction protocol does rotate astigmatism since -xform is provided. I would assume novactf does the same. If the input TS are already aligned, then indeed astigmatism is not rotated when using either method. You could use tomo - astigmatism rotation protocol but since you say one rotation is not good enough, this will not be good enough either. Maybe you could try the following workflow:

1. run aretomo alignment with ctf estimation and local refinement (but still get 1 rotation angle)

2. use ctf estimation from (1) and run novactf on raw TS and get ctf-corrected TS - no astigmatism rotation needed

3. run aretomo alignment again using ctf-corrected TS from (2) and aln file from (1). Here aretomo protocol allows you to specify protocol (1) as input.

4. use output interp TS from (3) for your own reconstruction method

Not sure if I understand... but does novaCTF provide CTF-corrected tilt series? The alternativy I see is to correct the CTF and then to align with aretomo

If you enter in RELION/EMAN and you return to the tilt series, the predicted coordinates will fall out of the correct position. If your protein is small this incorrect position are significative.

I cannot comment on this as I have no experience with relion/eman tomo pipelines.

To summarize, have you discussed this with e.g. Relion devs? E.g. Alister Burt or others, since relion 5 tomo with aretomo wrapper has been released for quite a while. Have they seen this problem with one rotation in the downstream workflow?

Looking at relion code, they attempt to undo the in-plane rotation of the tilt series alignment, but still, if your protein is small and your tomo is large, the predicted position for those proteins far from the tilt axis can be not fully correct or even incorrect. My feeling is that if the protein is more less correct, due to the ppt approach the software is able to center the particle, but when the protein is not small

[azazellochg]

Hi, you are right about novactf. It does make ctf-corrected stacks but these are multiple slabs corrected with a different defocus value. So your reconstruction program will have to accept this kind of input.

My concern for removing non-interpolated TS output is that many users might not like it. You can make a poll in discord :) Aretomo does not have a GUI, so several people use Scipion to run aretomo and produce imod/warp/relion-compatible output. Then they continue with their pipelines

[pconesa]

in thiscase if it is useful at least for some cases y vote for leaving them. Maybe, inform somehow about the "risk" of using the non interpolated one?

[Vilax]

I think the users should have the last word. I like the idea of a poll in tomography channel or in an email.