make ace2correct to correct for phase shift

[leginonbot]

---

Author Name: Anchi Cheng (@anchi2c) Original Redmine Issue: 4471, https://emg.nysbc.org/redmine/issues/4471 Original Date: 2016-09-26 Original Assignee: Philip Baldwin

---

phase plate phase shift needs to be included in order to make ctf correction that Appion pipeline need in 2D analysis

phase shift is defined in radian as the phase added to gamma (i.e., opposite of arcsin of amplitude contrast (0-1.0) ).

ctffind4 fits the phase shift and appion/makestack2.py output the ctf parameter result in a form that ace2correct understand as tmp_ctfvaluesfile.txt

For back compatibility purpose, please make sure it will run as if "gamma phase shift" is 0.0 if the line

    Gamma Phase Shift (radians): 0.442832

does not exists.

ace2correct.exe -ctf ./tmp_ctfvaluesfile.txt -apix 1.097 -img /gpfs/leginon/acheng/16sep22d/rawdata/16sep22d_a_00003sq_00036hln_00001hln_00002epn-a.mrc -out ./16sep22d_a_00003sq_00036hln_00001hln_00002epn-a.mrc.corrected.mrc

Example input ctf parameter file (tmp_ctfvaluesfile.txt) included

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-09-26T19:46:31Z

---

The source code is in myami/programs/ace2

ace2correct.m has parameter input and ctf.m has ctf equation

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-09-30T06:48:35Z

---

I edited ctf.m, ctf.h, ace2correct.m. There is another struct member called phase_shift throughout. If any parameter in the ctfvaluesfile.txt is not listed, then there are default values that are used. This is true now also for phase_shift (the default is zero).

Everything is compiling and running: on the other hand, I didn't do anything except create containers for the new value of phase shift. Now to alter ctf.m and the actual formula and check it is doing the correct thing...

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-09-30T07:18:32Z

---

In routine, ArrayP g2DCTF of function ctf.m

I added the following if statement as shown

---

f64 B = sqrt(1.0 - ampconst*ampconst);
f64 A = ampconst;// ensures A^2+B^2=1, and A cos g+ B sin g has maximum of 1;

if (phase_shift != 0){
    B = cos(phase_shift);
    A = sin(phase_shift);
}

---

So if the phase shift gets set, it ignores the amplitude-contrast parameter ampconst. The issue is that ampconst seems to be fixed, so phase_shift should take precedence in the cases where it is calculated

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-09-30T16:41:43Z

---

It does not make sense to abandom amplitude contrast when there is phase shift. ace2correct takes values from already estimated value which includes both terms. Your if statement will cause an offset in the correction, won't it ?

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-02T23:37:04Z

---

I will post some supporting figures, but the adjusted code seems to work pretty well. In the case where there is a calculated phase shift, it seems the program is just using this value to perform correction. I will explain method shortly.

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-03T08:08:02Z

---

In the attached jpeg, there is a blue squiggly curve which is drawn by means of the old (EMAN) ctfit program and is the Power Spectrum of the mrc file which Anchi asked me to look at. There is hardly any astigmatism in this image. The image is /gpfs/leginon/acheng/16sep22d/rawdata/16sep22d_a_00003sq_00036hln_00001hln_00002epn-a.mrc Let's call it "Uncorrected.mrc"

The version of ace2correct that I suggest creates a phase flipped version of Uncorrected.mrc: call it "Corrected.mrc"

I defined Added.mrc = (Corrected.mrc + Uncorrected.mrc )/2. Those frequencies correponding to phase flips should get zero'd out in Added.mrc. The power spectrum of Added.mrc is the red squiggly line. This is a log plot, so the region of frequencies where the phase flip occured, correspond to log(Added.mrc) dipping way down (the sudden drop offs). These sudden dives correspond closely to the CTF zeros, as they should. So everything seems to be in order.

There is one point, I haven't quite figured out: the correspondence of the value of the phase shift suggested by ctffind4 and employed by ace2correct, and the phase shift that I twiddled in EMAN2 to bring the solid smooth line into correspondence with the CTF oscillations of the Power Spectrum (squiggly blue line).

More later, PRB

[leginonbot]

---

Original Redmine Comment Author Name: Bridget Carragher (@bcarr-czi) Original Date: 2016-10-03T13:20:16Z

---

I am not sure I understand this. Are the red and blue (smooth) curves meant to match. How do we know the red curce is correct?

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-10-03T14:50:28Z

---

Regarding CTFFind4 output, Phase shift definition conversion is covered during the generation of the txt file I gave you already. It should be fine.

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-04T19:45:31Z

---

I looked again at the 1) outputs for ctffind4 2) the wrapper ctffind4.py. I have come to the conclusion that the correct way to interpret the ctffind4 outputs is to use A1= sin( arcsin(A0) + additional phase shift) in the program ace2correct.m .

Here A1 is the revised amplitude contrast to A0 (which is the 0.07) due to the "additional phase shift", that ctffind4 asks if you would like to find.

Notice if A0=0, then A1= sin(additional phase shift, about .45 radians). Alos if additional=0, then A1 = A0 (which is typically the 0.07)

This seems to be good to about 4 Angstroms. The ctffind4 program claims it is good to 3.5 Angstroms, but this seems a little bit optimistic.

Again, I check the answers as described in the attached pic/workflow. The Uncorrected image is corrected. The Corrected image is then added back to the uncorrected image, resulting in an image which has zero amplitude in frequency bands between the first and second CTF zeros, between the third and fourth zeros, etc (the regions where flipping occurred).

This is the only way I can think to see if the images are phase flipped in the correct manner.

This should be good enough for most programs in appion. We point out that Relion does not use CTF corrected stacks of 2d particles. So the detailed behavior of ace2correct will not affect the fine reconstructions done with Relion.

It was brought up on Monday to imitate exactly what Rado does in phase correction. I got various feedback from lab members: including i) Vlion, 2) using some of his own matlab code, and 3) that the new Relion2 does not necessarily need any assistance in interpreting ctffind outputs.

I think I have attended to the main issue here, and if others are satisfied will look to incoporate the simplied modifications to ace2correct.m and ctf.m.

Best, PRB

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-10-04T23:48:49Z

---

Hi, Phil, this will not work when additional phase shift + arcsin(A0) becomes larger than 90 degrees because the cosine term which traditionally is made by sqrt(1-sq(A0)) will not represent the behavior of a phase shift larger than 90 degree.

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-10-04T23:50:11Z

---

The cosine term is the B in ctf.m

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-05T05:26:29Z

---

Good point, Anchi. I had been too narrow minded with the data on hand.

Better is

ctf.m if (phase_shift != 0){ f64 phase_shift_New= phase_shift+ asin(ampconst); A = sin(phase_shift_New); B = cos(phase_shift_New); }

This will handle the case you point out, which I checked numerically as given above.

Shall I go ahead and commit changes to ace2correct.m and ctf.m ?

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-05T05:28:06Z

---

I also went a round of emails with Alexis Rohou (author ctffind4) who corroborated the approach here.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2016-10-05T19:28:14Z

---

How should we keep track of the amplitude contrast in the database. I think adding another parameters is overkill. Can we convert all amplitude contrast values to a single angle and then change the software around it to use the new format?

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-10-05T20:04:59Z

---

What is the new format ? I've already added extra_phase_shift in database so that amplitude contrast can stay the same.

Amplitude contrast is expressed the way it is for physical reason that it contribute to 7% increase in intensity when extrapolate to zero frequency. I don't think It is overkill to have phase shift as a different parameter in the database. We need to track how phase plate changes phase over time independent of using carbon or ice amplitude contrast.

In the ctf estimation programs, I believe the assumption is to keep amplitude contrast fixed which the user can enter from theory or measure by fitting a non-phase plate image of similar material. In the fit, it may be combined with phase shift form phase plate, but it will be separated out in the output.

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-05T20:19:28Z

---

I tend to agree with Anchi's point of view: having an extra_phase_shift. Mathematically amplitude contrast and phase shift cover the same information, but the way that ctffind4 relays the information, and the way that we think of it, is that there is the base amplitude contrast (the 0.07) and then the additional part due to the phase plate.

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-05T20:23:04Z

---

By the way, I did not mean to imply that I checked the scenario for phase plate data that had phase shifts greater than a quarter wave. I just meant that the new code compiled and ran correctly on the data that I had.

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-06T14:02:42Z

---

I tried looking at another micrograph that claimed a much higher phase shift (16sep22d 3sq 35hln 1hln 2epn), but it was hard to repeat the same calculation, because the CTF oscillations were so small in magnitude. I don't know how to reproduce the better looking (but also small) oscillations in the leginon/ACE panels. So, I will commit changes today

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-09T00:19:10Z

---

I have been trying yesterday to commit my changes to the trunk. I started with a fresh trunk today. Something won't work. Here is my sequence of commands

318 14:52 git clone http://pbaldwin@emg.nysbc.org/git/myami 335 14:58 git add programs/ace2/ctf.m 336 14:58 git add programs/ace2/ctf.h 337 14:59 git add programs/ace2/ace2correct.m 338 14:59 git commit -m "changes ace2correct.m and ctf.m to include phase plate line from ctffind4, refs #4471" 344 15:01 git pull 346 15:02 git push origin trunk

Here is the error

error: Cannot access URL http://pbaldwin@emg.nysbc.org/git/myami/, return code 22 fatal: git-http-push failed

Can someone help me? Thanks

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-10-09T15:53:11Z

---

Created an issue for Sargis as this is likely a server issue.

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-10T14:23:04Z

---

Ok, so ctf.m, ctf.h, ace2correct.m were correctly pushed to the trunk thanks to Anchi and Sargis.

Do I need to push the executable there? Or does somebody compile on the server?

Thank you, Philip

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-11T18:10:58Z

---

I just committed the executable ace2correct.exe also It is showing up on the repository correctly (REDMINE/projects/appion/repository/revisions/trunk/show/programs)

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-10-12T02:35:50Z

---

I tested ace2correct.exe on the test image with two phase shifts: Image 0. The uncorrected image Image 1. Corrected with the correct phase shift, phi Image 2. Corrected with phi + pi/2

When adding the corrected image 0 and 1, the power spectrum oscilates as Phil presented earlier (right half of the image I attach). When adding Image 1 and Image2, I get double the oscillation (left half of the attached image). confirming the phase shift correction is correct. See attached image.

A minor request: The program prints all other options it reads but not the extra phase shift. Could you make print out that as well so that it is more readable ? Thanks.

In addition, ace2correct.exe is also in myami/appion/bin. Please push your new version there as well.

[leginonbot]

---

Original Redmine Comment Author Name: Philip Baldwin (@prbprb2) Original Date: 2016-10-12T05:42:48Z

---

Ok, there is an extra phase shift line. I originally wrote such an output line, but removed it at some juncture.