difficult image for new ctfdisplay.py to do noise subtraction

[leginonbot]

---

Author Name: Anchi Cheng (@anchi2c) Original Redmine Issue: 3494, https://emg.nysbc.org/redmine/issues/3494 Original Date: 2015-08-31 Original Assignee: Neil Voss

---

This image attached has unusually high defocus and a bad drift correction. Cs=2.0 pixelsize=1.47 A

ctfestimate.py --runname=ctffindrunanchitest3 --rundir=/emg/data00/appion/acheng/15aug31a/ctf/ctffindrunanchitest3 --commit --projectid=1 --session=15aug31a --no-rejects --no-wait --continue --ampcarbon=0.15 --ampice=0.07 --fieldsize=512 --medium=ice --bin=2 --resmin=100 --resmax=10 --defstep=1000 --numstep=25 --dast=500 --expid=626 --jobtype=ctfestimate --mrclist=15aug31a_example1.mrc

Final params: def1: 5.81e-06 | def2: 5.93e-06 | angle: -52.5 | defratio 1.02
 ... Defocus Astig Percent Diff 1.02 -- 5.815e-06, 5.934e-06
Computing power spectra in 1024x1024 blocks...................................
 ... Computing median of power spectra series
 ... Compute PSD with fieldsize 1024 and 35 images complete in 20.99 sec
 ... Preform a rotational average and remove spikes...
PART 1: SETUP PARAMETERS AND ELLIPTICAL AVERAGE
 ... Number of available peaks is 250
 ... Determine and subtract noise model
PART 2: BACKGROUND NOISE SUBTRACTION
!!! WARNING: Not enough points (0) in section 3 to do background subtraction

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2016-02-15T16:58:55Z

---

Hi Anchi, Can I get the voltage for this micrograph, I want to include it in a test dataset.

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-02-15T19:40:31Z

---

200 kV, Cs = 2.0 mm

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2016-05-24T14:17:54Z

---

this one works for me now.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2016-05-24T14:34:12Z

---

close to and far from focus test images

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2017-04-13T01:03:16Z

---

This is still a problem. I am attaching a new example image 120 kV, 0.963 A/pixel, Cs 2.2 mm, Ctffind4 and Ctffind3 gave almost identical results. 0.79 um, 0.81 um, astig angle 36.3 and two 1d plots, from old code and new code.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-04-13T01:11:27Z

---

Hmm, I see that in the new plot it fit the curve just fine, but the generated CTF signal did not match. I will download the image and test it, probably Friday.

[leginonbot]

---

Original Redmine Comment Author Name: Bridget Carragher (@bcarr-czi) Original Date: 2017-04-13T01:21:34Z

---

Hi Neil, I thin this needs to be fixed ASAP or everyone will just not use it anymore and that would be sad. It is too slow and not giving good results now and so the users will vote with their feet and move on to something else. In my experience it is very hard to get them back after they leave! Let us know if you want to chat. Bridget

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-04-13T01:29:37Z

---

I have not changed anything to it in months, so what happened? There is a slight difference in the defocus values and at that close to focus it is often enough to shift the CTF oscillations off at high resolution, I see 0.786/0.808 um in the new and 0.784/0.812 um in the old.

I understand the need to be quick, but the CTF fitting is very touchy, so I do not like to make a modification that works with one image and then breaks the rest. I just got the kids to bed, so I will do some initial tests now, but unfortunately, my time is mostly booked for the next 24 hours. I have a quiz and 5 hours of lecture to prepare for tomorrow.

[leginonbot]

---

Original Redmine Comment Author Name: Bridget Carragher (@bcarr-czi) Original Date: 2017-04-13T01:34:54Z

---

OK, get those kids bedded down happily and don't fret too much! I am a big fan of your method and want to protect its use but it is not an emergency so do not lose any sleep over it or neglect those kids or that quiz. Let's just keep it on the high priority list and see what we can do to restore confidence.... B

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-04-13T01:46:37Z

---

Ignore this comment

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-04-13T01:49:06Z

---

never mind, I messed up the pixel size when I uploaded it

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-04-13T02:23:58Z

---

Something is wrong. According to the CTFFIND4 2d images it easily fits 7 Thon rings. Looking at my 2d images, it shows green for only 4 Thon rings, but the 2D image fits the 7 Thon rings despite them being red. When we look at the 1D graph it shows a different story.

The 1D graph and 2D graph are not in agreement. I am looking into this now.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-04-13T13:42:58Z

---

I just tested the current code on Joachim Frank's simulated images from the CTF challenge and the resolution went out to 2 Angstroms (max) for all of the images.

Are you sure that the pixel size and spherical aberration is correct for the image you uploaded? Should I try this on some similar images. Based on the fit it appears the things are getting out of sync once the spherical aberration term kicks in, which leads me to question the correctness of the pixel size and spherical aberration.

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2017-04-13T14:03:55Z

---

Pixel Size is correct. Your CTF estimation is way off. For this, the nominal defocus was 0.8 um. resmax I used was 3 Å, and resmin was also lowered to 30 Å, At such low defocus, defstep of 1000 is also not reasonable, so I was using 200. This is the typical setting for Phase plate data collection even though this one does not use phase plate, so I am not making it estimate phase shift. Low defocus, high resolution estimation is common now.

By the way, the CTFFind4 I used was 4.1.5

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-04-13T14:04:54Z

---

What was the CTFFIND 4 resolution it was reporting. I ran several run at low defsteps.

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2017-04-13T14:10:09Z

---

4.5 Å

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2017-04-13T14:12:06Z

---

Here is CTFFind4 output captured by Appion

Input image file name                              : 17apr12a_019en.mrc
Output diagnostic image file name                  : 17apr12a_019en-pow.mrc
Pixel size                                         : 0.963
Acceleration voltage                               : 120.0
Spherical aberration                               : 2.2
Amplitude contrast                                 : 0.15
Size of amplitude spectrum to compute              : 1024
Minimum resolution                                 : 30.0
Maximum resolution                                 : 4.27
Minimum defocus                                    : 2844.7
Maximum defocus                                    : 12844.7
Defocus search step                                : 200.0
Do you know what astigmatism is present?           : no
Slower, more exhaustive search?                    : no
Use a restraint on astigmatism?                    : yes
Expected (tolerated) astigmatism                   : 273.8
Find additional phase shift?                       : no
Do you want to set expert options?                 : no

Summary information for file 17apr12a_019en.mrc
Number of columns, rows, sections: 4096, 4096, 1
MRC data mode: 2
Bit depth: 32
Pixel size: 0.963 0.963 0.963
Bytes in symmetry header: 0

Working on micrograph 1 of 1
Estimated defocus values        : 8082.36 , 7862.55 Angstroms
Estimated azimuth of astigmatism: -53.71 degrees
Score                           : 0.14389
Pixel size for fitting          : 1.401 Angstroms
Thon rings with good fit up to  : 4.5 Angstroms
Did not detect CTF aliasing

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2017-04-13T14:13:45Z

---

and my appion script

ctffind4.py --ampcontrast=0.07 --fieldsize=1024 --resmin=30 --resmax=3 --defstep=0.02 --numstep=25 --dast=0.05 --min_phase_shift=10 --max_phase_shift=170 --phase_search_step=10 --runname=ctffind4run3 --rundir=/data/appiondata/17apr12a/ctf/ctffind4run3 --preset=en --commit --projectid=18 --session=17apr12a --best-images --no-wait --continue --bestdb --expid=9311 --jobtype=ctfestimate

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-04-13T15:09:15Z

---

my images

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-04-13T15:20:00Z

---

higher res ctffind4 fit. The shape of the curves is concerning to me, it gets very saw tooth. I have to stop right now, but I will look further.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-04-13T15:54:40Z

---

The plot thickens. I changed the pixel size of the image to 1.1 A (instead of 0.96 A). Ctffind4 still fits the data well:

Estimated defocus values        : 10379.04 , 10092.08 Angstroms
Estimated azimuth of astigmatism: -47.05 degrees
Score                           : .08995
Thon rings with good fit up to  : 4.7 Angstroms

[CTF run]   method: unknown | runname ctffind4run2
[CTF param] def1: 1.009 um | def2: 1.038 um | angle: 43.0 | ampcontr 0.15 | defratio 1.028
[CTF param]   additional phase shift: 0.0 degrees 
[CTF stats] conf_30-10: 0.954 | conf_5peak: 0.967 | res_0.8: 5.1A | res_0.5 4.8A

and now my system in agreement with the high resolution. The same effect can be achieved by adjusting the Cs value. So, I am not sure what to think.