search

leginon-org / leginon-redmine-archive

1 stars 0 forks source link

ctf crash #4261

Open leginonbot opened 5 months ago

leginonbot commented 5 months ago

Author Name: Scott Stagg (Scott Stagg) Original Redmine Issue: 4261, https://emg.nysbc.org/redmine/issues/4261 Original Date: 2016-06-16 Original Assignee: Neil Voss

ctf estimation is crashing on bad images:

Defocus search range: 15000 A to 65000 A (1.50 to 6.50 um)
 ... running ctf estimation at Thu Jun 16 10:13:28 2016
/panfs/storage.local/imb/stagg/software/rhel61/usr/local/bin/ctffind64.exe
16jun15a_03sq_01hl_02ex.mrc
16jun15a_03sq_01hl_02ex-pow.mrc
2.7,300.0,0.07,50869.1470675,6.4,2
512,100.0,10.0,15000.0,65000.0,1000.0,500.0
 ... ctf estimation completed in 1 min 18 sec
 ... nominal=4.0e-06, bestdef=4.0e-06,
 ... def_1=-2.2e-07, def_2=-2.1e-07, astig_angle=78.9, cross_corr=0.005,

 ... reading MRC: 16jun15a_03sq_01hl_02ex-pow.mrc size:(512, 512) dtype:float32
 ... writing JPEG: 16jun15a_03sq_01hl_02ex-pow.jpg
 ==== Committing data to database ==== 
 ... Committing ctf parameters for 16jun15a_03sq_01hl_02ex to database
!!! WARNING: |def1| > |def2|, flipping defocus axes
!!! WARNING: Negative defocus values, taking absolute value
 ... Reading image...
[CTF param] def1: 0.213 um | def2: 0.217 um | angle: -11.1 | ampcontr 0.07 | defratio 1.019
Final params: def1: 0.213 um | def2: 0.217 um | angle: -11.1 | defratio 1.02
 ... Defocus Astig Percent Diff 0.92 -- 0.213, 0.217 um
Computing power spectra in 2048x2048 blocks.......
 ... Computing median of power spectra series
 ... Compute PSD with fieldsize 2048 and 7 images complete in 18.14 sec
 ... Preform a rotational average and remove spikes...
PART 1: SETUP PARAMETERS AND ELLIPTICAL AVERAGE
Resolution range for the data is 1/9.4A to 1/2.1A
 ... far from focus images, increasing number of extrema (100)
 ... far from focus images, increasing number of extrema (200)
 ... far from focus images, increasing number of extrema (400)
 ... far from focus images, increasing number of extrema (800)
 ... far from focus images, increasing number of extrema (1600)
 ... far from focus images, increasing number of extrema (3200)
 ... far from focus images, increasing number of extrema (6400)
 ... far from focus images, increasing number of extrema (12800)
 ... far from focus images, increasing number of extrema (25600)
 ... far from focus images, increasing number of extrema (51200)
 ... Number of available peaks is 4
 ... setting the number of sections to default: 6
 ... reducing the number of sections to: 1
 ... Determine and subtract noise model
PART 2: BACKGROUND NOISE SUBTRACTION
 ... starting peak extension
Peak Extension Complete in 0.52 msec
PART 3: ENVELOPE NORMALIZATION
 ... Setting last 183 values to 0.9 (test = 0.9)
 ... starting peak extension
Peak Extension Complete in 365.02 usec
PART 3B: BAND-PASS FILTER
PART 4: PEAK EXTENSION
 ... starting peak extension
Peak Extension Complete in 396.01 usec
 ... starting peak extension
Peak Extension Complete in 289.92 usec
PART 5: CTF FIT AND CONFIDENCE
/usr/lib64/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
/usr/lib64/python2.7/site-packages/numpy/core/_methods.py:71: RuntimeWarning: invalid value encountered in dou
ble_scalars
  ret = ret.dtype.type(ret / rcount)
1/30A - 1/10A confidence is nan (overfocus nan)
5 peak confidence is 0.254 (overfocus 0.095)
PART 6: CTF RESOLUTION LIMITS
!!! WARNING: Res calc: Negative values in first 10 values
!!! WARNING: Res calc: Negative values in first 10 values
!!! WARNING: Res calc: Negative values in first 10 values
!!! WARNING: Res calc: Negative values in first 10 values
Resolution limit is 6.27 at 0.8 and 5.60 at 0.5
PART 7: MAKE 1D PLOT SUMMARY FIGURE
/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/ctfdisplay.py:710: RuntimeWarning: inva
lid value encountered in log
  logenvelopdata = numpy.log(envelopdata)
 ... Saving 1D graph to file 16jun15a_03sq_01hl_02ex-plots.png
PART 8: NORMALIZE THE 2D IMAGE
 ... converting 1d arrays to 2d and normalizing
.................. ... 2d image normalization complete
 ... Resolution ring of the edge of the image 2.98
 ... Resolution ring of 25th valley 3.661
 ... Final resolution ring of 2D Plot: 5.034
PART 9: DRAW THE 2D POWERSPEC IMAGE
 ... Saving 2D powerspectra to file: 16jun15a_03sq_01hl_02ex-powerspec.jpg
[CTF run]   method: ctffind | runname ctffindrun3
[CTF param] def1: 0.213 um | def2: 0.217 um | angle: -11.1 | ampcontr 0.07 | defratio 1.019
[CTF stats] conf_30-10: nan | conf_5peak: 0.254 | res_0.8: 6.3A | res_0.5 5.6A

Found 0 ctf values
[ 0.836  0.888  0.32   0.594  0.22   0.336 -0.348 -0.23   0.331  0.391
 -0.496 -0.338 -0.249 -0.235 -0.096]
[ 0.832  0.402  0.595  0.327  0.157 -0.073 -0.296 -0.109  0.036 -0.3    0.08
 -0.041 -0.248 -0.355 -0.375]
[ 0.836  0.888  0.32   0.594  0.22   0.336 -0.348 -0.23   0.331  0.391
 -0.496 -0.338 -0.249 -0.235 -0.096]
[ 0.832  0.402  0.595  0.327  0.157 -0.073 -0.296 -0.109  0.036 -0.3    0.08
 -0.041 -0.248 -0.355 -0.375]
(5.0, 6.2651646649631321, 5.5994494661336889, 3.6610214449479703)
16jun15a_03sq_01hl_02ex-powerspec.jpg
Traceback (most recent call last):
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/bin/ctfestimate.py", line 460, in <module>
    imgLoop.run()
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/appionLoop2.py", line 95, in run
    self.loopCommitToDatabase(imgdata)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/appionLoop2.py", line 153, in loopCom
mitToDatabase
    return self.commitToDatabase(imgdata)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/bin/ctfestimate.py", line 362, in commitToDatab
ase
    ctfinsert.validateAndInsertCTFData(imgdata, self.ctfvalues, self.ctfrun, self.params['rundir'])
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/ctfinsert.py", line 72, in vali
dateAndInsertCTFData
    ctfq.insert()
  File "/panfs/storage.local/imb/stagg/software/myamiss/sinedon/data.py", line 390, in insert
    db.insert(self, **kwargs)
  File "/panfs/storage.local/imb/stagg/software/myamiss/sinedon/dbdatakeeper.py", line 253, in insert
    self._insert(newdata, force=force)
  File "/panfs/storage.local/imb/stagg/software/myamiss/sinedon/dbdatakeeper.py", line 291, in _insert
    raise InsertError(e.args[-1])
sinedon.dbdatakeeper.InsertError: Unknown column 'nan' in 'field list'
leginonbot commented 5 months ago

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2016-06-20T15:27:00Z

Hi Scott, any chance I could get the image, 16jun15a_03sq_01hl_02ex.mrc. I do not have any 0.2 um images to test.

But clearly something went wrong because the first zero of the CTF was not in the power spectra. The 30A - 10A confidence was invalid. I can put a check in, but without the image I cannot properly fix it.

leginonbot commented 5 months ago

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2016-06-20T16:59:27Z

It's a crap image. If it could fail nicely without crashing the run, it would be great. I've uploaded it.

leginonbot commented 5 months ago

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2016-06-21T21:47:38Z

I think you uploaded the wrong image. I need 16jun15a_00003sq_00001hl_00002ex.mrc (hl 1 instead of hl 2).

Also, I calculated the following values:

Cs=2.7
pixelsize=1.26 A
voltage=300 kV
leginonbot commented 5 months ago

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2016-06-22T13:50:45Z

Whoops. Sorry. Yes, those values are correct.