CTFdisplayTool error

[leginonbot]

---

Author Name: Gabriel Lander (@gclander) Original Redmine Issue: 4994, https://emg.nysbc.org/redmine/issues/4994 Original Date: 2017-05-31 Original Assignee: Scott Stagg

---

We get this error when we run CTFfind4.py on our K2 images. None of the previous stats & images are saved, we just have the output from CTFfind4.

==== Committing data to database ==== ... Committing ctf parameters for 17may30d_021gr_02sq_01hl_02ed-a to database ... Reading image... [CTF param] def1: 1.635 um | def2: 1.677 um | angle: 76.9 | ampcontr 0.07 | defratio 1.026 [CTF param] additional phase shift: 0.0 degrees Final params: def1: 1.635 um | def2: 1.677 um | angle: 76.9 | defratio 1.03 ... Defocus Astig Percent Diff 1.29 -- 1.635, 1.677 um Computing power spectra in 2048x2048 blocks........ ... Compute PSD with 8 subfields and fieldsize 2048 complete in 10.3 sec !!! WARNING: Logging images failing CtfdisplayTool to /gpfs/group/em/appion/glander/17may30d/ctf/ctffind4runtest/failed_ctfdisplay_images.txt

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-05-31T18:11:12Z

---

I wonder if it is a out of memory error. Do all images fail or just some of them?

[leginonbot]

---

Original Redmine Comment Author Name: Gabriel Lander (@gclander) Original Date: 2017-05-31T18:12:29Z

---

every image, on the CCD data too. Don't think it can be memory, we're running on the exact same systems as before, only change was the myami/trunk update.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-05-31T18:15:03Z

---

Nothing has really changed in the apCtf folder (at least by me) since June 2016. Phase shifting does not work, or I have never tested it.

[leginonbot]

---

Original Redmine Comment Author Name: Gabriel Lander (@gclander) Original Date: 2017-05-31T21:20:58Z

---

okay it's nothing with the code, it seems to have something to do with the python finagling we did to get the K2 webserver display stuff working. what a mess.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2017-06-14T18:21:21Z

---

I'm having this problem too. We just updated our Python, numpy, scipy. Can y'all post what versions of those things y'all have? Here's my stuff. [sstagg@krios-0-1 ctffind4run4]$ python Python 2.7.5 (default, Nov 6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import scipy scipy.version '0.18.0' import numpy numpy.version '1.12.0'

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-06-14T20:26:21Z

---

Scott, I think it may be a problem with matplotlib, but I don't know. These are my test environments:

CentOS 6 has:

scipy-0.7.2-8.el6.x86_64.rpm
numpy-1.4.1-9.el6.x86_64.rpm
python-matplotlib-0.99.1.2-1.el6.x86_64.rpm

CentOS 7 has:

numpy-1.7.1-11.el7.x86_64.rpm
scipy-0.12.1-3.el7.x86_64.rpm
python-matplotlib-1.2.0-15.el7.x86_64.rpm

Debian 8.8 has:

>>> numpy.__version__
'1.8.2'
>>> scipy.__version__
'0.14.0'
>>> matplotlib.__version__
'1.4.2'

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-06-14T20:29:44Z

---

Wow, your versions are quite new. I was going to test Debian 9 when it comes out next week, but I do not want to install from sources it gets messy too quick. Maybe there is a problem with the latest versions of the code, perhaps related to python3 compatibility.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2017-06-14T20:36:00Z

---

Ugh. Our HPC just updated to CentOS 7. I have matplotlib 1.2.0. If I posted an 8K image here, could you test it on one of your systems? I can downgrade in a virtualenv if we can figure out a set of libraries that work.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-06-19T13:52:56Z

---

Hi Scott, yes I can test it on my CentOS 7 system. I have my docker setup that is pretty portable too.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2017-06-19T19:33:28Z

---

OK. I uploaded an image

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-06-19T20:14:37Z

---

Hi Scott, I need the Cs, pixel size, and voltage.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2017-06-19T20:15:51Z

---

Oh, yeah. 300 keV, 2.7 mm, 1.29 apix.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-06-20T15:45:01Z

---

Worked for me. Using Docker installed CentOS 7 on my Mac Book Pro, numpy/scipy versions above:

uploadImages.py -n scott1 -d 'scott test image' -p 1 -C --mpix 1.29e-10 --mag=60000 --kv 300 \
 --cs 2.7 --image-dir=/emg/sw/myami/appion/scott/ --defocus=-1.6e-6 --type=normal

run 1:

Total execution time : 42 seconds
2017-06-20 15:27:29  : CTFFind finished cleanly.
...
Computing power spectra in 4096x4096 blocks.........
 ... Compute PSD with 16 subfields and fieldsize 4096 complete in 37.87 sec
Using 9 of 9 subfields in final power spectra
...
[CTF param] def1: 3.100 um | def2: 3.117 um | angle: 83.1 | ampcontr 0.07 | defratio 1.006
[CTF stats] conf_30-10: 0.928 | conf_5peak: 0.939 | res_0.8: 8.4A | res_0.5 6.7A

run 2:

Total execution time : 3 minutes and 25 seconds
2017-06-20 15:34:42  : CTFFind finished cleanly.
...
 ... Compute PSD with 16 subfields and fieldsize 4096 complete in 35.0 sec
Using 9 of 9 subfields in final power spectra
...
[CTF param] def1: 3.101 um | def2: 3.111 um | angle: -73.1 | ampcontr 0.07 | defratio 1.003
[CTF stats] conf_30-10: 0.928 | conf_5peak: 0.938 | res_0.8: 7.9A | res_0.5 6.7A

run 3: changed field size to 2048 to match Scott's situation

Total execution time : 3 minutes and 18 seconds
2017-06-20 15:40:33  : CTFFind finished cleanly.
...
Computing power spectra in 2048x2048 blocks.................................................
 ... Compute PSD with 64 subfields and fieldsize 2048 complete in 43.85 sec
Using 48 of 49 subfields in final power spectra
...
[CTF param] def1: 3.098 um | def2: 3.108 um | angle: -76.1 | ampcontr 0.07 | defratio 1.003
[CTF stats] conf_30-10: 0.984 | conf_5peak: 0.985 | res_0.8: 6.8A | res_0.5 6.6A

Scott are you using trunk or version 3.2?

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2017-06-20T16:38:00Z

---

Trunk. I didn't update the field size, though. How did you do that?

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-06-20T16:40:10Z

---

We talked about this earlier, I thought:

appion/appionlib/apCtf/ctfpower.py, line 35:

old version:

twopowerint = int(math.floor(twopowerfloat))
fieldsize = 2**twopowerint

new version:

twopowerint = int(math.floor(twopowerfloat))-1
fieldsize = 2**twopowerint

alternative version:

twopowerint = 0 #override in next step
fieldsize = 2048

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2017-06-20T17:09:04Z

---

Sorry, you told me about it, but I hadn't gotten around to trying it. I'll try it now, and see if that fixes it.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2017-06-20T17:35:49Z

---

Still didn't work. Any other suggestions? I just updated trunk, and I'm using python 2.7.5 and

scipy.version '0.18.0' numpy.version '1.12.0' matplotlib.version '1.2.0'

... ctf estimation completed in 20.11 sec
 ... reading 17may11c_07gr_04sq_044hl_02ex-pow.txt
 ... reading MRC: 17may11c_07gr_04sq_044hl_02ex-pow.mrc size:(1024, 1024) dtype:float32
 ... writing JPEG: 17may11c_07gr_04sq_044hl_02ex-pow.jpg
 ==== Committing data to database ==== 
 ... Committing ctf parameters for 17may11c_07gr_04sq_044hl_02ex to database
 ... Reading image...
[CTF param] def1: 2.904 um | def2: 2.911 um | angle: -67.7 | ampcontr 0.07 | defratio 1.003
[CTF param]   additional phase shift: 0.0 degrees 
Final params: def1: 2.904 um | def2: 2.911 um | angle: -67.7 | defratio 1.00
 ... Defocus Astig Percent Diff 0.13 -- 2.904, 2.911 um
Computing power spectra in 2048x2048 blocks................................................
 ... Compute PSD with 48 subfields and fieldsize 2048 complete in 60.92 sec
!!! WARNING: Logging images failing CtfdisplayTool to /lustre/cryo/lustre/appiondata/17may11c/ctf/ctffind4run5/failed_ctfdisplay_images.txt
[CTF run]   method: unknown | runname ctffind4run5
[CTF param] def1: 2.904 um | def2: 2.911 um | angle: -67.7 | ampcontr 0.07 | defratio 1.003
[CTF param]   additional phase shift: 0.0 degrees 

Maybe you can suggest a way to get it to provide more info on the failure?

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-06-20T17:39:53Z

---

Anchi put it a grab all try/except, so we have to remove it to find the error.

appion/appionlib/apCtf/ctfinsert.py, line 109:

old

    try:
        ctfdisplaydict = ctfdisplay.makeCtfImages(imgdata, ctfvalues, fftpath, fftfreq)
    except:
        appendFailedImage(rundir, imgdata, ctfvalues,'makeCtfImages exception')
        return ctfvalues

move it out of the try and test

    ctfdisplaydict = ctfdisplay.makeCtfImages(imgdata, ctfvalues, fftpath, fftfreq)
    try:
        pass
    except:
        appendFailedImage(rundir, imgdata, ctfvalues,'makeCtfImages exception')
        return ctfvalues

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2017-06-20T17:53:15Z

---

Traceback (most recent call last):
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/bin/ctffind4.py", line 451, in <module>
    imgLoop.run()
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/appionLoop2.py", line 95, in run
    self.loopCommitToDatabase(imgdata)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/appionLoop2.py", line 153, in loopCommitToDatabase
    return self.commitToDatabase(imgdata)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/bin/ctffind4.py", line 380, in commitToDatabase
    ctfinsert.validateAndInsertCTFData(imgdata, self.ctfvalues, self.ctfrun, self.params['rundir'])
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/ctfinsert.py", line 48, in validateAndInsertCTFData
    ctfvalues = runCTFdisplayTools(imgdata, ctfvalues, opimagedir, fftpath, fftfreq)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/ctfinsert.py", line 109, in runCTFdisplayTools
    ctfdisplaydict = ctfdisplay.makeCtfImages(imgdata, ctfvalues, fftpath, fftfreq)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/ctfdisplay.py", line 1616, in makeCtfImages
    ctfdisplaydict = a.CTFpowerspec(imgdata, ctfdata, fftpath, fftfreq, twod=twod)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/ctfdisplay.py", line 1480, in CTFpowerspec
    powerspec, self.trimfreq = self.powerSpectra(image, ctfdata)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/ctfdisplay.py", line 1385, in powerSpectra
    genctfspec = genctf.generateCTF2dFromCtfData(ctfdata, adjusted_apix, self.volts, fieldsize)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/genctf.py", line 150, in generateCTF2dFromCtfData
    return generateCTF2d(focus1, focus2, theta, shape, mpix, cs, volts, ampconst, extra_phase_shift)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/genctf.py", line 179, in generateCTF2d
    radiisq = generateRadial2d(shape, xfreq, yfreq)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/genctf.py", line 322, in generateRadial2d
    r = Radial(halfshape, xfreq, yfreq, center=False)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/apCtf/genctf.py", line 302, in __init__
    self.radial = numpy.fromfunction(self.distance, shape, dtype=numpy.float64)
  File "/usr/lib64/python2.7/site-packages/numpy/core/numeric.py", line 2156, in fromfunction
    args = indices(shape, dtype=dtype)
  File "/usr/lib64/python2.7/site-packages/numpy/core/numeric.py", line 2094, in indices
    res = empty((N,)+dimensions, dtype=dtype)
TypeError: 'numpy.float64' object cannot be interpreted as an index

I've seen this error with other matplotlib operations. I thought my HPC had downgraded numpy to fix it. I'll ping them and update if I get a solution.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-06-20T17:57:43Z

---

It is an issue with numpy v1.12 or later. Looking into a fix.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2017-06-20T18:58:06Z

---

Fixed. Stupid error when the shape dimensions are not integers, see commit commit:8ef4b475

1. fix for numpy 1.12 or newer
shape = numpy.array(shape, dtype=numpy.uint16)
self.angular = numpy.fromfunction(self.arctan, shape, dtype=numpy.float64)

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2017-06-20T19:32:12Z

---

That fixed it on my end.