manualmasker.py got an recursion depth exceeded error from numpil.fromstring while saving mask to png format

[leginonbot]

---

Author Name: Anchi Cheng (@anchi2c) Original Redmine Issue: 4252, https://emg.nysbc.org/redmine/issues/4252 Original Date: 2016-06-10 Original Assignee: Anchi Cheng

---

Even though it is using the imagefile.arrayToImage function in apImage

  File "/home/acheng/myami/pyami/numpil.py", line 138, in fromstring
    return getattr(Image, getPilFromStringFuncName())(data,decoder_name, *args)
  File "/home/acheng/myami/pyami/numpil.py", line 138, in fromstring
    return getattr(Image, getPilFromStringFuncName())(data,decoder_name, *args)
RuntimeError: maximum recursion depth exceeded

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-06-11T00:04:24Z

---

Fix it by not check for attribute of Image itself since it may be linked to the extended module made in numpil.py Not sure why, but by checking attribute in a new image object instead solves the problem.

Gabe, Please check on your end that this will work also. This can be tested by running manualmasker. When a mask is added to a loaded image and you click "forward", the error will cause the program to crash without saving the mask png image.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2016-06-20T18:47:07Z

---

I thought I'd switch to redmine from email for my similar issue. I'm on trunk with numpil.py, and I'm still getting this error after svn updating.

manualpicker.py --runname=manrun1 --rundir=/lustre/cryo/lustre/appiondata/16jun20a/extract/manrun1 --commit -m 16jun20a_00006sq_00004hl_00011ex.mrc,16jun20a_00006sq_00004hl_00010ex.mrc,16jun20a_00006sq_00004hl_00009ex.mrc --projectid=1 --session=16jun20a --no-rejects --no-wait --continue --median=2 --lowpass=15 --highpass=0 --planereg --bin=4 --diam=200 --pixlimit=4.0 --shape=plus --expid=11321 --jobtype=manualpicker
  File "/panfs/storage.local/imb/stagg/software/myamiss/pyami/numpil.py", line 143, in fromstring
    return getattr(Image, getPilFromStringFuncName())(data,decoder_name, *args)
  File "/panfs/storage.local/imb/stagg/software/myamiss/pyami/numpil.py", line 143, in fromstring
    return getattr(Image, getPilFromStringFuncName())(data,decoder_name, *args)
  File "/panfs/storage.local/imb/stagg/software/myamiss/pyami/numpil.py", line 143, in fromstring
    return getattr(Image, getPilFromStringFuncName())(data,decoder_name, *args)
...
 File "/panfs/storage.local/imb/stagg/software/myamiss/pyami/numpil.py", line 131, in getPilFromStringFuncName
    im = Image.new('1', (1,1))
  File "/usr/lib64/python2.7/site-packages/PIL/Image.py", line 1801, in new
    return Image()._new(core.fill(mode, size, color))
  File "/usr/lib64/python2.7/site-packages/PIL/Image.py", line 462, in _new
    new = Image()
RuntimeError: maximum recursion depth exceeded

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-06-20T21:43:47Z

---

I don't know what to say. A work around is to remove the replaced definition in pyami/numpil.py and put a flag specific to sites:

For those have to use fromstring remove this at the end of numpil.py so it retain its original definition.

Image2.frombytes = fromstring

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-06-20T21:59:07Z

---

See above.

Alex, Image.new('1', (1,1)) is called because it need to verify which attribute to use. If you can find out from which module it is doing Image.fromstring from, it would help me finding out how the confusion got started. In the mean time, I think the code removal I mentioned above will work for you.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2016-06-20T22:05:41Z

---

What versions of Python, scipy, and PIL are you using? I recall the mention of pillow is that something that I need to install?

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-06-20T22:27:41Z

---

pillow is a fork of PIL that has replaces PIL for those use newer versions of python. You will either install the original PIL or pillow. The two should not coexist because in the python module import, they are called the same name.

I thought replacing the attributes by checking works, and it did work for us for manualpicker and manualmasker etc. but there are places that there is another import Image from PIL like in apProTomo2Aligner.py. Maybe it got confused again. It seems that in Alex's case scipy.misc.imsave uses PIL but could not distinguish the local numpil.py extended module from the original. I guess it is the problem of me saying

Image2 = Image

Neil, if you know how to distinguish PIL from pillow with a less harmful call like a version text, we should try that.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2016-06-21T14:08:07Z

---

Hi Anchi, do not forget we need to override tostring and tobytes as well.

Here is the hack I did on my machine:

vi site-packages/PIL/Image.py

    def fromstring(self, *args, **kw):
        raise Exception("fromstring() has been removed. " +
                        "Please call frombytes() instead.")

I changed it to

    def fromstring(self, *args, **kw):
        return self.frombytes(*args, **kw)

You have to do it in two locations...

Unfortunately, this is not a great solution for general users. Is there a way to overwrite the function in Appion? Just so annoying, but CentOS 6 still uses the classic PIL (last real update was Dec 2006), we could just force users to upgrade to pillow, it is pretty straightforward.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2016-06-21T14:42:57Z

---

I just upgraded to Pillow 3.2, and I'm getting a new error.

Traceback (most recent call last):
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/bin/manualpicker.py", line 733, in <module>
    imgLoop.run()
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/appionLoop2.py", line 88, in run
    results = self.loopProcessImage(imgdata)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/particleLoop2.py", line 101, in loopProcessImage
    self.peaktree = filterLoop.FilterLoop.loopProcessImage(self, imgdata)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/filterLoop.py", line 83, in loopProcessImage
    peaktree = self.processImage(imgdata, self.filtarray)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/bin/manualpicker.py", line 484, in processImage
    peaktree = self.runManualPicker(imgdata)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/bin/manualpicker.py", line 651, in runManualPicker
    self.app.panel.openImageFile(imgpath)
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/bin/manualpicker.py", line 63, in openImageFile
    self.setImage(image.astype(numpy.float32))
  File "/panfs/storage.local/imb/stagg/software/myamiss/leginon/gui/wx/ImagePanel.py", line 316, in setImage
    self.setNumericImage(imagedata)
  File "/panfs/storage.local/imb/stagg/software/myamiss/leginon/gui/wx/ImagePanel.py", line 404, in setNumericImage
    self.setBitmap()
  File "/panfs/storage.local/imb/stagg/software/myamiss/leginon/gui/wx/ImagePanel.py", line 191, in setBitmap
    wximage = self.numpyToWxImage(self.imagedata)
  File "/panfs/storage.local/imb/stagg/software/myamiss/leginon/gui/wx/ImagePanel.py", line 226, in numpyToWxImage
    imagedata = Image.fromstring("L", (w, h), normarray.tostring())
  File "/panfs/storage.local/imb/stagg/software/myami_python/lib/python2.7/site-packages/PIL/Image.py", line 2077, in fromstring
    "Please call frombytes() instead.")
Exception: fromstring() has been removed. Please call frombytes() instead.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2016-06-21T16:01:03Z

---

Alex threatened to assign this to me. HELP! I'm drowning in complaints right now. What can I do quickly to get this working?

[leginonbot]

---

Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2016-06-21T16:05:04Z

---

We need to get this fixed, but I cannot do any more work on it until I defend next week. Consider me a lame duck assignee until then!

[leginonbot]

---

Original Redmine Comment Author Name: Carl Negro (@carl9384) Original Date: 2016-06-21T16:34:30Z

---

As per this link https://github.com/TileStache/TileStache/issues/237

I uninstalled Pillow 3.2 with pip, and then reinstalled with ```sudo pip install Pillow==2.9.0

...and manual masker works on my machine now. Not sure how this will affect other programs. If we need other programs to use Pillow 3.2 I can write up a virtual environment for manual masker that'll use 2.9.0. 

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2016-06-21T16:35:56Z

---

Hi Scott, You can do my edit above or use Carl's system to install older version.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2016-06-21T17:05:22Z

---

OK, I did that, but now the recursion depth error is back.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2016-06-21T17:29:38Z

---

OK. I fixed it (I think) by putting in a check to see if the Pillow version is greater than 3. Please test it at NYSBC. If it barfs maybe you can tune my conditional at the end of numpil.py to something that works for y'all.

[leginonbot]

---

Original Redmine Comment Author Name: Carl Negro (@carl9384) Original Date: 2016-06-21T18:15:32Z

---

Scott's fix works on my machine. It was already working on my machine though, so we'll have to keep an eye on it.

[leginonbot]

---

Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2016-06-21T18:20:56Z

---

Appion-Protomo works again. Thanks Scott!

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2016-06-21T20:03:47Z

---

Thanks to all. Sorry for my absence. Rather hit the bed like a stone early

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2016-06-22T13:47:27Z

---

I just realized my conditional wasn't quite right for checking Pillow version. I made a small but significant change. Can y'all test again when you get a chance?

[leginonbot]

---

Original Redmine Comment Author Name: Gabriel Lander (@gclander) Original Date: 2016-08-02T01:53:07Z

---

Hi all, sorry I lost the thread on this - where does this bug stand? We're still getting a "fromstring" error in manualpicker.py:

Beginning Main Loop ... Pixel size: 0.637 ... reading filtered image from mrc file Traceback (most recent call last): File "/gpfs/home/glander/myami/appion/bin/manualpicker.py", line 733, in imgLoop.run() File "/gpfs/home/glander/myami/appion/appionlib/appionLoop2.py", line 88, in run results = self.loopProcessImage(imgdata) File "/gpfs/home/glander/myami/appion/appionlib/particleLoop2.py", line 101, in loopProcessImage self.peaktree = filterLoop.FilterLoop.loopProcessImage(self, imgdata) File "/gpfs/home/glander/myami/appion/appionlib/filterLoop.py", line 83, in loopProcessImage peaktree = self.processImage(imgdata, self.filtarray) File "/gpfs/home/glander/myami/appion/bin/manualpicker.py", line 484, in processImage peaktree = self.runManualPicker(imgdata) File "/gpfs/home/glander/myami/appion/bin/manualpicker.py", line 651, in runManualPicker self.app.panel.openImageFile(imgpath) File "/gpfs/home/glander/myami/appion/bin/manualpicker.py", line 63, in openImageFile self.setImage(image.astype(numpy.float32)) File "/gpfs/home/glander/myami/leginon/gui/wx/ImagePanel.py", line 322, in setImage self.setNumericImage(imagedata) File "/gpfs/home/glander/myami/leginon/gui/wx/ImagePanel.py", line 410, in setNumericImage self.setBitmap() File "/gpfs/home/glander/myami/leginon/gui/wx/ImagePanel.py", line 191, in setBitmap wximage = self.numpyToWxImage(self.imagedata) File "/gpfs/home/glander/myami/leginon/gui/wx/ImagePanel.py", line 226, in numpyToWxImage imagedata = Image.fromstring("L", (w, h), normarray.tostring()) File "/opt/applications/python/2.7.11/gnu/lib/python2.7/site-packages/PIL/Image.py", line 2077, in fromstring "Please call frombytes() instead.")

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2018-02-12T18:56:29Z

---

Got a report from an outside user who has Pillow 3.1.2 and had to downgrade to 2.9.0 on his appion 3.2. I thought I mention it here.