Pillow 2.0.0 breaks django-imagekit and sorl-thumbnail

[legrostdg]

Here is the traceback... I'm trying to convert a jpeg file to a thumbnail with django-imagekit (I've also tried sorl-thumbnail: same issue). Downgrading to Pillow 1.7.2 fixes the issue.

Traceback (most recent call last):
  File "/home/user/test-env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/user/test-env/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/user/test-env/local/lib/python2.7/site-packages/django/views/generic/base.py", line 86, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/user/test-env/local/lib/python2.7/site-packages/django/views/generic/detail.py", line 110, in get
    return self.render_to_response(context)
  File "/home/user/test-repo/geodata/views.py", line 28, in render_to_response
    return self.get_json_response(self.convert_context_to_json(context))
  File "/home/user/test-repo/geodata/views.py", line 55, in convert_context_to_json
    if m.image.all() else None,
  File "/home/user/test-env/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 63, in _get_url
    self._require_file()
  File "/home/user/test-env/local/lib/python2.7/site-packages/imagekit/models/fields/files.py", line 41, in _require_file
    self.validate()
  File "/home/user/test-env/local/lib/python2.7/site-packages/imagekit/models/fields/files.py", line 50, in validate
    return self.field.image_cache_backend.validate(self)
  File "/home/user/test-env/local/lib/python2.7/site-packages/imagekit/imagecache/base.py", line 27, in validate
    file.generate(save=True)
  File "/home/user/test-env/local/lib/python2.7/site-packages/imagekit/models/fields/files.py", line 59, in generate
    save)
  File "/home/user/test-env/local/lib/python2.7/site-packages/imagekit/generators.py", line 61, in generate_file
    img, content = self.process_content(fp, filename, source_file)
  File "/home/user/test-env/local/lib/python2.7/site-packages/imagekit/generators.py", line 41, in process_content
    imgfile = img_to_fobj(img, format, **options)
  File "/home/user/test-env/local/lib/python2.7/site-packages/imagekit/utils.py", line 44, in img_to_fobj
    return save_image(img, StringIO(), format, options, autoconvert)
  File "/home/user/test-env/local/lib/python2.7/site-packages/imagekit/utils.py", line 249, in save_image
    img.save(outfile, format, **options)
  File "/home/user/test-env/local/lib/python2.7/site-packages/PIL/Image.py", line 1467, in save
    save_handler(self, fp, filename)
  File "/home/user/test-env/local/lib/python2.7/site-packages/PIL/JpegImagePlugin.py", line 557, in _save
    ImageFile._save(im, fp, [("jpeg", (0,0)+im.size, 0, rawmode)])
  File "/home/user/test-env/local/lib/python2.7/site-packages/PIL/ImageFile.py", line 452, in _save
    e = Image._getencoder(im.mode, e, a, im.encoderconfig)
  File "/home/user/test-env/local/lib/python2.7/site-packages/PIL/Image.py", line 395, in _getencoder
    return encoder(mode, *args + extra)
TypeError: function takes at most 11 arguments (13 given)

[aclark4life]

You mean downgrading to 1.7.8 (from 2.0.0) I assume? 1.7.2 is a really old Pillow release...

[aclark4life]

As for the error, looks like it could be a Py3 vs. Py2 issue... maybe. In which case, using 1.7.8 is probably the right thing to do until django-imagekit supports Python 3 .

[legrostdg]

@aclark4life I meant 1.7.8, you're right :-)

[matthewwithanm]

Hey all! I'm one of the people who works on django-imagekit.

@aclark4life Sorry for being dense, but what's making you think this could be a Python 3 issue? It looks like @legrostdg's using 2.7.

[aclark4life]

@matthewwithanm Hello! Just a guess. Pillow 2.0.0 now contains Python 3 support... so it's not necessarily going to work everywhere with code that only supports 2.x (then again, I haven't looked very closely, and I could be wrong.)

[legrostdg]

For reference: https://github.com/jdriscoll/django-imagekit/issues/193

[matthewwithanm]

@aclark4life Sorry, I'm not sure I understand what you're saying. Pillow 2's README says that it supports 2.6+, and it seems to work fine for me, so I believe it (: I don't see how other libraries' py3 support would affect anything.

I've dug into the differences between Pillow 1.7.8 and 2.0.0 a bit and I think that @legrostdg's and @rosscdh's problem is related to changes in the JPEG encoder. In 1.7.8, it was being passed 9 "extra" arguments whereas in 2.0, it's also being passed qtables and exif data. In both versions, the mode and rawmode are prepended to the argument list in Image._getencoder (which accounts for the "11" and "13" in the error). So I would guess that somehow they've wound up using the old shared library.

I'm not really sure how this would have happened. I would assume it has something to do with their systems or how they installed Pillow (and not anything about Pillow itself), but I don't know. Can you offer them any guidance? Or, if I'm way off base, let me know and I'll dig around some more.

[aclark4life]

@matthewwithanm We have a (roughly) related issue about detecting "bad" libjpeg support here: #169. Other than that, yeah sounds like it could be "user error".

[rosscdh]

@matthewwithanm Thanks for your effort Guys. Much appreciated. User error == pip install pillow? (Im pretty sure were both on macs? I wonder if we both used brew to install the appropriate libjepgs?)

On Wed, Apr 3, 2013 at 1:58 AM, Alex Clark ☺ notifications@github.comwrote:

@matthewwithanm https://github.com/matthewwithanm We have a (roughly) related issue about detecting "bad" libjpeg support here: #169https://github.com/python-imaging/Pillow/issues/169. Other than that, yeah sounds like it could be "user error".

— Reply to this email directly or view it on GitHubhttps://github.com/python-imaging/Pillow/issues/173#issuecomment-15810267 .

[matthewwithanm]

@rosscdh Can you try this?

import PIL
import _imaging
print PIL.__file__
print _imaging.__file__

It might be helpful to see if the imaging shared object is where you expect it to be.

[rosscdh]

Ill do that this evening when im on my A machine again :)

On Wed, Apr 3, 2013 at 7:18 AM, Matthew Tretter notifications@github.comwrote:

@rosscdh https://github.com/rosscdh Can you try this?

import PIL import _imaging print PIL.file print _imaging.file

It might be helpful to see if the imaging shared object is where you expect it to be.

— Reply to this email directly or view it on GitHubhttps://github.com/python-imaging/Pillow/issues/173#issuecomment-15818849 .

[legrostdg]

@matthewwithanm I'm using debian sid, and installed Pillow with pip, but... I also installed python-imaging with apt:

dpkg -l | grep python-imaging
ii  python-imaging                                 1.1.7-4                            amd64        Python Imaging Library

You're right, there is a problem somewhere:

print PIL.__file__
/home/user/test-env/local/lib/python2.7/site-packages/PIL/__init__
print _imaging.__file__
/usr/lib/python2.7/dist-packages/PIL/_imaging.so

How should I deal with this _imaging.so, which is taken system wide? Should I do a report on debian bug tracker?

[aclark4life]

@legrostdg apt-get uninstall python-imaging, you can't use both PIL and Pillow at the same time. And yes, I would ask Debian to switch python-imaging to Pillow.

[aclark4life]

@rosscdh By "user error" I mean some circumstance in the environment (i.e. beyond Pillow's control) causing Pillow to fail. #169 is about making Pillow smarter in such circumstances.

[legrostdg]

@aclark4life shouldn't Pillow in the virtualenv supersede PIL installed system-wide? I can't uninstall python-imaging, as it is used by other packages...

[aclark4life]

Perhaps, but I suspect it's currently not smart enough to handle anything but absolute clarity re: the environment. (i.e. Pillow is PIL so you have two competing installations). As for python-imaging, can you list a few of the packages that depend on it in Debian? Just curious…

[legrostdg]

I mean, why does python (run into a virtualenv) use /usr/lib/python2.7/dist-packages/PIL/_imaging.so instead of /home/user/test-env/local/lib/python2.7/site-packages/_imaging.so?

The list of reverse dependencies of python-imaging is pretty big:

apt-cache rdepends python-imaging
python-imaging
Reverse Depends:
  wsjt
  python-sympy
  python-skimage
  rapid-photo-downloader
  python-libtiff
  python-dicom
  openshot
  opensesame
  oboinus
  nml 
  nikola
  mobyle
  mcomix
  gwibber-service
  gwibber
  griffith
  gpsdrive-scripts
  glue-sprite
  exaile-plugin-contextinfo
  python-djblets
  python-circuits
  cinnamon
  calibre
  yade
  xpra
  xmms2tray
  xbmc
  wsjt
  python-weboob-core
  virtualbricks
  twms
  trac-wikiprint
  trac-odtexport
  trac-diavisview
  tilecache
  python-sympy
  svgtoipe
  streamtuner2
  python-stepic
  python-sphinx
 |python-sorl-thumbnail
  snowballz
  python-skimage
  simple-image-reducer
  python-seqdiag
  screenlets
  rst2pdf
  remuco-base
  rapid-photo-downloader
  python-qrtools
  python-pyx
  python-uniconvertor-dbg
  python-uniconvertor
  python-trml2pdf
  python-scrapy
  python-scipy
  python-reportlab
  python-qrencode
  python-imaging-tk
  python-imaging-sane
  python-imaging-doc
  python-imaging-dbg
  python-fabio
  python-enable
  python-docutils
  python-django-feincms
  python-aalib
  python-surfer
  pyicqt
  pyfai
  python-pyepl
  python-dicom
  python-captcha
  pyaimt
  psychopy
  plucker
  python-pgm
  photon
  photofilmstrip
  phatch-cli
  openshot
  oboinus
  python-nwdiag
  python-numm
  nml
  nautilus-image-manipulator
  python-django-mumble
  mobyle
  mitmproxy
  mediawiki-extensions-confirmedit
  mcomix
  lazygal
  last-align
  impressive
  ibid
  hplip
  python-guiqwt
  griffith
  grass-gui
  gourmet
  glue-sprite
  python-glitch
  python-gamera
  fretsonfire-game
  python-freevo
  fontypython
  fofix
  fgo
  exaile-plugin-contextinfo
  didjvu
  dicompyler
  cfv
  calibre
  python-blockdiag
  bkchem
  python-mpltoolkits.basemap
  advene
  python-actdiag
  python-aafigure
  python-pygpu
  xmms2tray
  wsjt
  wotsap
  tucan
  trac-wikiprint
  tpclient-pywx
  tilecache
  python-sympy
  python-stepic
  python-sphinx
  python-soya-doc
  snowballz
  screenlets
  rst2pdf
  remuco-base
  python-pyx
  python-uniconvertor
  python-uniconvertor-dbg
  python-trml2pdf
  python-sponge
  python-scrapy
  python-reportlab
  python-qrencode
  python-reportlab
  python-qrencode
  python-imaging-tk
  python-imaging-tk
  python-imaging-tk
  python-imaging-sane
  python-imaging-doc
  python-imaging-dbg
  python-docutils
  python-django-filebrowser
  python-django-djblets
  pymsnt
  pyicqt
  python-pyepl
  python-dicom
  pyaimt
  psychopy
  plucker
  python-plastex
  pkpgcounter
  python-pisa
  python-pgm
  photon
  phatch-cli
  openerp-server
  ocrfeeder
  python-django-mumble
  moovida-plugins-bad
  mnemosyne
  mediawiki-extensions-confirmedit
  mbot
  lazygal
  last-align
  python-jppy
  impressive
  ibid
  python-htmlgen
  hplip
  gwibber
  griffith
  gpsdrive-scripts
  gourmet
  python-gamera
  fretsonfire-game
  python-freevo
  fontypython
  fofix
  comix
  cfv
  calibre
  bkchem
  python-aafigure

[aclark4life]

No idea I'm afraid… perhaps check LD_LIBRARY_PATH

[legrostdg]

@aclark4life you recently closed the issue, but it is does not appear to be solved for me (as for other debian users that have python-imaging installed). I understand that it may be considered as a problem with virtualenv (as it does not isolate the env from the system installed libraries) or a debian specific problem (for not having switched from PIL to Pillow)... However debian wheezy is now released, and I don't think the switch could be done before the next release (in 2015?). What do you recommend? (I'll take care of doing a debian bug report to switch from PIL to Pillow very soon, sorry for not having done this yet.)

[legrostdg]

For reference, here is the bug report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=708148

[matthewwithanm]

An idea: perhaps _imaging could be kept compatible with PIL, and functions whose signatures have changed could be moved into a new, Pillow-specific module?

[aclark4life]

Long term _imaging has to go, the question is when will we drop support for _imaging and require: from PIL import Image.core as _imaging.

[wiredfool]

Short term it's going to go, as soon as I can fix the build/test time import issues.

[wiredfool]

And, additionally, I don't see us ever supporting a mixed install of PIL and Pillow. One or the other is fine, Mixing the two is a testing nightmare. There are already enough combinations of Python Version, OS, and 32/64 bittedness to handle without randomly replacing half the code in the middle of it and trying to have some semblance of an expected behaviour.

[aclark4life]

We would never support both in the same environment. If anything, we'd check to see if PIL is installed, remove it, then install Pillow in its place (perhaps similar to the way Distribute replaces Setuptools on install)

[matthewwithanm]

There are already enough combinations of Python Version, OS, and 32/64 bittedness to handle without randomly replacing half the code in the middle of it and trying to have some semblance of an expected behaviour.

Yeah, that's a good point. What about just renaming Pillow's _imaging to something different? It's a poor man's packaging but it would mean Pillow wouldn't be trying to load PIL's _imaging module.

[wiredfool]

The solution that we're(/I'm) working on is to put everything in the PIL namespace, so that if it's replaced, it's fully replaced. You can see it at #191, though it needs to be updated to include last weeks changes.

[aclark4life]

@matthewwithanm I think you are missing the big picture here. We are a PIL fork. We aren't trying to co-exist with PIL, we're replacing it. Pillow is PIL for all intents and purposes relevant here.

[matthewwithanm]

@aclark4life No, I get that, but coexisting in a single environment is different than coexisting on the same machine, isn't it? This just seems like it's going to be such a common problem and, honestly, I'd like to avoid a bunch of issues on my projects.

[aclark4life]

@matthewwithanm Co-existing means PIL in site-packages when someone installs Pillow. They'll both get installed, but the resulting environment will be unpredictable because both the PIL distribution and the Pillow distribution contain a PIL module, etc. The current fix for this "common problem" is to not install both.

[brokenrecord]

@legrostdg Reinstall (uninstall/install) PIL (i.e after Pillow 2.0.0) - problem solved.

! updated

[ncoghlan]

Also note that there's a bug in pkg_resources which can complicate co-existence when a virtualenv is configured to leave the system packages visible (rather than being completely isolated) and main.requires is used to control sys.path configuration: https://bitbucket.org/pypa/setuptools/issue/6/

[aclark4life]

@ncoghlan Good one, thanks