Avatars resizing (or uploading of the thumbs) does not work

[mitio]

We've received complaints that avatars don't show up after uploading.

It seems that the original image file gets correctly uploaded and pushed to the AWS storage.

Here is an example avatar: http://events.codeweek.eu/admin/avatar/avatar/106/ The original file is here and works: http://codeweekeu.s3.amazonaws.com/avatars/rakelsolva/rakel-solva.jpg The 80px thumb is missing (I suppose) from AWS: http://codeweekeu.s3.amazonaws.com/avatars/rakelsolva/resized/80/rakel-solva.jpg

When I test locally, avatar thumbs are properly generated.

On the live server I can't find any application logs or error logs and I was not able to activate them. Tried fiddling with the LOGGING and DEBUG settings in the various settings.py and restarting Apache, but to no avail. At one point a logfile appeared, but it remained empty. Because of this, so far I've got no idea what might be broken.

I see that avatars are processed using django-avatars. Not sure how they get to AWS.

The last avatar that worked was uploaded on Oct. 11, 2014, 12:59 p.m. All avatars uploaded after that date and time are "broken".

Since I'm not a Python & Django developer, I'd appreciate help on this. /cc @kerstin

[antitoxic]

This is how files get to s3: https://github.com/codeeu/coding-events/blob/master/codeweekeu/settings_production.py#L34

Next lines are also aws-related.

Docs:

• https://docs.djangoproject.com/en/1.6/ref/settings/#default-file-storage
• http://boto.readthedocs.org/en/latest/index.html

I don't see which package from https://github.com/codeeu/coding-events/blob/master/requirements.txt provides s3boto (neither s3boto nor django-storages are there)

[mitio]

Thanks to a few more hours of pain and a great deal of help from fellow life guru @antitoxic, I've managed to fix this. Will update the issue with more information in a bit.

[mitio]

TL;DR

A JPEG-processing library was not installed on the server and the django-avatar package was silently failing.

Details

Here is a longer explanation, as I believe it contains useful lessons for all of us–both for debugging similar problems as well as for how to code certain things.

So, here we go.

Avatars are being processed with the django-avatar package.

First, with the help of @antitoxic we tried to activate logging in the app. We disabled email notifications for errors and activated only file-based logging. We disabled RAVEN (Sentry) which is an error reporting service. We tried all kinds of stuff, even running a server directly via manage.py and accessing it via an SSH port forwarding, but we did not see any errors logged when uploading avatars.

We tried applying the same production settings locally (with S3 uploading enabled) and—guess what—it works locally. Thumbnails get created and successfully uploaded. Not in production, though.

I used the S3 credentials to open the S3 folder with a third-party app and to list the folder, to see if the thumbs were not being uploaded with a wrong filename. They weren't. There were only the original files there. No thumbnails.

Then I had a closer look at the django-avatar module. It's not a big one. I found out that the thumbnails are being created in Avatar.create_thumbnail(), in models.py.

These particular two lines caught my attention:

except IOError:
    return  # What should we do here?  Render a "sorry, didn't work" img?

That was a strong red flag–any IOErrors were being silently swallowed.

I prepared a simple test script to try to see if an IOError was being thrown.

The script essentially adds a new version of the create_thumbnail method without that except IOError clause, then tries to recreate the thumb of a certain existing problematic avatar.

I'm testing this script this way:

1. I SSH to the events server, then su - codeeu.
2. workon v1.1.14 to activate the Python virtual env.
3. cd ~/coding-events
4. curl https://gist.githubusercontent.com/mitio/135d3126bfb50806c242/raw/e8f37b2a4967560f60d68bb8112bbec5eade2e1f/test_create_thumbnail.py > test_create_thumbnail.py
5. ./manage.py shell
6. In the shell, just import test_create_thumbnail.

And, BAM!

IOError: decoder jpeg not available

A very telling and very important error message. A simple Google reveals a bunch of results for this, such as this StackOverflow question.

It turns out we're missing JPEG-processing capabilites.

And here's how to add them:

sudo apt-get install libjpeg-dev
pip uninstall pillow
pip install -r requirements.txt
sudo apache2ctl restart
./manage.py rebuild_avatars

The sudo commands can be run as root separately, but the rest should be run as codeeu in the loaded virtalenv.

And we're back in the game.

There are many lessons learned here. If I had to pick one, it's to reiterate on how bad it is to silently catch errors, especially if you're doing that in a library which other people are going to use.

Happy coding Tuesday.