Haystack `SearchIndex` gets image url but does not generate images

[FFX01]

Hi all!

So, I'm having somewhat of a strange issue here. I've tried looking around for a few days now for solutions, and I just can't seem to find any.

I am running django-imagekit in production to generate multiple sizes for user avatar images and such. My application has a completely decoupled backend that feeds data to a SPA via django rest framework. This works great!

My problem occurs within my search solution. I am using django haystack and elastic search to index all of my ~1,000,000 users including a downsized version of their avatar image generated with an ImageSpecField. I am storing the generated images in an Amazon S3 bucket. When I run the indexing operation, the correct urls for the generated images are generated. However, when I test the search functionality, it appears that the images do not exist! There is a 404 error for the image urls in the returned data from the search index. However, if I retrieve the user's data from my Db via django rest framework, the image will be found and displayed from the search index properly. So, it appears that the haystack SearchIndex class is generating the urls, but not triggering the image generation itself.

Now, I am 99% sure this is not a problem within django-imagekit. However, I have exhausted my options and I cannot afford the downtime required to reindex all of my data. I have also tried the generateimages management command, but that crashes because it runs out of memory due to the extremely large quantity of images.

The reason I am posting an issue here is because I cannot find a solution anywhere else and I figure that someone who works on this project might have some insight into how I can force the image generation without running into the issues listed above. Either that, or someone can point me in the right direction.

Here is some information about the server and the application:

Os: Ubuntu 16.04 LTS
Python: 3.5
Django: 1.10.3
django-imagekit: 3.3
django-haystack: 2.6.1

Here is a simplified version of my user model and my search index(all non problem specific fields have been redacted):

# User model
class User(AbstractUser):
    avatar = models.ImageField(
        upload_to="img/users/%Y/%m/%d",
        blank=True,
        null=True,
        default="default-avatar.jpg"
    )
    avatar_thumb = ImageSpecField(
        source="avatar",
        processors=[ResizeToFill(75, 100, upscale=True)],
        format="JPEG",
        options={
            "quality": 50
        }
    )
    avatar_small = ImageSpecField(
        source="avatar",
        processors=[ResizeToFill(150, 200, upscale=True)],
        format="JPEG",
        options={
            "quality": 80
        }
    )
    avatar_medium = ImageSpecField(
        source="avatar",
        processors=[ResizeToFill(300, 400, upscale=True)],
        format="JPEG",
        options={
            "quality": 95
        }
    )

# User Search index class
class UserIndex(indexes.SearchIndex, indexes.Indexable):
    avatar_small = indexes.CharField(model_attr="avatar_small")

    def get_model(self):
        return User

Any help or advice is greatly appreciated!

[vstoykov]

I see that you are not with the latest version of ImageKit. Can you upgrade to 4.0.2 and run generateimages again to see if memory problem will persist?

P.S. In 4.0.2 there is a fix leaking open files (#429) which can cause memory leak in Python 3.

[FFX01]

Thank you!

I will try that and report back.