Programmatically `.save()` makes admin page broken

[elcolie]

Simplified traceback is

KeyError: 'small_thumbnail'  During handling of the above exception, another exception occurred: 
ValueError: I/O operation on closed file.

Internal Server Error: /admin/vouchers/voucher/
Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/django/db/models/options.py", line 566, in get_field
        return self.fields_map[field_name] KeyError: 'small_thumbnail'  During handling of the above exception, another exception occurred:  
        Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/django/contrib/admin/utils.py", line 276, in lookup_field     
        f = _get_non_gfk_field(opts, name)
    File "/usr/local/lib/python3.6/site-packages/django/contrib/admin/utils.py", line 307, in _get_non_gfk_field     
        field = opts.get_field(name)
    File "/usr/local/lib/python3.6/site-packages/django/db/models/options.py", line 568, in get_field     
        raise FieldDoesNotExist("%s has no field named '%s'" % (self.object_name, field_name)) django.core.exceptions.FieldDoesNotExist: 
            Voucher has no field named 'small_thumbnail'  During handling of the above exception, another exception occurred:  
        Traceback (most recent call last):
    File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner     
        response = get_response(request)
    File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 158,  in _get_response     
        response = self.process_exception_by_middleware(e, request)
    File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 156, in _get_response     
        response = response.render()
    File "/usr/local/lib/python3.6/site-packages/django/template/response.py", line 106, in render     
            self.content = self.rendered_content
    File "/usr/local/lib/python3.6/site-packages/django/template/response.py", line 83, in rendered_content     
        content = template.render(context, self._request)
    File "/usr/local/lib/python3.6/site-packages/django/template/backends/django.py", line 61, in render     
        return self.template.render(context)
    File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/base.py", line 63, in __call__     
        args, kwargs)
    File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/base.py", line 222, in call_if_sampling     
        return self.call(module, method, wrapped, instance, args, kwargs)
    File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/django/template.py", line 18, in call     
        return wrapped(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 175, in render     
        return self._render(context)
    File "/usr/local/lib/python3.6/site-packages/django/test/utils.py", line 98, in instrumented_test_render     
        return self.nodelist.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 943, in render     
        bit = node.render_annotated(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 910, in render_annotated     
        return self.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 155, in render     
        return compiled_parent._render(context)
    File "/usr/local/lib/python3.6/site-packages/django/test/utils.py", line 98, in instrumented_test_render     
        return self.nodelist.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 943, in render     
        bit = node.render_annotated(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 910, in render_annotated     
        return self.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 155, in render     
        return compiled_parent._render(context)
    File "/usr/local/lib/python3.6/site-packages/django/test/utils.py", line 98, in instrumented_test_render     
        return self.nodelist.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 943, in render     
        bit = node.render_annotated(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 910, in render_annotated     
        return self.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 155, in render     
        return compiled_parent._render(context)
    File "/usr/local/lib/python3.6/site-packages/django/test/utils.py", line 98, in instrumented_test_render     
        return self.nodelist.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 943, in render     
        bit = node.render_annotated(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 910, in render_annotated     
        return self.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 67, in render     
        result = block.nodelist.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 943, in render     
        bit = node.render_annotated(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 910, in render_annotated     
        return self.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/loader_tags.py", line 67, in render     
        result = block.nodelist.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 943, in render     
        bit = node.render_annotated(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 910, in render_annotated     
        return self.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/library.py", line 214, in render     
        _dict = self.func(*resolved_args, **resolved_kwargs)
    File "/usr/local/lib/python3.6/site-packages/django/contrib/admin/templatetags/admin_list.py", line 326, in result_list     
        'results': list(results(cl))}
    File "/usr/local/lib/python3.6/site-packages/django/contrib/admin/templatetags/admin_list.py", line 302, in results     
        yield ResultList(None, items_for_result(cl, res, None))
    File "/usr/local/lib/python3.6/site-packages/django/contrib/admin/templatetags/admin_list.py", line 293, in __init__     
        super().__init__(*items)
    File "/usr/local/lib/python3.6/site-packages/django/contrib/admin/templatetags/admin_list.py", line 212, in items_for_result     
        f, attr, value = lookup_field(field_name, result, cl.model_admin)
    File "/usr/local/lib/python3.6/site-packages/django/contrib/admin/utils.py", line 285, in lookup_field     
        value = attr(obj)
    File "/usr/local/lib/python3.6/site-packages/imagekit/admin.py", line 39, in __call__     
        'original_image': original_image,
    File "/usr/local/lib/python3.6/site-packages/django/template/loader.py", line 62, in render_to_string     
        return template.render(context, request)
    File "/usr/local/lib/python3.6/site-packages/django/template/backends/django.py", line 61, in render     
        return self.template.render(context)
    File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/base.py", line 63, in __call__     
        args, kwargs)
    File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/base.py", line 222, in call_if_sampling     
        return self.call(module, method, wrapped, instance, args, kwargs)
    File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/django/template.py", line 18, in call     
        return wrapped(*args, **kwargs)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 175, in render     
        return self._render(context)
    File "/usr/local/lib/python3.6/site-packages/django/test/utils.py", line 98, in instrumented_test_render     
        return self.nodelist.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 943, in render     
        bit = node.render_annotated(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/base.py", line 910, in render_annotated     
        return self.render(context)
    File "/usr/local/lib/python3.6/site-packages/django/template/defaulttags.py", line 313, in render     
        if match:
    File "/usr/local/lib/python3.6/site-packages/imagekit/cachefiles/__init__.py", line 131, in __bool__     
        existence_required.send(sender=self, file=self)
    File "/usr/local/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 178, in send     
        for receiver in self._live_receivers(sender)
    File "/usr/local/lib/python3.6/site-packages/django/dispatch/dispatcher.py", line 178, in <
        listcomp>     for receiver in self._live_receivers(sender)
    File "/usr/local/lib/python3.6/site-packages/imagekit/registry.py", line 53, in existence_required_receiver     
        self._receive(file, 'on_existence_required')
    File "/usr/local/lib/python3.6/site-packages/imagekit/registry.py", line 61, in _receive     
        call_strategy_method(file, callback)
    File "/usr/local/lib/python3.6/site-packages/imagekit/utils.py", line 166, in call_strategy_method     
        fn(file)
    File "/usr/local/lib/python3.6/site-packages/imagekit/cachefiles/strategies.py", line 15, in on_existence_required     
        file.generate()
    File "/usr/local/lib/python3.6/site-packages/imagekit/cachefiles/__init__.py", line 94, in generate     
        self.cachefile_backend.generate(self, force)
    File "/usr/local/lib/python3.6/site-packages/imagekit/cachefiles/backends.py", line 109, in generate     
        self.generate_now(file, force=force)
    File "/usr/local/lib/python3.6/site-packages/imagekit/cachefiles/backends.py", line 96, in generate_now     
        file._generate()
    File "/usr/local/lib/python3.6/site-packages/imagekit/cachefiles/__init__.py", line 103, in _generate     
        content.seek(0) ValueError: I/O operation on closed file.

requirements.txt

django==2.0.5
django-imagekit==4.0.2

utils.py

def give_voucher(userprofile: UserProfile, profile_created: bool):
    """
    Give the voucher to mobile_phone user if profile_created
    :param mobile_phone:
    :param profile_created:
    :return:
    """
    voucher_record = get_vouchered_people(userprofile.mobile_phone)
    logger.info(f"profile_created: {profile_created} voucher_record: {voucher_record}")
    if profile_created and voucher_record is not None:
        # New user give voucher. For integrity reason I have to use get_or_create()
        # No worry. It does not create authenticated user to the system
        # I intentionally put my identity to not freak out people
        admin, created = User.objects.get_or_create(username='admin', email='sarit@elcolie.com')

        # Config require 3rd parties therefore I decided to put it here rather than `base.py`
        bkk = timezone('Asia/Bangkok')
        expiry_date = datetime(2018, 5, 31, 23, 59, tzinfo=bkk)
        expiry_utc = expiry_date.astimezone(pytz.utc)

        # Use name from the given records XLSX since platform unable to supply firstname&lastname
        description = f"Thank you {voucher_record.get('name')} for your continued support of Hunter Poke! As a token of appreciation Hunter would like to give you a free 500Baht voucher to continue your healthy & delicious way of eating :)"
        company = Company.objects.get(name__icontains='hunter poke')
        voucher = Voucher.objects.create(
            amount=Money('500', 'THB'),
            expiry_date=expiry_utc,
            company=company,
            customer=userprofile.user,
            title=f"Hunter Poke VIP Customer Voucher",
            description=description,
            created_by=admin,
            updated_by=admin,
        )
        filename = str(uuid.uuid4()) + '.jpg'
        with open(str(settings.BASE_DIR('apps/vouchers/S__10838070.jpg')), 'rb') as file:
            data = File(file)
            voucher.image.save(filename, data, True)
            voucher.save()

        voucher.branches.set([branch for branch in company.branches.all()])
        logging.info(f"{voucher} has been created")

models.py

class Voucher(AbstractThumbnail, AbstractTimestamp):
    image = models.ImageField(default='uc.png', upload_to='vouchers')
    thumbnail = models.ImageField(default='uc.png', upload_to='voucher_thumbnails')
    small_thumbnail = ImageSpecField(source='image',
                                     processors=[ResizeToFill(settings.THUMBNAIL['small']['size_x'],
                                                              settings.THUMBNAIL['small']['size_y'])],
                                     format=settings.THUMBNAIL_FORMAT,
                                     options={'quality': settings.THUMBNAIL_QUALITY})
    medium_thumbnail = ImageSpecField(source='image',
                                      processors=[ResizeToFill(settings.THUMBNAIL['medium']['size_x'],
                                                               settings.THUMBNAIL['medium']['size_y'])],
                                      format=settings.THUMBNAIL_FORMAT,
                                      options={'quality': settings.THUMBNAIL_QUALITY})
    large_thumbnail = ImageSpecField(source='image',
                                     processors=[ResizeToFill(settings.THUMBNAIL['large']['size_x'],
                                                              settings.THUMBNAIL['large']['size_y'])],
                                     format=settings.THUMBNAIL_FORMAT,
                                     options={'quality': settings.THUMBNAIL_QUALITY})

admin.py

from django.contrib import admin
from imagekit.admin import AdminThumbnail
from reversion.admin import VersionAdmin

from poinkbackend.apps.vouchers.models import Voucher

@admin.register(Voucher)
class VoucherAdmin(VersionAdmin):
    small_thumbnail = AdminThumbnail(image_field='small_thumbnail')
    medium_thumbnail = AdminThumbnail(image_field='medium_thumbnail')
    large_thumbnail = AdminThumbnail(image_field='large_thumbnail')
    list_display = [
        'id',
        'amount',
        'expiry_date',
        'company',
        'customer',
        'order',
        'image',
        'thumbnail',
        'small_thumbnail',
        'medium_thumbnail',
        'large_thumbnail',
        'title',
        'description',
        'created_at',
        'updated_at',
        'created_by',
        'updated_by',
    ]
    list_display_links = ['id', 'amount']
    search_fields = ['title', 'description']

I confirm file is there and I can save the record with image When I use djangoAdmin page do save. The thumbnails are fine.

Problem: When I use utils.py to create my record. Django admin page will call thumbnails and raises IOError file not found

Question: Where am I wrong?

[vstoykov]

@elcolie can you edit your comment and fix new lines of the traceback because everything looks on one line and is very hard to read.

[elcolie]

@vstoykov I have updated my issue based on your comment. And I also add admin.py. Thank you for very fast response

[vstoykov]

@elcolie your traceback still does not look like python traceback. Look here for example how traceback should look like https://github.com/matthewwithanm/django-imagekit/issues/454#issuecomment-362672382

[elcolie]

@vstoykov Sorry to keep you waiting. First day I tried regex to align me text. Second day(this day) I give up and use multiple cursor. Please let me know if you need more source code to drill down the issue. I will provide you as fast as I can.

[vstoykov]

OK you are using utils.py to save your record when?

1. In the same request response cycle where you see the exception?
2. From some async task and the exception is shown every time you want to open the admin after that?

I need more context about how exaclty and when you are executing this code in your utils.py.

[elcolie]

@vstoykov I have update utils.py to the issue. Function will give the customer a voucher ONLY first request to the system. Because Django will create userprofile and issue the voucher. This process will be done only once because of profile_created will be True for first time otherwise False

[vstoykov]

If you put in your admin only one thumbnail for example in list_display to have only small_thumbnail (medium_thumbnail and large_thumbnail to be removed) did the problem continues to appear?

[elcolie]

@vstoykov No. The problem is gone when I use only single one not all three. Any ideas?

[vstoykov]

And when all of the thumbnails are shown and you change the image trough administration there is no problem right?

[elcolie]

@vstoykov Yes. You are correct.

[elcolie]

Any ideas?

[vstoykov]

DId you use some non standard storage like S3 from django-storages or any other 3th party storage for MEDIA_STORAGE?

[elcolie]

Yes. I did. They are as follows

'storages',
'collectfast'

[vstoykov]

Ok this is duplicate of #391

[elcolie]

I use solution from leonsmith in https://github.com/matthewwithanm/django-imagekit/issues/391 It works!