memory error with some photos

[MrBovert]

With some fotos i got a memory error when uploading them: 2014-03-25 21:27:20,694 (<type 'exceptions.MemoryError'>, MemoryError(), <traceback object at 0x033A45D0>)

It happens when executing line 382: exiftags = exif.process_file(f)

Any idea what goes wrong? May be a mistake in the exif modul?

[richq]

Can you try with the latest, which removed the old embedded exif.py and requires the upstream python-exif package?

[MrBovert]

It took a while but now I have moved the exif routine to exifread. Unfortunately the error is the same. It happen reproducable always with the same images (from 2009). I would be thankful to get any help.

[richq]

Argh. This code looks pretty suspicious in exifread - https://github.com/ianare/exif-py/blob/master/exifread/classes.py#L175 - instead of reading in count and then splitting, it should read in bytes until a \x00 is found or count is reached, and have some limit like this ancient bug report says http://sourceforge.net/p/exif-py/bugs/6/

Any chance you could try the EXIF.py command line tool that comes with exifread to see if we can track down the bug in exifread and send them a patch?

Also what I can do is add a smaller try/except block in the uploadr.py code to mitigate the problem, get the file uploaded albeit without the extra exif info. Although a MemoryError might not be really catchable, and the whole python process could be broken at that point.

[MrBovert]

Thanks Richard for the quick answer. I will try and come back.

[MrBovert]

OK, back again. I used the debugger of IDLE and found out that it is exactly the suspicious code you pointed to in exifread which generates the memory error at my system.

I tried to limit the counter first to 2**30 and then to 1024 as described in the ancient bug report. In both cases the memory error has gone but the next error appears:

2014-04-09 20:13:17,928 (<type 'exceptions.TypeError'>, TypeError("object of type 'NoneType' has no len()",), <traceback object at 0x029A0418>)

Unfortunately I do not understand the code of dump_IFD so I stuck again. Seems to deal with the structure of EXIF so I think I can not leave it out.

A patch which leaves out the EXIF Information in case of an error while reading it would be a good workaround for me.

[richq]

Nice one. I'll see if I can write a patch for exifread too. Unfortunately (for you, not for me ;) I'm on holiday from tomorrow so won't be able to do much until I'm back.

I've pushed the try/except around the exifread code so the photo should upload now.

On 04/09/2014 09:01 PM, Martin wrote:

OK, back again. I used the debugger of IDLE and found out that it is exactly the suspicious code you pointed to in exifread which generates the memory error at my system.

I tried to limit the counter first to 2**30 and then to 1024 as described in the ancient bug report. In both cases the memory error has gone but the next error appears:

2014-04-09 20:13:17,928 (, TypeError("object of type 'NoneType' has no len()",), )

Unfortunately I do not understand the code of dump_IFD so I stuck again. Seems to deal with the structure of EXIF so I think I can not leave it out.

A patch which leaves out the EXIF Information in case of an error while reading it would be a good workaround for me.

— Reply to this email directly or view it on GitHub https://github.com/richq/folders2flickr/issues/21#issuecomment-40003222.

[MrBovert]

The try/except around the exifread first seems to work but then many funny things happend. Seems as if the python system was corrupted. So I limited the count in exifread/classes/dump_IFD again to 500 MByte and added an except TypeError in uploadr.py:

        f = open(image, 'rb')
        try:
          exiftags = exifread.process_file(f) #MG dive into memory error on some images
        except MemoryError:
            exiftags = {}
        except TypeError:
            exiftags = {}
        f.close()

With this construction all photos could be uploaded.

Thanks for the help and enjoy your holidays.

[richq]

MrBovert, any chance you could test my exif-py fork? I've got a hacked example file to test with, and I've sent ianare a pull request with a proposal to fix the problem so it should eventually get sorted

git clone https://github.com:richq/exif-py.git
cd exif-py
./EXIF.py /path/to/your/memoryerror.JPEG

That caused a crash previously with my test image, but after my fix I think it should work better.

[MrBovert]

Sorry for the delay. Unfortunately I deleted the list of fotos which where unloadable after my workaround worked. But I still have to load up some more photos. Doing this I will add a print to catch the MemoryEerror or TypeError and with it a foto which does not load up normaly. Coming back then.