search

jim-easterbrook / Photini

An easy to use digital photograph metadata (Exif, IPTC, XMP) editing application.
https://photini.readthedocs.io/
GNU General Public License v3.0
156 stars 24 forks source link

Crash when batch importing files #49

Closed aseques closed 3 years ago

aseques commented 3 years ago

I am importing a directory with a couple hundred of photos, photini works for a time (I set it to move the photos), but at the end it stops without any message. All the photos of the directory are moved apparently but I can't be sure. Photini version is current HEAD: c403b5f337cc2a78475419a6bb0a1ed2aa4c740f Kubuntu 20.04 with neon packages Python 3.8.2

Fatal Python error: Cannot recover from stack overflow. Python runtime state: initialized

Current thread 0x00007fd664b4f740 (most recent call first): File "/usr/lib/python3/dist-packages/gi/overrides/GExiv2.py", line 44 in open_path File "/usr/local/lib/python3.8/dist-packages/photini/exiv2.py", line 75 in init File "/usr/local/lib/python3.8/dist-packages/photini/exiv2.py", line 566 in init File "/usr/local/lib/python3.8/dist-packages/photini/exiv2.py", line 628 in open_old File "/usr/local/lib/python3.8/dist-packages/photini/metadata.py", line 1095 in init File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 63 in init File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 730 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied File "/usr/local/lib/python3.8/dist-packages/photini/pyqt.py", line 113 in wrapper File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 778 in show_thumbnail File "/usr/local/lib/python3.8/dist-packages/photini/imagelist.py", line 732 in open_file File "/usr/local/lib/python3.8/dist-packages/photini/importer.py", line 632 in file_copied ... Aborted

jim-easterbrook commented 3 years ago

It appears you're running out of memory. Each photo is opened in Photini as you import / move it, and it's generally a good idea to limit the number of photos you open at the same time. Try moving them in batches of 100 or fewer, closing the opened images after each batch.

jim-easterbrook commented 3 years ago

Looking at this again it might possibly be a recursion problem, if copying files is quicker than Qt's thumbnail generation and display.

jim-easterbrook commented 3 years ago

I think I've found the problem, but no solution yet. The importer runs in its own thread for maximum speed. After each image has been copied the thread sends a signal to the main thread to update the image area thumbnail display. During this process Qt's processEvents is called to keep the display updated, but that can also trigger processing of another signal from the importer thread, even though the previous one hasn't been fully processed yet.

jim-easterbrook commented 3 years ago

I think commit 0e18604 might be all that's needed to fix this.

aseques commented 3 years ago

@jim-easterbrook do you think that there are picutures that could have disapeared instead of moved? I could recover them from backup if needed, but I can't recall the original number of photos.

jim-easterbrook commented 3 years ago

I don't think so, but if the Python interpreter crashes in the middle of moving a file I don't know what could happen.

aseques commented 3 years ago

I'm closing this, there was a crash again, but I didn't have the verbose flag so I can't provide more information at this time, when running photini a second time it did't detect missing files, so I assume that it crashed after doing the work.

jim-easterbrook commented 3 years ago

I'm reopening it again. :)

I'm not totally sure my fix is robust and I really need to find a better way.

jim-easterbrook commented 3 years ago

I hope commit 1bcd9f7 is a better solution.

jim-easterbrook commented 3 years ago

My earlier mutex solution is not working for me since I updated my OS version. (It never was a nice solution.) In commit 0ce6148 I've implemented a much simpler solution which I hope will be more reliable.