search

RhetTbull / osxphotos

Python app to work with pictures and associated metadata from Apple Photos on macOS. Also includes a package to provide programmatic access to the Photos library, pictures, and metadata.
MIT License
2.17k stars 100 forks source link

Crashed during export #703

Open persimmonsoft opened 2 years ago

persimmonsoft commented 2 years ago

Describe the bug I ran an export to a mounted shared folder on my NAS. During the export osxphotos crashed.

To Reproduce src if local Photos db and dst is shared folder on NAS

osxphotos export --cleanup --update --overwrite \ --exiftool --exiftool-option '-m' --exiftool-option '-F' \ --person-keyword \ --skip-original-if-edited --skip-live --skip-bursts --skip-raw \ --in-album --directory '{folder_album}'\ --db "$src" "$dst"

Output: 2022-05-23 06:38:21,925 - WARNING - export_db.py - 467 - disk I/O error 2022-05-23 06:38:21,926 - WARNING - export_db.py - 703 - disk I/O error 2022-05-23 06:38:21,926 - WARNING - export_db.py - 729 - disk I/O error Something went wrong and osxphotos encountered an error: disk I/O error Crash log written to '/Users/dave/osxphotos_crash.log' Please file a bug report at https://github.com/RhetTbull/osxphotos/issues with the crash log attached.

Expected behavior I expected the export to run to completion without error.

Desktop (please complete the following information): macOS - 10.15.7 osxphotos - 0.49.6 (latest at the time of running)

Additional context Crash log provided osxphotos_crash.log

RhetTbull commented 2 years ago

I'll take a look. In the meantime please try to run the export again and let me know if it still crashes. This appeared to be disk error during creation of the export database.

RhetTbull commented 2 years ago

These errors all appear to be disk I/O errors, specifically during creation of the export database. It is possible the NAS is flaky and causing intermittent errors. This kind of intermittent error is something that's difficult to test and reproduce. I just ran a test of an export where both export location and the Photos library were located on a NAS using 10.15.7, so as close to your config as I can make it, and everything worked as expected.

osxphotos includes automatic retry during export (--retry X) but it does not attempt to catch every possible disk i/o error as these should be rare and the operating system should handle them. If you have frequent disk i/o errors, you have bigger problems.

I am working on adding some additional retry to the export database code (#569) but in the mean time, you could try adding --retry 3 --ramdb to your export options. The first automatically retries an export if there's an error during export and the second loads the export database into RAM during the export then writes it to the export folder upon completion. This will significantly speed up export to a NAS as well as avoid issues with disk i/o during database read/write.

RhetTbull commented 2 years ago

In version 0.49.9 I added additional retry to all methods that read/write to the export database which should help in situations like this. I still recommend using --retry and --ramdb when writing to a NAS. You can upgrade with pipx upgrade osxphotos assuming you used pipx to install.

persimmonsoft commented 2 years ago

I did re-run it with 0.49.8 and it was successful. It did take a very long time, but I expect that may be my setup. I will investigate. Thanks for the help and the great tool!

oPromessa commented 2 years ago

Hey @persimmonsoft

  1. you can also complement with the use of --exportdb to have a local (and faster) DB instead of a remote (and slower) DB.Of course you have to be mindful you always have to use the --exportdb the same DB on subsequent re-exports.

  2. For performance with NAS, you can compare my results... on https://github.com/RhetTbull/osxphotos/issues/582#issuecomment-1053623946

persimmonsoft commented 2 years ago

Thanks oodles. I'll give that a try!

persimmonsoft commented 2 years ago

I tried the --retry 3 --ramdb approach and osxphotos crashed. I will go back to the --exportdb approach for now. Let me know if you need more info

Something went wrong and osxphotos encountered an error:
RetryError[<Future at 0x10ddcfc10 state=finished raised OperationalError>]
Crash log written to '/Users/dave/osxphotos_crash.log'
Please file a bug report at https://github.com/RhetTbull/osxphotos/issues with the crash log attached.
Exception ignored in: <function ExportDBInMemory.__del__ at 0x10a7e4160>
Traceback (most recent call last):
  File "/Users/dave/.local/pipx/venvs/osxphotos/lib/python3.10/site-packages/osxphotos/export_db.py", line 880, in __del__
    self.close()
  File "/Users/dave/.local/pipx/venvs/osxphotos/lib/python3.10/site-packages/tenacity/__init__.py", line 324, in wrapped_f
    return self(f, *args, **kw)
  File "/Users/dave/.local/pipx/venvs/osxphotos/lib/python3.10/site-packages/tenacity/__init__.py", line 404, in __call__
    do = self.iter(retry_state=retry_state)
  File "/Users/dave/.local/pipx/venvs/osxphotos/lib/python3.10/site-packages/tenacity/__init__.py", line 361, in iter
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x105d63250 state=finished raised AttributeError>]
Something went wrong and osxphotos encountered an error:
RetryError[<Future at 0x113637e20 state=finished raised OperationalError>]
Crash log written to '/Users/dave/osxphotos_crash.log'
Please file a bug report at https://github.com/RhetTbull/osxphotos/issues with the crash log attached.
Exception ignored in: <function ExportDBInMemory.__del__ at 0x110008160>
Traceback (most recent call last):
  File "/Users/dave/.local/pipx/venvs/osxphotos/lib/python3.10/site-packages/osxphotos/export_db.py", line 880, in __del__
    self.close()
  File "/Users/dave/.local/pipx/venvs/osxphotos/lib/python3.10/site-packages/tenacity/__init__.py", line 324, in wrapped_f
    return self(f, *args, **kw)
  File "/Users/dave/.local/pipx/venvs/osxphotos/lib/python3.10/site-packages/tenacity/__init__.py", line 404, in __call__
    do = self.iter(retry_state=retry_state)
  File "/Users/dave/.local/pipx/venvs/osxphotos/lib/python3.10/site-packages/tenacity/__init__.py", line 361, in iter
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x10b59f250 state=finished raised AttributeError>]

osxphotos_crash.log

RhetTbull commented 2 years ago

This crash is during open of the export database. It's possible the export database in the export directory (/Volumes/photo/Albums/.osxphotos_export.db) was corrupted the last time you tried to run osxphotos and it crashed or it's possible the database was never able to be opened in the first place. osxphotos will retry opening the database multiple times before crashing with this error so the fact this occurred means something is likely flaky with your external volume. I'd recommend using --exportdb as @oPromessa suggested but I'd also check the health of your external volume and/or network.