Open PetrochukM opened 1 year ago
PetrochukM
commented
1 year ago I think my options moving forward are:
Let me know if there is anything else I can do to help track this down!
RhetTbull
commented
1 year ago In investigating this, I also discovered that it wasn't adding the photos to a "missing" album.
Were you using the --add-missing-to-album option? That wasn't included on your command line above.
Try the following debug command:
osxphotos debug-dump --dump photos --uuid 6AED2C77-58FC-4104-B73F-7AC55749EB75 --uuid 8FF24096-4709-400F-B97C-2470B2F93DC2 > debug.txt
Then post the debug.txt here or send to me at rturnbull+git@gmail.com (and reference this issue #883 in the subject). Note: the debug output may contain data you consider sensitive such as location. You may want to look at the text file first and delete any data you don't want to share.
RhetTbull
commented
1 year ago One more bit of debug data that would be helpful:
Run the command osxphotos repl --missing
This will open an interactive osxphotos python prompt. At the prompt, type the following:
>>> for p in photos:
... print(f"{p.uuid}, {p._path_5()}, {os.path.exists(p._path_5())}")Type or paste the first line after the >>> then hit return and type the second line (it will ident automatically)
It should look something like this:
>>> for p in photos:
... print(f"{p.uuid}, {p._path_5()}, {os.path.exists(p._path_5())}")
8E1D7BC9-9321-44F9-8CFB-4083F6B9232A, /Users/rhet/Downloads/IMG_2000.JPG, False
A1DD1F98-2ECD-431F-9AC9-5AFEFE2D3A5C, /Volumes/MacBook Mojave/Users/Shared/Pumpkins4.jpg, FalseThis will print the expected path (where the original photo should be) for each missing photo and print out True if the file actually exists on disk or False if not.
PetrochukM
commented
1 year ago Done!! Thank you!
PetrochukM
commented
1 year ago In investigating this, I also discovered that it wasn't adding the photos to a "missing" album.
Were you using the --add-missing-to-album option? That wasn't included on your command line above.
Yeah, sorry, I tried it another time. I can confirm I tried it because an album was created, and some files were added. Unfortunately, those files above were not.
RhetTbull
commented
1 year ago Got the data -- thanks. These may be related to #884. The files are HEIC format ( 'originalFilename': 'IMG_4804.HEIC') but the UTI is listed as public.jpeg and Photos thinks the file on disk is a jpeg: 'filename': '6AED2C77-58FC-4104-B73F-7AC55749EB75.JPG',
Try the following command and send the output:
file ~/Pictures/Photos\ Library.photoslibrary/originals/6/6AED2C77-58FC-4104-B73F-7AC55749EB75*
PetrochukM
commented
1 year ago Okay, so, I tried file but that came back with "no matches found" so then I tried find, and then file again with the new path.
% file ~/Pictures/Photos\ Library.photoslibrary/originals/6/6AED2C77-58FC-4104-B73F-7AC55749EB75*
zsh: no matches found: /Users/michaelpetrochuk/Pictures/Photos Library.photoslibrary/originals/6/6AED2C77-58FC-4104-B73F-7AC55749EB75*
% find ~/Pictures/Photos\ Library.photoslibrary/ -name "6AED2C77-58FC-4104-B73F-7AC55749EB75*"
/Users/michaelpetrochuk/Pictures/Photos Library.photoslibrary//scopes/cloudsharing/resources/derivatives/masters/6/6AED2C77-58FC-4104-B73F-7AC55749EB75_4_5005_c.jpeg
/Users/michaelpetrochuk/Pictures/Photos Library.photoslibrary//scopes/cloudsharing/data/1370202095/8C86169C-ABEB-4AC2-BD73-82DE5FF85496/6AED2C77-58FC-4104-B73F-7AC55749EB75.JPG
% file ~/Pictures/Photos\ Library.photoslibrary/scopes/cloudsharing/data/1370202095/8C86169C-ABEB-4AC2-BD73-82DE5FF85496/6AED2C77-58FC-4104-B73F-7AC55749EB75*
/Users/michaelpetrochuk/Pictures/Photos Library.photoslibrary/scopes/cloudsharing/data/1370202095/8C86169C-ABEB-4AC2-BD73-82DE5FF85496/6AED2C77-58FC-4104-B73F-7AC55749EB75.JPG: JPEG image data, JFIF standard 1.01, resolution (DPI), density 300x300, segment length 20, Exif Standard: [TIFF image data, big-endian, direntries=14, manufacturer=Apple, model=iPhone 12, orientation=upper-left, xresolution=198, yresolution=206, resolutionunit=2, software=15.2, datetime=2021:12:31 22:27:03, hostcomputer=iPhone 12], baseline, precision 8, 1536x2049, components 3I also tried find with another image in the set. I am not sure why it says cloudsharing, I didn't share that photo, as far as I can recall.
% find ~/Pictures/Photos\ Library.photoslibrary/ -name "A472728D-021F-4315-87AB-DC770116C911*"
/Users/michaelpetrochuk/Pictures/Photos Library.photoslibrary//scopes/cloudsharing/resources/derivatives/masters/A/A472728D-021F-4315-87AB-DC770116C911_4_5005_c.jpeg
/Users/michaelpetrochuk/Pictures/Photos Library.photoslibrary//scopes/cloudsharing/data/1370202095/8C86169C-ABEB-4AC2-BD73-82DE5FF85496/A472728D-021F-4315-87AB-DC770116C911.JPGI think I've figured this out. All the affected photos are shared photos in a shared iCloud album (the old-style shared album, not iCloud shared library). The path for these photos has apparently changed -- in previous versions of macOS they were in Photos Library.photoslibrary/resources/cloudsharing/... but they're now in Photos Library.photoslibrary/scopes/cloudsharing/... and osxphotos wasn't aware of this so could not find the files. I've just recently gotten a machine with Ventura so hadn't tested this before. Now that I know where Ventura is storing the files I can implement a fix.
This is possible related to #626 but I'll need to check on a machine running with shared iCloud albums on Monterey (which I don't have access to).
PetrochukM
commented
1 year ago That's so fascinating! It sounds like we are close to a fix.
I am still curious how these ended up in a shared folder. These are pretty private photos that I wouldn't have any reason to share. They are also not in a shared album.
I also downloaded them and re-imported them, and it sounds like the shared state persisted. This could be connected to the earlier issue I experienced where Apple's cache persisted even after deleting and re-importing files.
I wonder if there is a way to clear the cache, so that database issues like this don't persist.
PetrochukM
commented
1 year ago OH, I figured it out. I have photos with the same exact name in a shared album, that was shared with me! I think I have been downloading and re-importing the wrong photos because the names were the same!
RhetTbull
commented
1 year ago In investigating this, I also discovered that it wasn't adding the photos to a "missing" album. That feature didn't work for these photos either which I thought was strange.
Shared photos cannot be added to regular albums unless they are imported to the library, hence these photos would not be added.
RhetTbull
commented
1 year ago @PetrochukM I believe this should be fixed in v0.55.7. Please upgrade (pipx upgrade osxphotos assuming you installed with pipx) and let me know if you're still experiencing the problem.
RhetTbull
commented
1 year ago @PetrochukM have you had a chance to try the latest osxphotos and if so, are you still experiencing this issue?
PetrochukM
commented
1 year ago Hey! I have been trying it, and I still have missing photos. I need to double-check what's happening; sorry for the slow response!
RhetTbull
commented
1 year ago Please run the debug dump again (substituting the uuid of one of the photos still missing, which you can get from the verbose output).
osxphotos debug-dump --dump photos --uuid 6AED2C77-58FC-4104-B73F-7AC55749EB75 --uuid 8FF24096-4709-400F-B97C-2470B2F93DC2 > debug.txt
PetrochukM
commented
1 year ago Okay, I took a look, and this is what is going on. I now have photos missing when two photos have the same filename and different file extensions. In this case, one photo is shared, and the other is not. The image which is shared, in this case, does not get exported.
This is only happening with 1 - 2 photos for me!
The photos are NOT in a shared album like we were looking at before; they were shared in iMessage and now are in my Photos.
RhetTbull
commented
1 year ago Please run the debug-dump command for a couple of the photos in question. You can get the UUID from the verbose output:
For example, in this line:
Exporting IMG_4804.HEIC (6AED2C77-58FC-4104-B73F-7AC55749EB75.JPG) (1/21)
The UUID is 6AED2C77-58FC-4104-B73F-7AC55749EB75
So use that in the debug dump command:
osxphotos debug-dump --dump photos --uuid 6AED2C77-58FC-4104-B73F-7AC55749EB75 --uuid 8FF24096-4709-400F-B97C-2470B2F93DC2 > debug.txt
For the last day or so, I have been unable to handle the missing photos issue.
Describe the bug
osxphotosis reporting that I am missing photos, even after... I tried:--retry 3 --download-missingosxphotosThis is the command I am running:
In investigating this, I also discovered that it wasn't adding the photos to a "missing" album. That feature didn't work for these photos either which I thought was strange. It did work for some other photos which I managed to resolve; however, for these photos it just didn't work at all.
To Reproduce
I am not sure how to reproduce it on another computer without uploading my entire library.
Expected behavior
These photos are present; I can download and re-import them. I am trying to figure out how to resolve this.
Desktop (please complete the following information):
OS: macOS Ventura Version 13.1 (22C65)
Additional context
Here is the verbose log: