Improve full-migrate - Githubissues

lukehmcc commented 9 months ago

Improvements:

Requires user to specify output directory for fullMigrate. It will now no longer mess with the Takeout dir at all. No restructure, no folder creation.
Restructure step has been removed. Instead of organizing the folders and then running the migration code, the migration code is just called directly.
Handling files that need their metadata forcefully rewritten has been improved. It will no longer reprocess tens of thousands of photos just because a single one in that folder was malformed. This fixes #10.
Remove all sync calls so it can be better integrated into other code once it's listed on npm.
Updates exports & their naming.

mtalexan commented 9 months ago

The counter for the migration doesn't seem to be getting populated. After every directory it always lists 0 files processed, migrated, and failed.

Done! Processed 0 files.
Files migrated: 0
Files failed: 0

mtalexan commented 9 months ago

Also, as you iterate thru the Takeout folders, you're reprocessing the entire PhotosError folder after each one if there were any files added to it from the Takeout folder being processed. If there were photos added to the PhotosError folder during processing of the first Takeout folder, those then get re-processed on every subsequent Takeout folder processing that has errors as well.

I think you want to only process the PhotosError folder once after you've processed all the Takeout folders, don't you? If you're unable to fix any images in the PhotosError (which is likely), they'll just stay in that folder so you're probably better off just doing it once at the very end.

mtalexan commented 9 months ago

You don't appear to be filtering the file extensions when processing PhotosError either. It's trying to process JSON sidecar files when it tries to address the images in PhotosError, producing output like the following:

Cannot fix metadata for /output/PhotosError/20210918_123441.jpg.json as .json is an unsupported file type.
Cannot fix metadata for /output/PhotosError/20210918_123441.jpg(1).json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/20210918_123441.jpg
Cannot fix metadata for /output/PhotosError/20210918_120622.jpg.json as .json is an unsupported file type.
Cannot fix metadata for /output/PhotosError/20210918_120622.jpg(1).json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/20210918_120622.jpg
Cannot fix metadata for /output/PhotosError/20210627_135434_04.mp4.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/20210627_135434_04.mp4
Cannot fix metadata for /output/PhotosError/20210627_134405_10.mp4.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/20210627_134405_10.mp4
Cannot fix metadata for /output/PhotosError/2021-08-13 19.01.07.png.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/2021-08-13 19.01.07.png
Cannot fix metadata for /output/PhotosError/2021-08-08 10.30.36.png.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/2021-08-08 10.30.36.png
Cannot fix metadata for /output/PhotosError/2021-08-02 22.37.37.png.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/2021-08-02 22.37.37.png
Cannot fix metadata for /output/PhotosError/2021-07-26 22.42.40.png.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/2021-07-26 22.42.40.png

lukehmcc commented 9 months ago

The counter for the migration doesn't seem to be getting populated. After every directory it always lists 0 files processed, migrated, and failed.
Done! Processed 0 files.
Files migrated: 0
Files failed: 0

This is a known issue, I pushed the latest commit so @garzj could take a look at it as I was struggling to find the issue.

lukehmcc commented 9 months ago

You don't appear to be filtering the file extensions when processing PhotosError either. It's trying to process JSON sidecar files when it tries to address the images in PhotosError, producing output like the following:

Cannot fix metadata for /output/PhotosError/20210918_123441.jpg.json as .json is an unsupported file type.
Cannot fix metadata for /output/PhotosError/20210918_123441.jpg(1).json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/20210918_123441.jpg
Cannot fix metadata for /output/PhotosError/20210918_120622.jpg.json as .json is an unsupported file type.
Cannot fix metadata for /output/PhotosError/20210918_120622.jpg(1).json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/20210918_120622.jpg
Cannot fix metadata for /output/PhotosError/20210627_135434_04.mp4.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/20210627_135434_04.mp4
Cannot fix metadata for /output/PhotosError/20210627_134405_10.mp4.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/20210627_134405_10.mp4
Cannot fix metadata for /output/PhotosError/2021-08-13 19.01.07.png.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/2021-08-13 19.01.07.png
Cannot fix metadata for /output/PhotosError/2021-08-08 10.30.36.png.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/2021-08-08 10.30.36.png
Cannot fix metadata for /output/PhotosError/2021-08-02 22.37.37.png.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/2021-08-02 22.37.37.png
Cannot fix metadata for /output/PhotosError/2021-07-26 22.42.40.png.json as .json is an unsupported file type.
File already exists (you can ignore this): /output/PhotosError/2021-07-26 22.42.40.png

I think this is just confusing logging. It isn't processing *.json files. It tells the user when it doesn't process them which it probably shouldn't

for (let file of errFiles) {
      if (file.endsWith('.json')) {
        console.log(
          `Cannot fix metadata for ${file} as .json is an unsupported file type.`
        );
        continue;
...

lukehmcc commented 9 months ago

Also, as you iterate thru the Takeout folders, you're reprocessing the entire PhotosError folder after each one if there were any files added to it from the Takeout folder being processed. If there were photos added to the PhotosError folder during processing of the first Takeout folder, those then get re-processed on every subsequent Takeout folder processing that has errors as well.

I think you want to only process the PhotosError folder once after you've processed all the Takeout folders, don't you? If you're unable to fix any images in the PhotosError (which is likely), they'll just stay in that folder so you're probably better off just doing it once at the very end.

Should be fixed now. Pushed the error checking up the chain so it should only check after items have been processed.

garzj commented 8 months ago

The counter for the migration doesn't seem to be getting populated. After every directory it always lists 0 files processed, migrated, and failed.
Done! Processed 0 files.
Files migrated: 0
Files failed: 0

Yes, it was just the counter, I pushed a fix.

garzj / google-photos-migrate

Improve full-migrate #16