All my pictures are gone

[sebastiaandegeus]

I ran this tool and stopped and continued later because of too many files and used the --trash option. Now after emptying the trash all my pictures are gone in the generated directories.

So the tool has created the entire tree of directories even with dates and locations but all directories are empty.

I even checked if it was working half way and it was fine. All pictures were there in the respective directories and now all are gone.

How is this possible?

--- checklist by @jmathai

• [x] Check for file existence when skipping duplicates and tests
• [x] Do not allow destination folder to be inside source folder and tests
• [x] Print statistics after completion and tests

[sebastiaandegeus]

Literally dozens of GB of pictures and videos have completely vanished.

[jmathai]

@sebastiaandegeus can you include the operating system and the exact commands you ran?

[sebastiaandegeus]

OS X El capitan 10.11.6

./elodie.py import --destination=/Users/sebas/Pictures/Organized /Users/sebas/Pictures --trash

It processed everything just fine. Even the .png files that could not be processed are still there. But everything else is gone in the source and destination directories.

[jmathai]

Thanks. Do you still have the output from when the script ran?

Can you also include command the result of an ls -lsah /Users/sebas/Pictures/Organized/{leaf folder}?

I did try it myself and after emptying the trash the files remained in the destination.

Python version.

~ python --version
Python 2.7.11

~ ./elodie.py import --destination=/Users/jaisen/Downloads/gh-145 /Users/jaisen/Downloads/samples --trash
/Users/jaisen/Downloads/samples/_DSC3872.JPG -> /Users/jaisen/Downloads/gh-145/2016-10-Oct/Unknown Location/2016-10-01_19-19-00-_dsc3872.jpg
/Users/jaisen/Downloads/samples/_DSC3873.JPG -> /Users/jaisen/Downloads/gh-145/2016-10-Oct/Unknown Location/2016-10-01_19-19-04-_dsc3873.jpg
/Users/jaisen/Downloads/samples/_DSC3874.JPG -> /Users/jaisen/Downloads/gh-145/2016-10-Oct/Unknown Location/2016-10-01_19-19-04-_dsc3874.jpg

After running script.

~ tree /Users/jaisen/Downloads/gh-145
/Users/jaisen/Downloads/gh-145
└── 2016-10-Oct
    └── Unknown\ Location
        ├── 2016-10-01_19-19-00-_dsc3872.jpg
        ├── 2016-10-01_19-19-04-_dsc3873.jpg
        └── 2016-10-01_19-19-04-_dsc3874.jpg

2 directories, 3 files

After emptying Trash.

tree /Users/jaisen/Downloads/gh-145
/Users/jaisen/Downloads/gh-145
└── 2016-10-Oct
    └── Unknown\ Location
        ├── 2016-10-01_19-19-00-_dsc3872.jpg
        ├── 2016-10-01_19-19-04-_dsc3873.jpg
        └── 2016-10-01_19-19-04-_dsc3874.jpg

2 directories, 3 files

[jmathai]

Another thought. Is it possible that the script was run, the photos in the destination folder were deleted, the trash was emptied and then the script was run again?

[sebastiaandegeus]

I've been testing the script and it seems that processing the same directory multiple times doesn't work most of the time. I keep starting the script and nothing happens. The only output that shows up are files like .DS_Store or .PNG that cannot be processed.

The .jpg files that are not processed get moved to trash with the --trash flag.

The result is catastrophic because files have been moved to the trash and none of the have been added to the destination directory.

[sebastiaandegeus]

python --version
=> Python 2.7.12

ls -lsah Organized/2016-01-Jan
total 16
 0 drwxr-xr-x   6 sebas  staff   204B Oct 29 16:50 .
 0 drwxr-xr-x  51 sebas  staff   1.7K Oct 29 11:21 ..
16 -rw-r--r--@  1 sebas  staff   6.0K Oct 29 16:50 .DS_Store
 0 drwxr-xr-x   2 sebas  staff    68B Oct 29 15:34 Amsterdam
 0 drwxr-xr-x   2 sebas  staff    68B Oct 29 16:39 Rotterdam
 0 drwxr-xr-x   2 sebas  staff    68B Oct 29 16:11 Unknown Location

tree Organized/2016-07-Jul
Organized/2016-07-Jul
├── Amsterdam
├── Bonn
├── Den\ Haag
├── Dordrecht
├── Nordrhein-Westfalen
├── Rotterdam
├── Unknown\ Location
└── Zuid-Holland

10 directories, 0 files

[sebastiaandegeus]

I haven't manually deleted anything in the generated directories. But I have emptied the trash. I'm not really sure it is actually related to the emptying of the trash. I think the script emptied the directories or removed them and recreated them while at the same time not processing the files at all except for moving everything to the trash where I deleted them.

[sebastiaandegeus]

It's 60 GB of files in many many directories that elodie created.

tree of all directories. https://gist.github.com/sebastiaandegeus/77999780e8d049a07913bc9990a49bbe

[phifogg]

If you process the same pictures over and over again to test (which I did as well :) ) it is normal if nothing happens. elodie will check the checksum of the pictures to verify if it has them already. there is a .elodie directory in your home dir, check the files in there. There is a hash.json which is the reason for this.

[jmathai]

@sebastiaandegeus I would try and follow the steps in this guide before using your computer much more. http://web.archive.org/web/20100524162402/http://www.squidoo.com/recovertrashmac

@phifogg is correct.

I think I may have recreated the issue on my machine. Let me continue to investigate but please follow the steps in the link above which suggests minimizing writes to the hard disk.

[sebastiaandegeus]

My destination directory was inside the source directory. Could the script with the --trash option processed also the destination directory and removed everything?

[jmathai]

My destination directory was inside the source directory.

I don't believe that is the issue. Give me some time to reply but for now focus on getting the files back.

[sebastiaandegeus]

I think the following happened:

1. script runs with destination directory inside source directory and trash option
2. script processes files and starts creating directories and puts the files in there and moves the originals to the trash like it's supposed to
3. script stores processed file hashes in the hash.json
4. script starts also processing the destination directory files (bug) 5?. script checks hash and sees the file has been processed before 5?. scripts actually processes the file and puts in the same place
5. script decides to move that file to trash
6. scripts keeps doing this until all files are in the trash.
7. I look in the trash this morning and conclude, hooray all files have been processed. Script is finished and empty the trash. But I'm stupid and forget to check the destination directory beforehand... (very big mistake on my part)

I'm not sure on step 5 but the result would be the same. Ignoring or processing would result in the same file in the same place and then it gets trashed.

I would like to add that I really love this tool regardless of what happened. I think I have most of my missing files in a backup.

[jmathai]

I think I have most of my missing files in a backup.

That's really all I care about right now. 👍

I will have a much more thorough look tonight. I think your explanation is accurate but there might be something else going on with the --trash option but I'll confirm that.

Ultimately, it should be much harder than this to shoot oneself in the foot.

[jmathai]

@sebastiaandegeus I think you're right that the destination folder being inside the source folder was part of what caused the problem you encountered. Precisely for the reason you explained.

I've added checklist of things to address for this issue. You hit a really nasty set of behaviors that are ultimately bugs because the library should know not do delete photos under these circumstances.

This uncovered some other potential bugs if the script were used in a way we hadn't considered. Feel free to add suggestions to the checklist.

Thanks and sorry about that.

[jmathai]

Closing this issue after checking off the todo list. Thanks for reporting and helping debug.