search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
51.08k stars 2.7k forks source link

Preserve directory structure on backup #38

Closed ilkersigirci closed 1 year ago

ilkersigirci commented 2 years ago

Could you implement the backup mechanism to preserve folder structure?

-Consider I have  DCIM/Camera and Pictures/Telegram folders in the phone. -When doing backup, in the remote server, these photos are located as merged folder with random name like $UPLOAD_DIR/0268595a-45c0-4d09-af89-70ecb56b3c63/original/.... -But I want them to be same as the folder structure in my phone. -With this way, one can easily identify photos with their appropriate folder names.

comfreak89 commented 2 years ago

Yeah, its quite confusing when all photos are in the same folder - inclusive Telegram, Whatsapp, etc...

jbaez commented 2 years ago

I think albums #147 could be an alternative to replicating the folder structure in the server. That way from the client app, the photos can be organized in albums. Also, now that the "selective folders backup" feature has been implemented #137, there is no need of uploading all folders from the device.

Atemu commented 2 years ago

I'd also prefer if the file names were preserved.

Perhaps the files could be stored in $UUID/$FILENAME; i.e. original/80c75488857f94b5256f249925f73d5338f450db20c7fdd9f16f123e6dde9da0/08245964-3349-4469-a2f5-1d0e49122d94/actual-filename.jpg

Savasdotexe commented 2 years ago

I think albums #147 could be an alternative to replicating the folder structure in the server. That way from the client app, the photos can be organized in albums. Also, now that the "selective folders backup" feature has been implemented #137, there is no need of uploading all folders from the device.

With respects to what you'd like, I think an album is not the same or sufficient to "replicating" what the user here has asked for, which is preserving the folder structure, which I agree and I am a little surprised it doesn't do because the whole point of a backup is to restore and you can't do that if it's all changed.

An album can be created from videos/photos from various cameras and someone may prefer to have folders based on the source the photo/video was taken. So for example content from a digital camera and a smartphone camera are in two different directories, one can create an album from those, and tags can be based on subjects within the photo or video. When people start mixing up these structures, it kind of defeats the purpose.

alextran1502 commented 2 years ago

@Savasdotexe I agree with you. I've put more thought into this after the implementation of the selective backup features.

Since I can pull information from the asset's original folder from the mobile phone, it will also be nice to have them in the same structure on the server after backing up.

jbaez commented 2 years ago

I think albums #147 could be an alternative to replicating the folder structure in the server. That way from the client app, the photos can be organized in albums. Also, now that the "selective folders backup" feature has been implemented #137, there is no need of uploading all folders from the device.

With respects to what you'd like, I think an album is not the same or sufficient to "replicating" what the user here has asked for, which is preserving the folder structure, which I agree and I am a little surprised it doesn't do because the whole point of a backup is to restore and you can't do that if it's all changed.

An album can be created from videos/photos from various cameras and someone may prefer to have folders based on the source the photo/video was taken. So for example content from a digital camera and a smartphone camera are in two different directories, one can create an album from those, and tags can be based on subjects within the photo or video. When people start mixing up these structures, it kind of defeats the purpose.

🤔 Interesting point, but I see some issues with it.

For example, for uploading photos taken with a camera, that would need to be done from a future Web app, where you select photos from the file system to upload. In this case there is no "folder structure". The only way to have a folder structure would be to implement a Dropbox-like client to install in the computer (huge project on it's own, and Nextcloud already does this and it's great 😄), or to manually enter a folder destination when uploading.

The way I see the "backup" feature, is to have the photos from some folders in the phone backed up in case you lose the phone or your phone breaks. I'm not sure a feature to restore photos/videos to the original location in the phone would be a useful feature.

Another possible issue is having 2 mobile phones with the same folder structure. That would cause having a single folder structure for both devices with additional chance of name collision (if file names are also preserved)

I would normally have only my phone camera folder as auto-backup, and then maybe manually add a photo from WhatsApp (for example). This scenario is similar to the first example, I'm not sure you could get the folder structure in a "selected photos" context, and if it is possible, I think it might be unnecessary to create WhatsApp folder with those selected photos in the server.

I think the way the photos are organized in the server should be simple, performant and scalable. At the end of the day although the uploaded photos folder can be in a Docker bind mount to some folder in the NAS / Server (which would probably benefit from some folder structure) the normal way of using it in "production" would be having a Docker volume mount (more performance) or a Kubernetes volume (if running in K8), so the folder structure would not be easily accesible anyway.

Although a single folder would be "mixing up" photos from different devices, each photo should have metadata with its original name, camera used for taking the photo, etc. which could be used for filtering if needed.

Finally, I think one essential thing to have, is an export feature. This could export all the photos using the albums as folder structure and preserving the original names (like you would get by using Google export), so at least the organization done in the app using albums is not lost and you don't end up with a single folder with random photo names 😄

Savasdotexe commented 2 years ago

🤔 Interesting point, but I see some issues with it.

For example, for uploading photos taken with a camera, that would need to be done from a future Web app, where you select photos from the file system to upload. In this case there is no "folder structure". The only way to have a folder structure would be to implement a Dropbox-like client to install in the computer (huge project on it's own, and Nextcloud already does this and it's great 😄), or to manually enter a folder destination when uploading. ...

Well, those aren't really issues to what I said, besides a lot of external cameras sync with phones and tablets now days so they would still be using the same device.

Lets say you go on holiday, you takes pictures with your phone camera, your gopro and also digital camera (which both syncs photos/videos with the phone) and you also receives photos from a friend which went with you, via WhatsApp. You obviously would want a way to distinguish it from the others photos/videos and easily access it, so you would ideally create an album and lets say "Holiday trip 05/2022" and mark all those pictures into the album. You shouldn't be messing around with moving files, secondly if you did then WhatsApp app will not be able to locate the media files. Other applications may also have trouble finding its photos if it's not in the respected directories again. If you create copies then you're wasting storage.

Lets say something happens to your phone, but you had to factory reset losing all your videos/photos. The difference between folders and albums is evident here. All your photos are in the respected folders, an album is a collection for easy access. You could use albums to mimic folders, but it's just an extra step and misusing the term, plus upon restore potentially adding more work.

The way I see the "backup" feature, is to have the photos from some folders in the phone backed up in case you lose the phone or your phone breaks. I'm not sure a feature to restore photos/videos to the original location in the phone would be a useful feature.

The whole point of a backup is to potentially restore it exactly where it was so if you think it's not useful to do that, you're kind of missing the point. This is kind of unrelated to the topic here, but to answer you I presume that choice could be presented when you come to restore.

People prefer seamless operation, so backing up everything as you go is the way to go and having the user take steps to do this while it should be default is a bad call.

Another possible issue is having 2 mobile phones with the same folder structure. That would cause having a single folder structure for both devices with additional chance of name collision (if file names are also preserved)

Definitely a solvable issue.

spupuz commented 2 years ago

If having too many pictures in the same folder could lead to have problem ok listing them. Need to backup pic in year / month / day folder structure

Tschuuuls commented 2 years ago

Photo Sync does exactly this. Creates a Folder with device name, then subfolders for all the "Albums" on iOS/Folders on Android. It also preserves original Filenames. This behavior is pretty neat if you ever need to manually go in and look for a photo on an SMB share for example. Would be awesome, if Immich could work with those backups :)

ggantn commented 2 years ago

+1 for me.

ADIX7 commented 2 years ago

What about imageName from exif table and the current UUID? So something like originalName-UUID.originalExt for file name. Names collisions cant happen and every data is present currently so migration is not that hard.

Currently browsing is immich backup directory with samba/sftp/etc is not so convenient because the files are in a totally random order. Most phones and cameras put date and time to file names so they are order. Using only UUID in file names puts them in a random order. By using originalName-UUID format they would be mostly in date order and would be much more searchable.

alextran1502 commented 2 years ago

@ADIX7 I like your suggestion

Atemu commented 2 years ago

I also like it.

One thing to keep in mind is that a UUID string in the file name could make it long enough to cause issues on some platforms, some apps or file systems (namely Windows) under some circumstances.

This is an issues systems that use strong hashes in file names sometimes face. UUIDs are probably mostly fine but just something to keep in mind.

mammo0 commented 2 years ago

What about the suggestion of @Tschuuuls?

Creates a Folder with device name, then subfolders for all the "Albums" on iOS/Folders on Android. It also preserves original Filenames.

At the same time the feature request from #537 can be implemented, so a virtual album in Immich is also created.

My suggestion for the directory structure for uploaded photos from mobile devices:

<immich-assets-dir>/<immich-username>/<device-name-or-id>/<immich-album-id>-<immich-album-name>/<original-file-name>

I know that this does not preserve the original path like it's on the mobile device. But it's still human readable if you need to access the files directly e.g. via SMB.

But how should files be handled that were uploaded via the CLI tool or the web interface? My suggestion:

I don't know if a user can currently define an album to which the photos should be added after the upload.

I currently see two problems with my suggestion:

Pleas let me know your thoughts on this.

klaus1k commented 1 year ago

Hey, hope this is the right issue, didn't find a dedicated one on file names: I would love to have the photos backed up as they are - e.g. having their original filename preserved.

This would be perfect for my use case: <immich-assets-dir>/<immich-username>/upload/<original-file-name>

Is anything planned on this?

P.S. Love the project!

alextran1502 commented 1 year ago

@klaus1k Yes, the feature just got implemented https://github.com/immich-app/immich/pull/1098 it will be available on next release

klaus1k commented 1 year ago

Awesome, thanks!

ilkersigirci commented 1 year ago

Thank you for this update. Just awesome. Now can use immich as my main cloud photo storage for all my devices.