Follow symlinks when importing photos

[jimenezrick]

Hi @xemle,

I was wondering if it would be possible to make home-gallery follow symlinks when importing photos from sources. I saw this setting here where it's disabled:

https://github.com/xemle/home-gallery/blob/b78fce7152e99498b7137411b079d0a8757d8ac4/packages/cli/src/tasks/import-sources.js#L197

TL;DR: My photo archive is setup in a way that each photo file is a symlink to a central storage (because I use https://git-annex.branchable.com/ for redundant backups which uses symlinks).

Or if you don't want to make it a default behavior, maybe control by a yaml setting?

Thanks again @xemle for this great piece of software. I has been working flawlessly for me!

[xemle]

Hi @jimenezrick

thank you for your input. In #20 this topic was mentioned earlier. Your mentioned code reference is just for the file watcher to trigger the import with a file index update. The real change needs to be done in https://github.com/xemle/home-gallery/blob/master/packages/index/src/walker.js#L25 as stated in #20. Up to now I did not find the time to implement that.

If possible raise a PR. I would suggest to add a boolean source option followSymlink which than influence the file walker behavior and the file watcher.

According to your GitHub profile it seems that you might be able to implement that by your own. What do you think as little weekend challenge? Would that be fun or a burden?

TL;DR: My photo archive is setup in a way that each photo file is a symlink to a central storage (because I use https://git-annex.branchable.com/ for redundant backups which uses symlinks).

OK. Thank your for your context. That helps to understand your need.

BTW: Do you know borgbackup? I am using it and like it by its awesome feature list.

Thanks again @xemle for this great piece of software. I has been working flawlessly for me!

Thank you for your flowers. Happy that HomeGallery helps you.

[jimenezrick]

Yeah, no worries, I had a quick look at the code.

From your comments in https://github.com/xemle/home-gallery/issues/20 I agree is a simple as switching from lstat() to stat(). Personally I wouldn't add a config setting for this, on Posix platforms (e.g. Linux), you typically want to follow symlinks no matter what, unless there is a reason not to. I saw in other parts of the codebase stat() is always use instead. The most important check here https://github.com/xemle/home-gallery/blob/master/packages/database/src/build.js#L24 will works just fine with stat() (after following the symlink we can filter regular files instead of directories).

Raising a tiny PR for this if you are ok with this.

[jimenezrick]

BTW: Do you know borgbackup? I am using it and like it by its awesome feature list.

Yeah, had a look at borg in the past. I just chose git-annex because it's based on git (and i use git for pretty much everything). But the nice thing is that it uses git for the metadata and it puts large files on the side (hence it uses symlinks heavily to keep your folders structure intact even if the files are stored in a different place). It also supports pretty flexible replication mechanism across disks/S3.

[xemle]

Raising a tiny PR for this if you are ok with this.

Thank you for the PR. I hope you are right with:

you typically want to follow symlinks no matter what

I am a bit uncertain. But let's try out. I do not use symlinks so it does not affect me and it helps you. Good job!

[xemle]

BTW: Do you know borgbackup? I am using it and like it by its awesome feature list.

Yeah, had a look at borg in the past. I just chose git-annex because it's based on git (and i use git for pretty much everything). But the nice thing is that it uses git for the metadata and it puts large files on the side (hence it uses symlinks heavily to keep your folders structure intact even if the files are stored in a different place). It also supports pretty flexible replication mechanism across disks/S3.

Thank you for your details and clarification. If you like git, you might also like some internals of HomeGallery. The file indexer is inspired by git and also the fresh offline database api /api/database/tree/root.json.

[xemle]

@jimenezrick Do you need a release straight away or can you wait some days until I add some fixes regarding #106 ?

[xemle]

BTW what is about the change of

https://github.com/xemle/home-gallery/blob/b78fce7152e99498b7137411b079d0a8757d8ac4/packages/cli/src/tasks/import-sources.js#L197

I guess we need to change the value here too.

[jimenezrick]

@jimenezrick Do you need a release straight away or can you wait some days until I add some fixes regarding https://github.com/xemle/home-gallery/issues/106 ?

I can wait, thanks @xemle !

[jimenezrick]

BTW what is about the change of

https://github.com/xemle/home-gallery/blob/b78fce7152e99498b7137411b079d0a8757d8ac4/packages/cli/src/tasks/import-sources.js#L197

I guess we need to change the value here too.

Yeah, thanks! I don't use (yet) watched sources, hence I didn't notice it. I'm using mostly static galleries at the moment, but I've been playing with many other features.

https://github.com/xemle/home-gallery/pull/108

[xemle]

I've merged #108 please report if this solves your problem and we can close this issue

[jimenezrick]

Working as expected! (did a build from sources). I will wait for a future release including all the latest fixes. Many thanks and Merry Christmas!

[xemle]

@jimenezrick Thank you very much for your contribution.

v1.14.5 is released with our fixes.