Allow configuration of appdata directory location

[rcdailey]

I'm using Nextcloud 14.0.4 via the docker image. On my host machine, I mount a CIFS share to my filesystem, and I use this mounted directory as the data directory for Nextcloud. For performance reasons, my expectation was that Nextcloud would use the data directory as little as possible to avoid IO contention. Basically, it should only ever be used when an actual backed up file is accessed (actual files they sync). After all, the only reason I mount this way is so that the main storage for users goes to my NAS.

However, I'm observing that there's an appdata_abcdefg directory there, which has a lot of metadata in it. The most concerning piece of data in there are the image previews. I'd like a way to reduce the amount of usage that Nextcloud puts on the data directory. I'd like it to only store user data files, and the rest should reside on the local filesystem.

Is there a way to do this? How can I prevent metadata from going to my NAS for performance reasons?

[Joniator]

Since it seems there still is no officiaö solution, the best workarounds should be either

• creating a symlink to your local filesystem,
• mounting a local filesystem in the appdata folder, or
• (if using docker) use a seperate volume mount for the appdata folder.

[skjnldsv]

Yeah, not sure we want that :thinking: @MorrisJobke @rullzer ?

[danielb42]

@Joniator

Since it seems there still is no officiaö solution, the best workarounds should be either

* creating a symlink to your local filesystem,

Errors, as (quoting my log) "Following symlinks is not allowed".

* mounting a local filesystem in the appdata folder

I can't do this either since my nextcloud root dir is already a mounted gocryptfs. Mounting something else below isn't possible. (With ecryptfs's this works, but I want to migrate to gocryptfs. Which is what ultimately brought me here now.)

Please, please, please @MorrisJobke, make the appdata/ location configurable, or at least the appdata/preview location @rullzer. I really need to offload the previews/ directory to another drive.

[eknoes]

Yeah, not sure we want that @MorrisJobke @rullzer ?

Why could it be a problem? I run a similar setup like @rcdailey and having the appdata folder on the servers ssd instead of the network share could improve performance I guess. For me it does not seem to be hard to implement a configurable appdata location after taking a lot into the Appdata files. If you do not mind I would try to prepare a PR.

[skjnldsv]

Why could it be a problem?

Because there are other solutions already at the os level on how to mount other storage at different locations. Reinventing the wheel just make our work harder. Not saying this is a final statement of course, I'm not the expert on this area, but this is just what crossed my mind :)

[danielb42]

Why could it be a problem?

Because there are other solutions already at the os level on how to mount other storage at different locations. Reinventing the wheel just make our work harder.

No wheel currently available solves the issue.

There are at least two scenarios mentioned in this thread which establish a definitive need to outsource the appdata tree. First, preview-dir tends to get huge while being relatively uncritical in relation to the actual userdata. So it makes sense to store previews on other, maybe larger, cheaper, whatever, volumes. (Funny enough, the argument stands for larger/cheaper as well as for faster/more expensive. I sure want to access previews more often and faster than the actual images.)

We cannot do that by symlinking appdata/preview as following symlinks is forbidden by some component of nextcloud.

Second, using an filesystem-level encrypted volume for ones' entire nextcloud-data tree. As I mentioned above, mounting a "previews"-ecryptfs into a mounted "nextcloud-data"-ecryptfs works, but ecryptfs really sucks. Alternative encrypted filesystems like gocryptfs refuse to be mounted inside other mountpoints.

I also lack the insights on why following symlinks cannot be allowed or why having a configurable appdata location would be much of a problem, but I do hope I could still sketch why simply referring to OS mechanisms doesn't help here.

[eknoes]

I think also from a logical point of view it may make sense to have the appdata folder on a different location than the data folder, as it is used as a folder for temporary files. In the network storage setup, it would make sense to put them on a local SSD to speed up the usage of e.g. the Photos app, or am I wrong?

[Bojan023]

I agree that it makes sense to make the appdata directory location configurable from a admin point of view (especially the preview directory as @eknoes suggested). The workarounds mentioned and also hacking around with Docker volumes (for those using Docker) does not seem like an elegant solution.

[danielb42]

When this discussion went nowhere once again, I changed $allowSymlinks in lib/private/Files/Storage/Local.php from false to true. I am since able to symlink my thumbnail dir into appdata/ from anywhere I want and haven't noticed any side-effect from that in the recent months... just keep in mind to repeat after any update.

// update: Thanks @Meldrak for pointing out that there's no need to edit library files:

just a remark no need to change the file Local.php, just add the option to your config file instead 'localstorage.allowsymlinks' => true,

[Meldrak]

When this discussion went nowhere once again, I changed $allowSymlinks in lib/private/Files/Storage/Local.php from false to true. I am since able to symlink my thumbnail dir into appdata/ from anywhere I want and haven't noticed any side-effect from that in the recent months... just keep in mind to repeat after any update.

great I also symlink my appdata_instanceid to my ssd disk, it works well

just a remark no need to change the file Local.php, just add the option to your config file instead 'localstorage.allowsymlinks' => true,

[danielb42]

Thanks, but please note that is only possible since Nextcloud 21. In earlier versions $allowSymlinks is hardcoded to false in Local.php and nothing related is read from config.php.

[Meldrak]

so I had luck to not search for that before

[hendrik1120]

For 3 years now we have been storing image previews and configs in /data with no official way to change it? Let's wait 3 more years and hope everything will be on SSDs by then. :D

[keesfluitman]

So only symlinks are a solution to get previews on a different drive than the rest of the /data folder?

[Compunctus]

Ability to move that folder - or rather, it's css/js subfolders - would be extremely nice. I had to move my data to a slow hdd storage (due to size) - so each css/js access has a ~30ms penalty now.

That latency means nothing for file download/upload, richfiledocument templates, etc - but it completely destroys loading times due to each css/js query having +30ms to it's TTFB.

Why not store css/js somewhere else? Putting them near apps (also a per-instance mutable folder) sounds like a great idea.

P.S. the ideal solution would also allow user to move previews somewhere else. Turns out i have millions of them. Imagine someone trying to load their photo app...

[Meldrak]

Symlinking data will also move photos preview and decrease loading time

What I do for my usecase if it can help (nextcloud docker running on unraid): adapt paths to your configuration and environnement

add this line to your config file (www\nextcloud\config\config.php) 'localstorage.allowsymlinks' => true,

add a path variable from /home to your cache folder to nextcloud docker -v '/mnt/cache/system/nextcloud/':'/home':'rw'

add a init.d startup script to do the symlink (could be done with Linuxserver image)

#!/bin/bash

INSTANCEID=`cat /config/www/nextcloud/config/config.php | grep instanceid | cut -d "'" -f 4`

if [[ -L /data/appdata_"$INSTANCEID" && -d /data/appdata_"$INSTANCEID" ]];then
  exit
else
  mkdir /home/appdata_$INSTANCEID
  ln -s /home/appdata_$INSTANCEID/ /data/
fi

[a3linux]

Could I know if add 'localstorage.allowsymlinks' => true, is the ONLY to use a symlink for appdata_INSTANCEID folder to a symlink? I did this and still get can not open /appdata_INSTNACEID when I run files:scan-app-data.

Is it expected?

[langfingaz]

Could I know if add 'localstorage.allowsymlinks' => true, is the ONLY to use a symlink for appdata_INSTANCEID folder to a symlink? I did this and still get can not open /appdata_INSTNACEID when I run files:scan-app-data.

@a3linux You also have to make sure that the permissions on the new folder are the same.

[luxzg]

Would there be any volunteers knowledgeable enough to finish this feature?

Code is here, with comments, but it is stuck in limbo, and thus was closed today.

If someone would know enough to finish this we could get this issue closed.

Link: https://github.com/nextcloud/server/pull/36337#issuecomment-2090783913