[Bug]: External Storage Support $User variable not working

vicky-bs commented 1 year ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

When using the $user variable for external storage support. I get a red mark without anything in the logs

Steps to reproduce

Add External Storage support as Local
Add location in configuration along with $user variable eg /mnt/storage/nextcloud/$user
Error when $user is added. Error doesn't occur when $user is removed

Expected behavior

Plugin should allow $user variable to be included within the mount path so each user have their own folder created

Installation method

Community Docker image

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.0

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "check_data_directory_permissions": false,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "nextcloud.***.**.*",
        "overwriteprotocol": "https",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "nextcloud.***.**.**"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "27.0.2.0",
        "overwrite.cli.url": "https:\/\/nextcloud.***.**.*\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "updater.release.channel": "stable",
        "updatedirectory": "\/nc-updater",
        "loglevel": "2",
        "app_install_overwrite": [
            "nextcloud-aio"
        ],
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": "10485760",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "enabledPreviewProviders": {
            "1": "OC\\Preview\\Image",
            "2": "OC\\Preview\\MarkDown",
            "3": "OC\\Preview\\MP3",
            "4": "OC\\Preview\\TXT",
            "5": "OC\\Preview\\OpenDocument",
            "6": "OC\\Preview\\Movie",
            "7": "OC\\Preview\\Krita",
            "0": "OC\\Preview\\Imaginary"
        },
        "enable_previews": true,
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": "30",
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "htaccess.RewriteBase": "\/",
        "files_external_allow_create_new_local": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "allow_local_remote_servers": true,
        "preview_imaginary_url": "http:\/\/nextcloud-aio-imaginary:9000",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "***",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

- activity: 2.19.0
  - admin_audit: 1.17.0
  - calendar: 4.4.4
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contacts: 5.3.2
  - contactsinteraction: 1.8.0
  - dashboard: 7.7.0
  - dav: 1.27.0
  - deck: 1.10.0
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_antivirus: 5.2.1
  - files_external: 1.19.0
  - files_fulltextsearch: 27.0.1
  - files_pdfviewer: 2.8.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - fulltextsearch: 27.0.1
  - fulltextsearch_elasticsearch: 27.0.1
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - nextcloud-aio: 0.4.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.8.1
  - notifications: 2.15.0
  - notify_push: 0.6.3
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - richdocuments: 8.1.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - sociallogin: 5.4.3
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0
  - tasks: 0.15.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_totp: 9.0.0
  - unsplash: 2.2.1
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - bruteforcesettings: 2.7.0
  - encryption: 2.15.0
  - suspicious_login: 5.0.0
  - user_ldap: 1.17.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

solracsf commented 1 year ago

Plugin should allow $user variable to be included within the mount path so each user have their own folder created

This is the point here; folder (mountpoint full path) must exist I think, it is not created.

vicky-bs commented 1 year ago

My external mount point was /mnt/storage/nextcloud and in nextcloud under external storage configuration I provided /mnt/storage/nextcloud/$user and it show an error and doesn't accept this as a configuration. Is there another way to have this directly in he mount config file?

kesselb commented 1 year ago

This is the point here; folder (mountpoint full path) must exist I think, it is not created.

Yes.

I assume that's a good hint for our documentation page https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage_configuration_gui.html#usage-of-variables-for-mount-paths and therefore moved it over here.

joho1968 commented 1 year ago

I'm (probably) confused, isn't the whole issue that $user should be functional/working, i.e. a "feature request", rather than a documentation issue?

joshtrichards commented 1 year ago

Yeah, hey @kesselb I think the issue may be that it's not even possible to enter $user on that screen right now.

It errors out in the Web UI because the check fails. I'm not even sure how the check is supposed to function when substitution is used.

joshtrichards commented 1 year ago

Wait - I take that back. I typo my test.

joshtrichards commented 1 year ago

@vicky-bs - You can add it as long as you make sure that, at the very least, /mnt/storage/nextcloud/<your_username> exists so that the basic sanity check passes.

Of course for other users to be able to access their respective folders, you'll have to create every needed $user folder within /mnt/storage/nextcloud via whatever out-of-NC means is appropriate for your use case.

Below (ignore the first entry) are some test runs I just did:

In the /tmp/test2/$user scenario I created folders within /tmp/test2/ for each of my existing users ahead of time to guarantee it would work (and simulate a typical situation such as mounting a remote /home folder or whatever where there are pre-existing user files). That succeeded.

In the /tmp/test4/$user scenario I first tried to add while /tmp/test4/ was empty and I got the red failed check error you likely saw. I then created a directory for my admin username in /tmp/test4/ and triggered a re-check. That passed and turned green.

So it seems a couple bits missing from the docs for variable substitutions:

The configuration check when adding the External Storage runs against the logged in user's account (generally the admin). That sort of makes sense because I wasn't sure how the checks would work against a 10,000 user system. :)
The logged in user's folder in the new External Storage must exist, at a minimum, at configuration time
The individual target folder for any given user must exist before they can access their own mount (it'll show in their Files but as a red broken External Mount), but does not have to exist at configuration time (other than for the previously mentioned person doing the configuration)

That is not the language to go in the docs. More a note to myself or the party that does the doc PR. :)

P.S. Items 1&2 may be bypassed if using the command-line occ files_external:create approach to provision things, but I haven't confirmed it. But given that verification is a separate command occ files_external:verify it's a possibility.

vicky-bs commented 1 year ago

@joshtrichards . Thank you. I was able to create the folder and then change the configuration in external storage app. There sure is an opportunity to improve here. It's either that when user add $user in external app config, the folders for each user is automatically created with the right permission OR the documentation can be improved further to highlight that the folders need to exists before add the variable in configuration. Happy to help reword the documentation on this but I must say i'm a novice :). Let me know

joho1968 commented 1 year ago

It'd be so nice if NC could simply check the path up until $user and then accept it (since it knows $user is a supported macro) to allow for much easier on-boarding when using external storage.

nextcloud / documentation