search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.62k stars 3.99k forks source link

[Bug]: NC29 .well-known URLs, failed on: /.well-known/caldav #45033

Closed cvandesande closed 15 hours ago

cvandesande commented 4 months ago

⚠️ This issue respects the following points: ⚠️

Bug description

With NC28 I had no well-known URL errors, and no change to the NGINX configuration. After upgrading to NC29, I now have the following message: Your web server is not properly set up to resolve .well-known URLs, failed on: /.well-known/caldav For more details see the [documentation ↗](https://docs.nextcloud.com/server/29/go.php?to=admin-setup-well-known-URL).

In the NGINX logs, I see a 401 errors:

192.168.9.6 - - [25/Apr/2024:13:02:21 +0000] "PROPFIND /.well-known/caldav HTTP/1.1" 301 162 "-" "Nextcloud Server Crawler"
192.168.9.6 - - [25/Apr/2024:13:02:21 +0000] "GET /remote.php/dav HTTP/1.1" 401 569 "-" "Nextcloud Server Crawler"
192.168.9.6 - - [25/Apr/2024:13:02:21 +0000] "GET /.well-known/caldav HTTP/1.1" 301 162 "-" "Nextcloud Server Crawler"
192.168.9.6 - - [25/Apr/2024:13:02:21 +0000] "GET /remote.php/dav HTTP/1.1" 401 569 "-" "Nextcloud Server Crawler"

My Android DAVx5 client doesn't seem to have any issues and continues to work well.

A curl test, shows the 301 redirect working, followed by a 401, but I'm assume that's expected with an unauthenticated request

curl -IL https://nextcloud.mydomain.com/.well-known/caldav

HTTP/2 301 
server: nginx/1.25.4
date: Thu, 25 Apr 2024 13:12:51 GMT
content-type: text/html
content-length: 162
location: https://nextcloud.mydomain.com/remote.php/dav
referrer-policy: no-referrer
x-content-type-options: nosniff
x-download-options: noopen
x-frame-options: SAMEORIGIN
x-permitted-cross-domain-policies: none
x-robots-tag: noindex, nofollow
x-xss-protection: 1; mode=block
strict-transport-security: max-age=15768000; includeSubDomains; preload;

HTTP/2 401 
server: nginx/1.25.4
date: Thu, 25 Apr 2024 13:12:51 GMT
content-type: application/xml; charset=utf-8
set-cookie: oc_sessionPassphrase=NoBfvCYLv%2B7Hzw7eRoiL4VWZNyADXzcx2k5fgbu4FerYYBJjgl%2Fq8xLrTWscFxu3ithDlInkdabcfGg0nbL0wrG3B%2BPRkOCYjtibS4QvsluoWlkgrT5DdSWiQGUwHxi9; path=/; secure; HttpOnly; SameSite=Lax
set-cookie: __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
set-cookie: __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
set-cookie: oc2a3d1f7bcc=6eadda8dbfe0995012c16997240ee6b0; path=/; secure; HttpOnly; SameSite=Lax
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: no-store, no-cache, must-revalidate
pragma: no-cache
content-security-policy: default-src 'none';
www-authenticate: Basic realm="MyDomain, charset="UTF-8"
referrer-policy: no-referrer
x-content-type-options: nosniff
x-download-options: noopen
x-frame-options: SAMEORIGIN
x-permitted-cross-domain-policies: none
x-robots-tag: noindex, nofollow
x-xss-protection: 1; mode=block
strict-transport-security: max-age=15768000; includeSubDomains; preload;

There is an error in Nextcloud.log that appears relevant:

    "File": "/usr/share/nginx/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Auth/Plugin.php",
    "Line": 152,
    "message": "No public access to this resource., No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured, No 'Authorization: Bearer' header found. Either the client didn't send one, or the server is mis-configured, No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured"

Steps to reproduce

  1. Upgrade from NC28 to NC29
  2. well-known message in admin settings

Expected behavior

No well-known errors

Installation method

Community Manual installation with Archive

Nextcloud Server version

29

Operating system

Other

PHP engine version

PHP 8.2

Web server

Nginx

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.mydomain.com",
            "nextcloud"
        ],
        "apps_paths": [
            {
                "path": "\/usr\/share\/nginx\/html\/nextcloud\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/usr\/share\/nginx\/html\/nextcloud\/custom-apps",
                "url": "\/custom-apps",
                "writable": true
            }
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "29.0.0.19",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "forwarded_for_headers": [
            "HTTP_X_FORWARDED_FOR",
            "HTTP_X_FORWARDED",
            "HTTP_FORWARDED_FOR"
        ],
        "overwriteprotocol": "https",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "installed": true,
        "mail_smtpmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "loglevel": 0,
        "maintenance": false,
        "enable_previews": true,
        "secret": "***REMOVED SENSITIVE VALUE***",
        "filesystem_check_changes": 0,
        "filelocking.enabled": "true",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 1.5,
            "read_timeout": 1.5,
            "dbindex": 0
        },
        "trashbin_retention_obligation": "auto",
        "overwrite.cli.url": "https:\/\/nextcloud.mydomain.com",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpsecure": "ssl",
        "theme": "",
        "app_install_overwrite": [
            "joplin"
        ],
        "encryption.legacy_format_support": false,
        "encryption.key_storage_migrated": false,
        "default_language": "en",
        "default_phone_region": "CA",
        "maintenance_window_start": 1,
        "memories.db.triggers.fcu": true,
        "memories.exiftool": "\/usr\/share\/nginx\/html\/nextcloud\/custom-apps\/memories\/bin-ext\/exiftool-amd64-glibc",
        "memories.vod.path": "\/usr\/share\/nginx\/html\/nextcloud\/custom-apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.gis_type": 2,
        "enabledPreviewProviders": [
            "OC\\Preview\\Image",
            "OC\\Preview\\HEIC",
            "OC\\Preview\\Movie",
            "OC\\Preview\\TIFF"
        ]
    }
}

List of activated Apps

No response

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

{
  "reqId": "6jDGw0WbOAAarIrEpjXS",
  "level": 0,
  "time": "2024-04-25T13:21:06+00:00",
  "remoteAddr": "192.168.9.6",
  "user": "--",
  "app": "webdav",
  "method": "GET",
  "url": "/remote.php/dav",
  "message": "No public access to this resource., No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured, No 'Authorization: Bearer' header found. Either the client didn't send one, or the server is mis-configured, No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured",
  "userAgent": "Nextcloud Server Crawler",
  "version": "29.0.0.19",
  "exception": {
    "Exception": "Sabre\\DAV\\Exception\\NotAuthenticated",
    "Message": "No public access to this resource., No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured, No 'Authorization: Bearer' header found. Either the client didn't send one, or the server is mis-configured, No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured",
    "Code": 0,
    "Trace": [
      {
        "file": "/usr/share/nginx/html/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
        "line": 89,
        "function": "beforeMethod",
        "class": "Sabre\\DAV\\Auth\\Plugin",
        "type": "->",
        "args": [
          [
            "Sabre\\HTTP\\Request"
          ],
          [
            "Sabre\\HTTP\\Response"
          ]
        ]
      },
      {
        "file": "/usr/share/nginx/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 456,
        "function": "emit",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": [
          "beforeMethod:GET",
          [
            [
              "Sabre\\HTTP\\Request"
            ],
            [
              "Sabre\\HTTP\\Response"
            ]
          ]
        ]
      },
      {
        "file": "/usr/share/nginx/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 253,
        "function": "invokeMethod",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": [
          [
            "Sabre\\HTTP\\Request"
          ],
          [
            "Sabre\\HTTP\\Response"
          ]
        ]
      },
      {
        "file": "/usr/share/nginx/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 321,
        "function": "start",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": []
      },
      {
        "file": "/usr/share/nginx/html/nextcloud/apps/dav/lib/Server.php",
        "line": 374,
        "function": "exec",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": []
      },
      {
        "file": "/usr/share/nginx/html/nextcloud/apps/dav/appinfo/v2/remote.php",
        "line": 35,
        "function": "exec",
        "class": "OCA\\DAV\\Server",
        "type": "->",
        "args": []
      },
      {
        "file": "/usr/share/nginx/html/nextcloud/remote.php",
        "line": 172,
        "args": [
          "/usr/share/nginx/html/nextcloud/apps/dav/appinfo/v2/remote.php"
        ],
        "function": "require_once"
      }
    ],
    "File": "/usr/share/nginx/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Auth/Plugin.php",
    "Line": 152,
    "message": "No public access to this resource., No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured, No 'Authorization: Bearer' header found. Either the client didn't send one, or the server is mis-configured, No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured",
    "exception": {},
    "CustomMessage": "No public access to this resource., No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured, No 'Authorization: Bearer' header found. Either the client didn't send one, or the server is mis-configured, No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured"
  }
}

Additional info

No response

major-mayer commented 2 months ago

I am using Caddy and those rules resolved the problem for me:

redir /.well-known/carddav /remote.php/dav/ 301
redir /.well-known/caldav /remote.php/dav/ 301
MichaIng commented 2 months ago

@kesselb your fix for subdirs has not landed in NC 29.0.3 RC4, but IMO it makes sense to have this in NC 29.0.3. Is there still discussion needed, do you find time to open a PR or shall I do that the next hours?

kesselb commented 2 months ago

I didn't have the opportunity to continue working on it, and it's also not critical enough to rush for 29.0.3.

Moreover, the patch is not ready. CheckServerResponseTrait.runRequest is used by other setup checks and it's not correct to strip the web root for some (e.g. WebdavEndpoint and SecurityHeaders)

MichaIng commented 2 months ago

Understood. Looks like the method requires an option then. Is it for core only, or can it be generally used by other apps as well (so that we must not break the API)?

MichaIng commented 2 months ago

So while one part is missing support for subdirs, the other part that bugged me is the need for a trailing slash.

I checked how it happened, and whether it is really needed, and it is not. It is actually more wrong then right (while practically does no harm), as it can lead to double slashes in redirects. This PR makes the check accept both, with and without trailing slash, which it practically did before as well, since both work well with the Sabre/DAV backend: #46079

This prevents admins from the need to change their configs to mute the warning after upgrading to Nextcloud 29, while their redirects do work fine for CardDAV/CalDAV clients without the config change.

Lawkss commented 2 months ago

Personally I don't understand why this is still a thing. I think it should be able to work in subdirs out of the box. My host redirect config worked for all NC up to V28. What now, every 29.x version I need to re-apply the patch?

MichaIng commented 2 months ago

Yes it does work in subdirs OOTB and you do not need to change any config. It is only the Nextcloud internal check, which has changed and does not check the redirects correctly, when the instance is in a subdir. Similarly it is too strict about the trailing slash (which has however been fixed for next version 29.0.4).

andrewheeler82 commented 2 months ago

Hello, you're really doing a great job, but I find something like that ridiculous. You have to wait until 29.0.4 until something works again where it worked perfectly before 😭

MichaIng commented 2 months ago

I would have liked to have this in v29.0.3 as well, but it was already frozen. However, it is understandable that this was not seen and critical enough to merge it after freeze, because nothing is broken, everything "works". It is only about an admin panel warning, it is not about actually broken redirects. So you do not "have to wait until 29.0.4 until something works again", because everything still works like before. You only need to ignore the admin panel warning 😉.

sn0n commented 2 months ago

because nothing is broken, everything "works".

Gnome calendar and Contacts weren't working. My redirect was in fact broken, https://github.com/nextcloud/server/issues/45033#issuecomment-2186016510 Was the fix. I was missing the trailing /s and 301 at the end

andrewheeler82 commented 2 months ago

Your engine warning light is on but the manufacturer says it's OK, so just keep driving. The error will be fixed soon. 👍🏻👍🏻👍🏻

MichaIng commented 2 months ago

Gnome calendar and Contacts weren't working. My redirect was in fact broken,

That is unexpected. This worked before with NC28? The DAV backend has not changed. Only the admin panel checks moved to a new API, which is not affecting the redirects or the DAV backend at all. In fact I tested redirects without trailing slash, before opening #46079, and it worked all well: no slash, one slash or two slashes (theoretically possible with /.well-known/carddav => /remote.php/dav/ when doing a request to /.well-known/carddav/), all was handled successfully by Sabre/DAV and my CardDAV/CalDAV client (DAVx5, using generic method, and tested with Thunderbird again just now) was able to connect when giving it the hostname only without any path.

but the manufacturer says it's OK

I am not the manufacturer, just a volunteer contributor. And just to be clear: I only relaxed the test about the trailing slash so far, which you can mute as well by just adding the trailing slash to the redirect directives. The actual redirects work either way. If your Nextcloud is in a sub directory, this still needs to be done: https://github.com/nextcloud/server/issues/45033#issuecomment-2186479579

sn0n commented 2 months ago

It could be user error on my side, but I tried everything under the sun prior. Safe to ignore me for now I'm sure.

rafacouto commented 2 months ago

having this issue as well.

Caddy setup as reverse proxy, nextcloud manual install on its own VM. Caddy rewrite is working, returning a 401 result, resulting warning in Admin Overview.

caddy:2 and nextcloud:29 on docker: issue is solved by adding the trailing slash:

        redir /.well-known/carddav /remote.php/dav/ 301
        redir /.well-known/caldav /remote.php/dav/ 301
rodinux commented 2 months ago

I noticed two things:

1. We are using the trusted_domains without any "preprocessing". For example my dev-setup is tls-only. But the check request for the trusted domains goes to http:// because not protocol is given.  `overwriteprotocol`  is https and therefore the check for the trusted domains should also use tls.

2. The overwrite.cli.url includes the complete url including a webroot. We want to check if the redirect works and therefore need to strip the webroot. The patch below should fix that.
Index: apps/settings/lib/SetupChecks/CheckServerResponseTrait.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/apps/settings/lib/SetupChecks/CheckServerResponseTrait.php b/apps/settings/lib/SetupChecks/CheckServerResponseTrait.php
--- a/apps/settings/lib/SetupChecks/CheckServerResponseTrait.php  (revision fe4c1b28c7353deb7c7d3429a7490a5922bf1621)
+++ b/apps/settings/lib/SetupChecks/CheckServerResponseTrait.php  (date 1714992736792)
@@ -61,7 +61,7 @@
      $hosts = $this->config->getSystemValue('trusted_domains', []);
      $cliUrl = $this->config->getSystemValue('overwrite.cli.url', '');
      if ($cliUrl !== '') {
-         $hosts[] = $cliUrl;
+         $hosts[] = rtrim(str_replace($this->urlGenerator->getWebroot(), '', $cliUrl), '/');
      }

      $testUrls = array_merge(

This patch was the solution for me, thanks !

MichaIng commented 2 months ago

Just note this: https://github.com/nextcloud/server/issues/45033#issuecomment-2186517569 So it might cause issues with other calls.

But I see now that this still works since the trusted domains and CLI URI without webroot are just added to the array of test URLs, while it contains another entry which still contains the webroot:

https://my.domain.org/nextcloud/remote.php/webdav
my.domain.org/remote.php/webdav
https://my.domain.org/remote.php/webdav

When we want to implement this properly, e.g. with an option to define whether the webroot shall be part of the URL or not, the webroot would need to be removed from the first entry as well, to avoid unnecessary faulty requests. We could generate the hosts array with all entries first, then remove the webroot from all of them.

Another little enhancement: Since the request is done from the backend, which runs on the same host where CLI calls are done, the CLI URL should always work. I'd hence add it as first array entry, if present, while currently the frontend/browser request URL is the first entry, which might not work or be an unnecessary round trip. The trusted domains should be the last entries, since they cause further redirects if HTTPS is used, or might even fail if no HTTPS redirect is done, but plain HTTP not accepted.

StealUrKill commented 1 month ago

@kesselb overwrite URL is the public facing domain

grep overwrite config.php 
  'overwriteprotocol' => 'https',
  'overwrite.cli.url' => 'https://nextcloud.mydomain.com',
  'app_install_overwrite' =>

It's the same URL I used in my successful curl test should it be something different?

I had the same issues with just upgrading and nothing else.

I fixed it by removing the trailing domain

PREVIOUS

'overwrite.cli.url' => 'https://nextcloud.mydomain.com/drive',

NEW

'overwrite.cli.url' => 'https://nextcloud.mydomain.com',

kesselb commented 1 month ago

If the URL for your Nextcloud installation is https://nextcloud.mydomain.com/drive, then overwrite.cli.url should point to https://nextcloud.mydomain.com/drive as well.

Don't change overwrite.cli.url to a wrong value to make the warning disappear, just ignore the warning for now.

StealUrKill commented 1 month ago

If the URL for your Nextcloud installation is https://nextcloud.mydomain.com/drive, then overwrite.cli.url should point to https://nextcloud.mydomain.com/drive as well.

Don't change overwrite.cli.url to a wrong value to make the warning disappear, just ignore the warning for now.

I appreciate the info. I checked and seen it was all working. But I don't use the caldav or carddav. But I can definitely change it back.

andrewheeler82 commented 1 month ago

I was happy to hear that we are releasing version 29.0.4. In order to then determine the bug ist Stil Alive. Great 👍🏻

MichaIng commented 1 month ago

The trailing slash is now not needed anymore, but sub directories are still not supported. But a PR is open, and awaits review/suggested adjustments: #46255

tjirka-dcit commented 2 weeks ago

After update to 29.0.5 the bug is still there. I'm getting tired of manually patching it after every update.

andrewheeler82 commented 2 weeks ago

this software is just great. None of us have any idea how hard work it is. Same bug every version. dreams come true.

j-lakeman commented 1 week ago

@andrewheeler82 how about using your frustration trying to help fix the issue rather than spreading negativity through sarcasm and ranting? Heaps of contributors are working unpaid in their leisure time. According to your profile you haven't contributed much to the free software community.

rodinux commented 1 week ago

well, I see the warning also since 29.0.2, but in fact since 29.0.4 it's just a waning, syncing calendars works for me...

Ra72xx commented 1 week ago

But I dislike the fact that I have to ignore security warnings on public facing internet services, and that for several versions. That's no good.

andrewheeler82 commented 1 week ago

@j-lakeman sorry about that.

MichaIng commented 1 week ago

But I dislike the fact that I have to ignore security warnings

It is not a security warning, but a setup hint about a mostly convenience feature, so that users do not need to enter/paste the full CalDAV/CardDAV endpoint URL (which can be copy&pasted from the calendar/contacts apps' settings) into sync clients, but can use your Nextcloud base URL instead, from where clients are able to find the endpoint by themselves via .well-known addresses. Though, some clients seem to need it that way. The linked documentation page explains everything quite well.

I linked the PR above, which indeed has some unaddressed review suggestions for a while. Maybe @kesselb finds time to continue? Let me know if I can help with a complete code block suggestion about reordered URL priority and str_ends_with 🙂.

Virsacer commented 1 day ago

Yesterday I spent some time to analyze why Nextcloud was complaining about webfinger but not the other redirects. I found that the test-urls are not beeing properly built - so I "fixed" it by just adding more redirects:

Redirect 301 /nextcloud/.well-known/caldav /nextcloud/remote.php/dav
Redirect 301 /nextcloud/.well-known/carddav /nextcloud/remote.php/dav
Redirect 301 /nextcloud/.well-known/nodeinfo /nextcloud/index.php/.well-known/nodeinfo
Redirect 301 /nextcloud/.well-known/webfinger /nextcloud/index.php/.well-known/webfinger

Today I wanted to file a bug report and found this issue :D So until the patch is included in the stable release, you can use this simple trick...