provokateurin
commented
1 year ago I use nginx and ingress-nginx and calendar and contacts sync work for me :thinking:. We can try to compare configs and see why it doesn't work for you.
Open jessebot opened 1 year ago
provokateurin
commented
1 year ago I use nginx and ingress-nginx and calendar and contacts sync work for me :thinking:. We can try to compare configs and see why it doesn't work for you.
jessebot
commented
1 year ago Yes please, Thank you! 🙏 All of my configs are above under the "values.yaml" section. The ingress annotations seem to match the ones we have in this repo directly. I only had a slightly different nginx.config for a while before a kind soul merged a fix into this repo a little bit ago, but now I think it's the same too.
I also went and confirmed that I don't have modsecurity on yet (I'm sure that will be it's own can of worms when I turn that on 😂 ).
provokateurin
commented
1 year ago This is my whole values.yaml
image:
flavor: fpm-alpine
nextcloud:
host: [REDACTED]
ingress:
enabled: true
className: nginx
tls:
- hosts:
- [REDACTED]
secretName: [REDACTED]
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 10G
nginx.ingress.kubernetes.io/proxy-connect-timeout: 10m
nginx.ingress.kubernetes.io/proxy-read-timeout: 10m
nginx.ingress.kubernetes.io/proxy-send-timeout: 10m
nginx.ingress.kubernetes.io/proxy-buffering: "off"
cert-manager.io/cluster-issuer: letsencrypt-prod
external-dns.alpha.kubernetes.io/ttl: 1m
external-dns.alpha.kubernetes.io/target: [REDACTED]
nginx:
enabled: true
internalDatabase:
enabled: false
externalDatabase:
enabled: true
type: postgresql
host: "acid-nextcloud-cluster:5432"
existingSecret:
enabled: true
secretName: nextcloud.acid-nextcloud-cluster.credentials.postgresql.acid.zalan.do
usernameKey: username
passwordKey: password
redis:
enabled: true
persistence:
enabled: true
existingClaim: nextcloud
rbac:
enabled: true
cronjob:
enabled: truewith chart version 3.5.14. I use https://github.com/kubernetes/ingress-nginx 4.5.2 right now.
provokateurin
commented
1 year ago I guess the best way to test is to disable all your nginx customizations
jessebot
commented
1 year ago Awesome, thank you for all the details!
I'm also running the ingress-nginx-4.7.1 helm chart.
I agree, so I went ahead and removed most of the stuff I had there, and now my the relevant sections of my image, ingress, and nginx sections of my values.yaml looks like this:
image:
flavor: fpm
ingress:
enabled: true
className: nginx
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 10G
kubernetes.io/tls-acme: "true"
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For"
tls:
- secretName: nextcloud-tls
hosts:
- cloud.example.com
nginx:
enabled: true
# not sure if it matters, but I also have this trusted_proxies thing here:
nextcloud:
configs:
# adding your local ip might help on a self-hosted instance on your home network
proxy.config.php: |-
<?php
$CONFIG = array (
'trusted_proxies' => array(
0 => '127.0.0.1',
1 => '10.0.0.0/8'
),
'forwarded_for_headers' => array('HTTP_X_FORWARDED_FOR'),
);Pretty grateful to have removed a bunch of stuff that wasn't necessary, because most everything is still working, and this simplifies my troubleshooting a lot, but the same errors persist when I try to use the android nextcloud app with DAVx5 🤔 The macOS contacts/calendar apps stopped giving auth errors, but also aren't syncing anything. No logs in the admin panel except this:
[PHP] Error: Optional parameter $trustedServers declared before required parameter $groupManager is implicitly treated as a required parameter at /var/www/html/apps/dav/lib/CardDAV/SystemAddressbook.php#60
PROPFIND /remote.php/dav/addressbooks/users/jessebot/
from REDACTED_IP_ADDR by jessebot at 2023-07-11T16:12:24+00:00I think that's just from when I tried to import my vcards earlier, which is because of https://github.com/nextcloud/server/issues/38772 which should be solved soonish as there's an RC PR here, https://github.com/nextcloud/server/pull/39282, so I assume we'll get a new docker tag soonish. I don't know if that's partially breaking things though.
Oh also after removing all the nginx configs and ingress annotations I started getting this in the admin overview 🤔 :
provokateurin
commented
1 year ago I also see those warnings about well-know, but never bothered to investigate. As far as I know that should work ootb?
jessebot
commented
1 year ago I also see those warnings about well-know, but never bothered to investigate.
Ah yeah, I only recently got some time off work to start looking into them myself :) Added back the following ingress.annotations and that resolved the admin panel errors about "/.well-known/*":
nginx.ingress.kubernetes.io/server-snippet: |-
proxy_hide_header X-Powered-By;
rewrite ^/.well-known/webfinger /index.php/.well-known/webfinger last;
rewrite ^/.well-known/nodeinfo /index.php/.well-known/nodeinfo last;
rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;
location = /.well-known/carddav {
return 301 $scheme://$host/remote.php/dav/;
}
location = /.well-known/caldav {
return 301 $scheme://$host/remote.php/dav/;
}but caldav/carddav still throws the same errors ;(
✏️ edited to remove the robots.txt snippet that wasn't necessary
provokateurin
commented
1 year ago Have you enabled debug logging on the server to see if there is any more info in the logs?
jessebot
commented
1 year ago I plan on coming back to this after v27.0.1 of Nextcloud is released and available via docker, because it very may well be related to the bug I mentioned, and I haven't had a chance to enable debug logs (currently have it set to info), but I will revisit this soonish!
Looks like 27.0.1rc2 just dropped a couple of days ago, and for 27.0.0 they did 4 release candidates before settling, so probably another week or two and I'll upgrade, and approach this again with fresh eyes :) Thanks for all your help rubber-ducking in the meantime, Kate!
The nextcloud server team did the release :D Here's the PR: https://github.com/nextcloud/helm/pull/419 After I figure out upgrades, I'll come back to this issue.
qlonik
commented
1 year ago Hey. I have the same issue. My values on ingress are very similar to the original post. As soon as I set enable-cors to false on ingress (nginx.ingress.kubernetes.io/enable-cors: "false"), I managed to add my account for sync in official nextcloud app via DAVx5 and calendars and contacts seem to start synchronizing.
jessebot
commented
1 year ago Thanks @qlonik for chiming in! enable-cors should work though 🤔 Let me test and get back to you since #419 has been merged now.
jessebot
commented
1 year ago oh, I wonder if we can add something like the allow header like they suggest in this stack overflow post:
location / {
if ($http_origin ~* "^https?://(nextclouddomain.com/remote.php/dav//|www.nextcloudomain.com/remote.php/dav/)$") {
add_header Access-Control-Allow-Origin "$http_origin";
}
}I haven't had a chance to look into this due to a P1 with longhorn, but will update when I get back to testing this. If anyone in the meantime figures out the magic allow header to make this work either in the nginx.conf or a config-snippet in for ingress controller annotations, I'd love to see what you came up with :)
adborden
commented
1 year ago @provokateurin thanks for sharing what's working for you. Can you share your config for ingress-nginx? Are you using use-forwarded-headers?
I'm also running into this. I'm using Digital Ocean LB in front of ingress-nginx.
provokateurin
commented
12 months ago I had to resetup DAVx5 on my phone and now I run into the same problem
provokateurin
commented
12 months ago I'm going to take a look at this today.
I am pretty sure this got nothing to do with CORS because that is only interesting for browsers and any other clients don't care about it since it is not a server restriction.
My plan is to do a git bisect on this repo because I'm fairly sure that it is not a bug in a specific Nextcloud version since loads of other people outside of this helm chart would have the same problem.
provokateurin
commented
12 months ago So I tried a lot of different things and I still have no clue what is going on. For some reason I end up with
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\NotAuthenticated</s:exception>
<s:message>No public access to this resource., No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured, No 'Authorization: Bearer' header found. Either the client didn't send one, or the server is mis-configured, No 'Authorization: Basic' header found. Either the client didn't send one, or the server is misconfigured</s:message>
</d:error>when doing curl -v -X PROPFIND "http://cloud.example.com/.well-known/caldav" -L on my production instance.
With the exact same setup locally (terraform+k3d) I can not reproduce this and I also don't see the errors in the admin overview. I don't understand how there can be any difference between the two since they are using the exact same terraform config (other than hostnames).
Now I was wondering if it might have to do with some state that is bad inside the instance since there seems to be no difference really.
Sadly I can't test if my local instance works with DAVx5 :sweat_smile:
provokateurin
commented
12 months ago I tried my curl command against cloud.nextcloud.com and end up with the same unauthorized error. I am still able to successfully setup DAVx5 for that instance so it is probably not the source of the problems :/
provokateurin
commented
12 months ago @Northcode since you worked on https://github.com/nextcloud/helm/pull/241, do you mind sharing your setup and if it works as expected?
jessebot
commented
12 months ago Thank you @provokateurin for taking the lead on this 💙
provokateurin
commented
12 months ago @jessebot @adborden @qlonik Could all of you try to setup a fresh instance with your setup if possible? Maybe you will have the same as me where a fresh instance works just fine.
provokateurin
commented
12 months ago I just checked the admin overview again and saw this in the logs. The weird thing here is that it says http and not https since my prod instance is running on https. Looking at the redirect requests I also see that they only go to http and not https. I think this could be the problem here, although I don't understand why. I assume this might be because the nginx inside the container only does http and the ingress nginx does the https.
provokateurin
commented
12 months ago Yes my assumption was correct! You need to uncomment the nginx.ingress.kubernetes.io/server-snippet in the ingress annotations. The ingress needs to handle the redirects because only it knows about https. I still see a complaint about webfinger not being properly setup (will investigate), but I got DAVx5 to work!
(The reason why I wasn't able to reproduce it locally with my setup is that it doesn't use https.)
provokateurin
commented
12 months ago I can't find a difference between nodeinfo and webfinger so I'm not sure why that isn't working now :woman_shrugging:
Edit: The webfinger thing is a weird caching issue of my browser. In a new private window it works just fine.
jessebot
commented
12 months ago reopening till I can test too :)
provokateurin
commented
12 months ago Ah sorry, I thought you did test it already :see_no_evil:
provokateurin
commented
11 months ago Btw I think we could remove the robots.txt and deny all parts from the server snippet. In theory it will reduce the load because we can already deny requests at the ingress controller, but in practice it doesn't change much and only makes the server snippet more complex. @jessebot do you agree we can remove it?
pschichtel
commented
10 months ago I am pretty sure this got nothing to do with CORS because that is only interesting for browsers and any other clients don't care about it since it is not a server restriction.
@provokateurin but if you enable CORS in the nginx, doesn't that instruct nginx to reply to pre-flight requests, which use ... the OPTIONS method? https://developer.mozilla.org/en-US/docs/Glossary/Preflight_request
pschichtel
commented
10 months ago https://github.com/nextcloud/helm/commit/67ea6eabcd9e429a1836f615ec8c93b8badc3c81 says this is needed for source-ip preservation, but I don't see how CORS is involved in that and it's definitely a routing conflict. I guess some server snippet could be added to circumvent this conflict, but I don't think it's helpful anyway
provokateurin
commented
10 months ago I'm not sure what you mean. The problem here was that the ingress needed to do the redirect already because nextcloud itself (or it's nginx) doesn't know about https.
pschichtel
commented
10 months ago I was facing this exact issue even though I had the redirect in-place on nginx' side. The wrong redirect does not manifest itself with the error mentioned in the title here, however the OPTIONS routings conflict caused by the cors option in ingress-nginx will produce this exact error, because the OPTIONS request does not contain the dav header expected by clients (since ingress-nginx handles the request without passing it to nextcloud thinking it's a CORS-preflight-request).
pschichtel
commented
10 months ago I'm pretty certain about this because I spent the last 3 hours debugging this with curl and tcpdump while comparing it to another older nextcloud instance that didn't have the issue. With the cors option enabled in ingress-nginx, no OPTIONS request will ever reach nextcloud, as such nextcloud will never be able to add the dav header to the response.
In his quest to diagnose this, @jessebot has probably removed all the nginx annotations (snippet + cors stuff), which solved his issue but at the same time introduced the https->http redirect issue. The redirect issue was than later fixed by re-introducing the snippet, as can be read in the discussion above. The cors options were probably "accidentally" not added back, leaving him with a working setup.
alixinne
commented
10 months ago I've been hit by this and I can confirm @pschichtel's analysis. When running helm-nextcloud behind ingress-nginx, enabling CORS breaks DAV service discovery through the OPTIONS method due to this specific part of the nginx config template: https://github.com/kubernetes/ingress-nginx/blob/1bc745619d91b690c8985bbc16097e9fe804d2d2/rootfs/etc/nginx/template/nginx.tmpl#L959
One possible workaround is to override nginx.tmpl, as described by https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/custom-template/, and remove the entire block that returns HTTP 204. Note that nginx.tmpl depends on the controller's code, so make sure to pick the version of nginx.tmpl that matches your ingress-nginx helm chart to avoid errors.
Aside from this workaround, and entirely disabling CORS, I'm curious what would be the way forward for a more permanent fix.
pschichtel
commented
10 months ago @vtavernier unless someone can explain how nginx' CORS functionality make anys sense here, I don't think there is a reason to keep it in.
It has been added to the documentation as a part of "preserving client source IPs", which it most definitely doesn't help with. In the context of WebDAV in particular it has absolutely no purpose either. CORS handling should only have an impact if some javascript tries to access your nextcloud from another domain (the Cross-Origin part) in a browser.
alixinne
commented
10 months ago The phrasing is indeed really weird. From what I've read OPTIONS is used for some CORS pre-flight stuff but I haven't really took the time to dig through all that.
What actually worked for me to preserve source IPs using a k8s ingress-nginx + nginx/fpm nextcloud setup is adding the following snippet in nginx.config.custom, replacing the default nginx conf:
...
# Running behind a load-balancer already
real_ip_header X-Forwarded-For;
real_ip_recursive on;
set_real_ip_from 0.0.0.0/0;
...@S3LL1G28 did you try visiting the admin overview in an icognito tab? For me the warnings persisted because the browser cached the responses.
Routhinator
commented
10 months ago What people are missing here is reading the ingress-nginx documentation about CORS:
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#enable-cors
If you actually read the docs here, they are VERY clear about the problem and the solution. When you enable CORS the following defaults are applied:
nginx.ingress.kubernetes.io/cors-allow-methods: Controls which methods are accepted.
This is a multi-valued field, separated by ',' and accepts only letters (upper and lower case).
Default: GET, PUT, POST, DELETE, PATCH, OPTIONS Example: nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"nginx.ingress.kubernetes.io/cors-allow-headers: Controls which headers are accepted.
This is a multi-valued field, separated by ',' and accepts letters, numbers, _ and -.
Default: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization Example: nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For, X-app123-XPTO"
I have cors enabled and working perfectly with all Nextcloud features and CAL/CARDAV with the below.
TL;DR - you have to explicitly enable the methods your Nextcloud use case requires.
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-methods: "GET HEAD POST OPTIONS PUT PATCH DELETE PROPFIND MKCOL REPORT"
nginx.ingress.kubernetes.io/cors-allow-headers: "DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,X-Forwarded-For"
tenhishadow
commented
7 months ago @Routhinator @jessebot
It appears that this configuration adjustment doesn't alter the situation directly, but rather permits additional methods and headers at the Ingress level. However, it's important to note that the issue persists within the container, not at the Ingress level
Routhinator
commented
7 months ago Then you have an additional nginx or http proxy in line that you need to confugure with the same settings based on the documentation for that http server. If you are using nextcloud in fpm mode then it adds another nginx in line by necessity to provide access to static files and images, it will also need to be reconfigured to allow the same methods.
tenhishadow
commented
7 months ago @Routhinator exactly, but then it seems to me it should be in the default configuration of this container
jessebot
commented
7 months ago Sorry I'm backlogged to the ends of the earth right now, so slowly trying to get to all my many notifications.
In his quest to diagnose this, @jessebot has probably removed all the nginx annotations (snippet + cors stuff), which solved his issue but at the same time introduced the https->http redirect issue. The redirect issue was than later fixed by re-introducing the snippet, as can be read in the discussion above. The cors options were probably "accidentally" not added back, leaving him with a working setup.
@pschichtel oh hey, I think I saw you in another repo recently too :) to be clear: this has never worked for me, but I still haven't tested any of the other suggestions. I will try to this week though.
@Routhinator and @tenhishadow we can either add it as a default config OR you (or anyone from the community) can submit a PR to add a section to our docs about this, if you've tested it and everything is working :)
tenhishadow
commented
7 months ago @jessebot Okay, I'll try to get it all done and tested then. As far as I understand, the change should be made here, right? https://github.com/nextcloud/helm/blob/main/charts/nextcloud/templates/nginx-config.yaml
If it works out, I'll raise the PR and link to this issue and mine https://github.com/nextcloud/helm/issues/562
jessebot
commented
7 months ago @tenhishadow if it's a snippet, it should be added here (commented out like everything else, but still add the info on how to do in the docs so people know what's going on there): https://github.com/nextcloud/helm/blob/30c69c16298c189dd78c930660a17a2b5b774773/charts/nextcloud/values.yaml#L28
If it's for the actual nginx config, then yeah it would go in that location you linked, but I'm always cautious about editing that, as many users have different preferences about that file. From what @Routhinator mentioned, it looks like this is fine as a snippet though, unless I'm unclear on how this works (totally possible) 🤔
jessebot
commented
7 months ago Ok, let's do a full update on where we are. I'm on helm chart verison 4.6.7.
I finally went through and in the above commit, tried the new cors stuff that routhinator suggested (both with and without commas for the allowed cors methods). It did not work. I followed the instructions here in the nextcloud docs.
You can see my whole nextcloud config, including the ingress config here (and a second attempt here), but here's my current ingress section specifically:
values.yamlHere's the DAVx5 logs from my android phone, but I did a search and replace of my domain with cloud.example.com, and my username with frienduser: dav-logs-anonymized.log. Important failing bits here:
I'm on Android 14 and I downloaded the latest version of nextcloud app (3.29.0) from the google play store today.
Was there something else suggested that I have missed?
I also tried the suggestion in https://github.com/nextcloud/helm/issues/410#issuecomment-1880210225 and it returned the same thing as attempt 1. 🤔
Here's the commit where I tried it, but the gist of it is that I added this to my values.yaml:
nginx:
enabled: true
config:
# This generates the default nginx config as per the nextcloud documentation
default: true
# Running behind a load-balancer already so trying what was suggested in https://github.com/nextcloud/helm/issues/410#issuecomment-1880210225
custom: |-
real_ip_header X-Forwarded-For;
real_ip_recursive on;
set_real_ip_from 0.0.0.0/0;Does the nginx config suggestion require disabling the default config? Or maybe it needs to be in the server block?
Perhaps I need to try the suggestion in https://github.com/nextcloud/helm/issues/410#issuecomment-1880210225 without the suggestion in https://github.com/nextcloud/helm/issues/410#issuecomment-1913273235 ?
I will first try disabling cors to see what that does...
Disabling cors works! That does me a concern though :,) Here's the magic commit that worked. When I check https://scan.nextcloud.com with my personal domain, it doesn't return any issues. And when I check the admin page for my domain (cloud.example.com/settings/admin/overview), there's no security warnings there either.
Perhaps we don't need cors? @pschichtel suggested it not being necessary, but I don't understand enough to know why it was there to begin with, because I thought it was necessary for security. 🤔 How do you protect from cross origin stuff without cors, or should it be nested in the nginx config itself?
@tenhishadow if you find a way to make it work with cors, let me know! (I will also test further in when I have time)
jessebot
commented
7 months ago @provokateurin to your earlier question, yes, (sorry, I'm late on responding to everything everywhere 🥲 ) we can remove the robots.txt block from the ingress annotation snippets as it doesn't break anything to remove it, and it's already in place here. I feel like having it set to allow all is weird though, as do you really want to advertise your private nextcloud for search engines to scan? robots.txt is supposed to be, or originally was back in the day, a way to have search engines crawl your domain, for SEO. But for a private nextcloud domain, say you're in China, do you want Google to crawl your domain? Or if you're in the U.S., do you want baidu to crawl your domain? Maybe you want your country's primary search engines to crawl your domain, but I feel like its unlikely.
provokateurin
commented
7 months ago Attempt 3 - disable cors in ingress nginx annotations
This can not be a correct solution because CORS only affects web browsers and any other clients ignore it. It might fix the the errors on the admin page, but it can not be a direct fix for the real problem. Maybe disabling it has some side effect somewhere else that makes it work, but CORS itself can not fix or break this.
pschichtel
commented
7 months ago CORS itself can not fix or break this.
@provokateurin Except when the reverse proxy prevents the OPTIONS request from going through to the application, because it thinks it is a CORS preflight request by a browser. This is what happens with CORS enabled in ingress Nginx and is exactly what breaks this.
@jessebot CORS is for security, but not in the way you think. CORS grants access to browsers which wouldn't otherwise have access. So if anything, CORS weakens security in this case. Moving the same generic rule into the Nginx container will likely result in the exact same result for the exact same reason. For proper CORS support you would need to know specifically which resources need cross origin access in browsers. Usually the application itself has this knowledge, so I'd suggest leaving CORS headers to the PHP code.
The commit that introduced the CORS settings does not give a valid reason for introducing it.
I could imagine there might be federation use cases where such cross origin requests might be happening, or maybe some embedded content use cases. What ever it might be, I don't think it warrants a generic catch-all CORS rule in the reverse proxy.
pschichtel
commented
7 months ago Where the confusion might come from: most people think of sending some headers to allow cross origin access, but that's not the whole story. I initially also thought that Nginx would just add the cors headers to appropriate responses as other reverse proxies do it. Nginx goes all the way and completely intercepts any OPTIONS requests and responds with 200 + the configured CORS headers. Which is great for applications that need CORS for mutating endpoints (POST, PUT, DELETE) , but don't want to implement the protocol (which might be a bit of a php thing). But it's obviously bad for applications that need OPTIONS requests for other purposes like in this case service discovery.
The mdn covers it well: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
provokateurin
commented
7 months ago Except when the reverse proxy prevents the OPTIONS request from going through to the application
But DAV clients don't make OPTIONS requests afaik?
jessebot
commented
7 months ago The commit that introduced the CORS settings does not give a valid reason for introducing it.
@pschichtel thanks for the info! Also, the CORS setting on this repo specifically was originally introduced because nextcloud (specifically v27 and older iirc) used to throw a warning in the admin panel if you didn't have it enabled. It doesn't seem to throw that error anymore as of at least v29 anyway.
This can not be a correct solution because CORS only affects web browsers and any other clients ignore it. It might fix the the errors on the admin page, but it can not be a direct fix for the real problem. Maybe disabling it has some side effect somewhere else that makes it work, but CORS itself can not fix or break this.
Except when the reverse proxy prevents the OPTIONS request from going through to the application
But DAV clients don't make OPTIONS requests afaik?
@provokateurin I finally got a chance to try again with the cors settings, just to be sure, and this time, when trying to use dav, after confirming the 405 error again on the android DAVx5 client side, I then checked the nginx container in the nextcloud pod and discovered this:
IP_REDACTED - frienduser [01/May/2024:09:33:58 +0000] "PROPFIND /remote.php/dav HTTP/1.1" 207 242 "-" "DAVx5/4.3.16.1-ose (2024/04/20; dav4jvm; okhttp/4.12.0) Android/14" "IP_REDACTED"
IP_REDACTED - frienduser [01/May/2024:09:33:59 +0000] "PROPFIND / HTTP/1.1" 405 157 "-" "DAVx5/4.3.16.1-ose (2024/04/20; dav4jvm; okhttp/4.12.0) Android/14" "IP_REDACTED"It doesn't look like it needs OPTIONS on the surface there. It looks like it needs PROPFIND and it looks like if I set the following annotations for my ingress object, it should allow PROPFIND:
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, HEAD, POST, OPTIONS, PUT, PATCH, DELETE, PROPFIND, MKCOL, REPORT"
nginx.ingress.kubernetes.io/cors-allow-headers: "DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,X-Forwarded-For"however, if you look at the logs above, the nginx container allows PROPFIND to /remote.php/dav, but not /? 🤷
I then turned off cors, deleted my local DAVx5 account, and tailed the logs of the nginx container in the nextcloud pod while creating a new account in the DAVx5 client. The logs reveal:
OPTIONS is requested here:
IP_REDACTED - frienduser [01/May/2024:10:02:23 +0000] "OPTIONS /remote.php/dav/principals/users/frienduser/ HTTP/1.1" 200 0 "-" "DAVx5/4.3.16.1-ose (2024/04/20; dav4jvm; okhttp/4.12.0) Android/14" "IP_REDACTED"So for right now, cors can be disabled for me I guess. 🤷 As an aside, I also recently noted the following in our default nginx.conf... wondering if maybe setting this to also catch ^DAVx5 would help? https://github.com/nextcloud/helm/blob/30c69c16298c189dd78c930660a17a2b5b774773/charts/nextcloud/templates/nginx-config.yaml#L70-L74
Perhaps adding the following to nginx.conf could be useful in our quest?
location = / {
if ( $http_user_agent ~ ^DAVx5 ) {
return 302 /remote.php/webdav/$is_args$args;
}
} ...but what's weird is that the correct IP isn't being gotten from nginx, but I guess that's a different issue, unless what alixinne said is what's breaking this... the first and second IP_REDACTED above in the prior post's logs are the same IP, which feels bad also from a bruteforce IP ban perspective, because I can't easily ban an offending IP. Perhaps I did the nginx custom config wrong:
nginx:
enabled: true
config:
# This generates the default nginx config as per the nextcloud documentation
default: true
# Running behind a load-balancer already so trying what was suggested in https://github.com/nextcloud/helm/issues/410#issuecomment-1880210225
custom: |-
real_ip_header X-Forwarded-For;
real_ip_recursive on;
set_real_ip_from 0.0.0.0/0;Maybe it needs to be something like in a block, like a server block or location = / block? maybe like this? 🤷
nginx:
enabled: true
config:
# This generates the default nginx config as per the nextcloud documentation
default: true
# Running behind a load-balancer already so trying what was suggested in https://github.com/nextcloud/helm/issues/410#issuecomment-1880210225
custom: |-
server {
real_ip_header X-Forwarded-For;
real_ip_recursive on;
set_real_ip_from 0.0.0.0/0;
}Under the preserving source IP section of our docs, as pschichtel mentioned in https://github.com/nextcloud/helm/issues/410#issuecomment-1880199873, it does say to enable cors and allow cors headers for X-Forwarded-For, but I don't know how to do that without breaking my dav stuff :/
Either @pschichtel or @alixinne can you post your nginx section of your values.yaml if you have this working? I'm not sure where the *real_ip* stuff should go in it. 🤷
Describe your Issue
I can't seem to get my calendar and contacts synced on Android (but I also tried on my m1 macbook running macOS 13.4.1). This has never worked for me, but only now am I looking into it. The logs seem to suggest it starts to work with:
But then I see this:
Principal https://cloud.example.com/remote.php/dav/principals/users/jessebot/ doesn't provide caldav servicefolllowed by it seemingly trying the base domain again and returning this:
(full verbosity logs in the next section)
I know there's kind of a lot of caldav/carddav forum posts already, but none of them cover using an nginx container and the ingress-nginx controller on k8s. The closest we have is a few issues referencing docker, but my current configs are fairly, if not fully, aligned with what seem to be all the docs across the nextcloud/server, nextcloud/docker, and nextcloud/helm repos. The other issue is that many of the forum posts are ancient referencing nextcloud versions 13-17ish.
Logs and Errors
So this log is actually from the DAVx5 app after following the guide in the server docs here (note: I've tried both with my personal username and password and also an app password I generated from my security settings and both fail with
...doesn't provide caldav service):DAVx5 verbose logging
```logtalk 2023-07-11 15:01:53 105 [network.HttpClient] <-- END HTTP (419-byte, 231-gzipped-byte body) 2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Found current-user-principal: https://cloud.example.com/remote.php/dav/principals/users/jessebot/ 2023-07-11 15:01:53 105 [network.HttpClient] --> OPTIONS https://cloud.example.com/remote.php/dav/principals/users/jessebot/ h2 2023-07-11 15:01:53 105 [network.HttpClient] Content-Length: 0 2023-07-11 15:01:53 105 [network.HttpClient] Accept-Encoding: identity 2023-07-11 15:01:53 105 [network.HttpClient] User-Agent: DAVx5/4.3.4.1-ose (2023/06/16; dav4jvm; okhttp/4.11.0) Android/13 2023-07-11 15:01:53 105 [network.HttpClient] Accept-Language: en-NL, en;q=0.7, *;q=0.5 2023-07-11 15:01:53 105 [network.HttpClient] Host: cloud.example.com 2023-07-11 15:01:53 105 [network.HttpClient] Connection: Keep-Alive 2023-07-11 15:01:53 105 [network.HttpClient] Cookie: oc_sessionPassphrase=mvZ%2FR8ZizTqiwfl3Eb%2FSUgwUWL9H3FXLLXxsFyEWfWY0kzrPcknYY9lyJZdBUN1wPVqccKNzamEv0B5FhseZtMxQnrJqpUw%2BEvqcKj%2FslgEVPaNSD9uLcd2UlYtcQx5K; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; ocprwvfxs7k4=16ba82cdc7e9dddafecddebf347f39a8 2023-07-11 15:01:53 105 [network.HttpClient] --> END OPTIONS 2023-07-11 15:01:53 105 [at.bitfire.dav4jvm.BasicDigestAuthHandler] Adding Basic authorization header for https://cloud.example.com/remote.php/dav/principals/users/jessebot/ 2023-07-11 15:01:53 105 [network.HttpClient] <-- 204 https://cloud.example.com/remote.php/dav/principals/users/jessebot/ (37ms) 2023-07-11 15:01:53 105 [network.HttpClient] date: Tue, 11 Jul 2023 13:01:53 GMT 2023-07-11 15:01:53 105 [network.HttpClient] strict-transport-security: max-age=15724800; includeSubDomains 2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-origin: * 2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-credentials: true 2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-methods: GET, PUT, POST, DELETE, PATCH, OPTIONS 2023-07-11 15:01:53 105 [network.HttpClient] access-control-allow-headers: X-Forwarded-For 2023-07-11 15:01:53 105 [network.HttpClient] access-control-max-age: 1728000 2023-07-11 15:01:53 105 [network.HttpClient] content-length: 0 2023-07-11 15:01:53 105 [network.HttpClient] <-- END HTTP (0-byte body) 2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Principal https://cloud.example.com/remote.php/dav/principals/users/jessebot/ doesn't provide caldav service 2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] No principal found at user-given URL, trying to discover for domain cloud.example.com 2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Looking up SRV records for _caldavs._tcp.cloud.example.com 2023-07-11 15:01:53 105 [util.DavUtils] Using Android 10+ DnsResolver 2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Didn't find caldav service, trying at https://cloud.example.com:443 2023-07-11 15:01:53 105 [util.DavUtils] Using Android 10+ DnsResolver 2023-07-11 15:01:53 105 [servicedetection.DavResourceFinder] Trying to determine principal from initial context path=https://cloud.example.com/.well-known/caldav 2023-07-11 15:01:53 105 [network.HttpClient] --> PROPFIND https://cloud.example.com/.well-known/caldav h2 2023-07-11 15:01:53 105 [network.HttpClient] Depth: 0 2023-07-11 15:01:53 105 [network.HttpClient] User-Agent: DAVx5/4.3.4.1-ose (2023/06/16; dav4jvm; okhttp/4.11.0) Android/13 2023-07-11 15:01:53 105 [network.HttpClient] Accept-Language: en-NL, en;q=0.7, *;q=0.5 2023-07-11 15:01:53 105 [network.HttpClient] Content-Type: application/xml; charset=utf-8 2023-07-11 15:01:53 105 [network.HttpClient] Content-Length: 198 2023-07-11 15:01:53 105 [network.HttpClient] Host: cloud.example.com 2023-07-11 15:01:53 105 [network.HttpClient] Connection: Keep-Alive 2023-07-11 15:01:53 105 [network.HttpClient] Accept-Encoding: gzip 2023-07-11 15:01:53 105 [network.HttpClient] Cookie: oc_sessionPassphrase=mvZ%2FR8ZizTqiwfl3Eb%2FSUgwUWL9H3FXLLXxsFyEWfWY0kzrPcknYY9lyJZdBUN1wPVqccKNzamEv0B5FhseZtMxQnrJqpUw%2BEvqcKj%2FslgEVPaNSD9uLcd2UlYtcQx5K; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; ocprwvfxs7k4=16ba82cdc7e9dddafecddebf347f39a8 2023-07-11 15:01:53 105 [network.HttpClient] 2023-07-11 15:01:53 105 [network.HttpClient]301 Moved Permanently
405 Not Allowed
Describe your Environment
Kubernetes distribution: k3s
Helm Version (or App that manages helm): ArgoCD version 2.6.7
Helm Chart Version: 3.5.15 which uses the nextcloud 27.0.0 image
values.yaml:
```yaml image: repository: nextcloud flavor: fpm pullPolicy: Always replicaCount: 1 ```image
```yaml ingress: enabled: true className: nginx annotations: nginx.ingress.kubernetes.io/proxy-body-size: 4G kubernetes.io/tls-acme: "true" cert-manager.io/cluster-issuer: letsencrypt-prod nginx.ingress.kubernetes.io/enable-cors: "true" nginx.ingress.kubernetes.io/cors-allow-headers: "X-Forwarded-For" nginx.ingress.kubernetes.io/server-snippet: |- server_tokens off; proxy_hide_header X-Powered-By; rewrite ^/.well-known/webfinger /index.php/.well-known/webfinger last; rewrite ^/.well-known/nodeinfo /index.php/.well-known/nodeinfo last; rewrite ^/.well-known/host-meta /public.php?service=host-meta last; rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json; location = /.well-known/carddav { return 301 $scheme://$host/remote.php/dav/; } location = /.well-known/caldav { return 301 $scheme://$host/remote.php/dav/; } location = /robots.txt { allow all; log_not_found off; access_log off; } location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ { proxy_buffer_size 128k; proxy_buffers 4 256k; proxy_busy_buffers_size 256k; deny all; } location ~ ^/(?:autotest|occ|issue|indie|db_|console) { proxy_buffer_size 128k; proxy_buffers 4 256k; proxy_busy_buffers_size 256k; deny all; } tls: - secretName: nextcloud-tls hosts: - cloud.example.com labels: {} path: / pathType: Prefix ```ingress
```yaml nginx: ## You need to set an fpm version of the image for nextcloud if you want to use nginx! enabled: true image: repository: nginx tag: alpine pullPolicy: IfNotPresent config: # This generates the default nginx config as per the nextcloud documentation default: false custom: |- worker_processes auto; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; #tcp_nopush on; keepalive_timeout 65; #gzip on; upstream php-handler { server 127.0.0.1:9000; # unsure if this still works, worked in php7.x-fpm server unix:/var/run/php/php8.1-fpm.sock; } server { listen 80; # set max upload size client_max_body_size 10G; fastcgi_buffers 64 4K; # Enable gzip but do not remove ETag headers gzip on; gzip_vary on; gzip_comp_level 4; gzip_min_length 256; gzip_proxied expired no-cache no-store private no_last_modified no_etag auth; gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy; # Pagespeed is not supported by Nextcloud, so if your server is built # with the `ngx_pagespeed` module, uncomment this line to disable it. #pagespeed off; # HTTP response headers borrowed from Nextcloud `.htaccess` add_header Referrer-Policy "no-referrer" always; add_header X-Content-Type-Options "nosniff" always; add_header X-Download-Options "noopen" always; add_header X-Frame-Options "SAMEORIGIN" always; add_header X-Permitted-Cross-Domain-Policies "none" always; add_header X-Robots-Tag "noindex, nofollow" always; add_header X-XSS-Protection "1; mode=block" always; # Remove X-Powered-By, which is an information leak fastcgi_hide_header X-Powered-By; # Path to the root of your installation root /var/www/html; # Specify how to handle directories -- specifying `/index.php$request_uri` # here as the fallback means that Nginx always exhibits the desired behaviour # when a client requests a path that corresponds to a directory that exists # on the server. In particular, if that directory contains an index.php file, # that file is correctly served; if it doesn't, then the request is passed to # the front-end controller. This consistent behaviour means that we don't need # to specify custom rules for certain paths (e.g. images and other assets, # `/updater`, `/ocm-provider`, `/ocs-provider`), and thus # `try_files $uri $uri/ /index.php$request_uri` # always provides the desired behaviour. index index.php index.html /index.php$request_uri; # Rule borrowed from `.htaccess` to handle Microsoft DAV clients location = / { if ( $http_user_agent ~ ^DavClnt ) { return 302 /remote.php/webdav/$is_args$args; } } location = /robots.txt { allow all; log_not_found off; access_log off; } # Make a regex exception for `/.well-known` so that clients can still # access it despite the existence of the regex rule # `location ~ /(\.|autotest|...)` which would otherwise handle requests # for `/.well-known`. location ^~ /.well-known { # The following 6 rules are borrowed from `.htaccess` location = /.well-known/carddav { return 301 /remote.php/dav/; } location = /.well-known/caldav { return 301 /remote.php/dav/; } # Anything else is dynamically handled by Nextcloud location ^~ /.well-known { return 301 /index.php$uri; } try_files $uri $uri/ =404; } # Rules borrowed from `.htaccess` to hide certain paths from clients location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/) { return 404; } location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) { return 404; } # Ensure this block, which passes PHP files to the PHP process, is above the blocks # which handle static assets (as seen below). If this block is not declared first, # then Nginx will encounter an infinite rewriting loop when it prepends `/index.php` # to the URI, resulting in a HTTP 500 error response. location ~ \.php(?:$|/) { # Required for legacy support rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri; fastcgi_split_path_info ^(.+?\.php)(/.*)$; set $path_info $fastcgi_path_info; try_files $fastcgi_script_name =404; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param PATH_INFO $path_info; #fastcgi_param HTTPS on; # Avoid sending the security headers twice fastcgi_param modHeadersAvailable true; fastcgi_param front_controller_active true; # Enable pretty urls fastcgi_pass php-handler; fastcgi_intercept_errors on; fastcgi_request_buffering off; } location ~ \.(?:css|js|svg|gif)$ { try_files $uri /index.php$request_uri; expires 6M; # Cache-Control policy borrowed from `.htaccess` access_log off; # Optional: Don't log access to assets } location ~ \.woff2?$ { try_files $uri /index.php$request_uri; expires 7d; # Cache-Control policy borrowed from `.htaccess` access_log off; # Optional: Don't log access to assets } location / { try_files $uri $uri/ /index.php$request_uri; } } } resources: {} ```nginxAdditional context, if any
screenshot of admin overview page
screenshot of Mobile & Desktop setting
but it also throw errors when the profile is imported:
macOS error example