search

nextcloud / helm

A community maintained helm chart for deploying Nextcloud on Kubernetes.
GNU Affero General Public License v3.0
295 stars 258 forks source link

Uploading files over 1Mb fails with 413 file too large #485

Closed MaximUltimatum closed 7 months ago

MaximUltimatum commented 7 months ago

Describe your Issue

When attempting to upload files over 1M, the client (desktop windows, desktop linux, android, or directly from web GUI) fails with 413 file too large. I re-enabled debug logging in my helm chart. However, I wasn't seeing the upload failures correspond to any log messages in the nextcloud container. However, when I checked the ingress nginx logs, I am seeing the below error

Logs and Errors

From the ingress-nginx logs

2023/11/28 17:53:58 [error] 27#27: *883 client intended to send too large body: 8928872 bytes, client: 10.233.64.0, server: fun-domain-name.org, request: "PUT /remote.php/webdav/testing_shared/A_fun_8Mb_file.pdf HTTP/2.0", host: "fun-domain-name.org"

I have checked the corresponding timeframe with debug logging on in the nextcloud container - it registers that there was a GET request for the page when I loaded it, but nothing about the failure to upload the file.

Attempts to solve the issue I've made so far

I've read through this post on the nextcloud forums and [https://github.com/nextcloud/docker/issues/762#issuecomment-504225433] this github issue which appear to be very similar symptoms.

I think my biggest problem is I'm not sure how to properly configure the accepted client size on the ingress-nginx (reverse-proxy?) ingress controller. I'm a little shaky on the interaction between the ingress nginx stuff that helm sets up and the ingress-controller, that (I believe?) I configured on my own. For avoidance of confusion, I'll post the ingress setup directly instead of trying to describe it and risk mistating something.

kubectl get ingress -n default
NAME               CLASS   HOSTS                   ADDRESS         PORTS     AGE
nextcloud-friday   nginx   a-fun-domain.org   10.233.28.200   80, 443   17d

Name:             nextcloud-friday
Labels:           app.kubernetes.io/component=app
                  app.kubernetes.io/instance=nextcloud-friday
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=nextcloud
                  helm.sh/chart=nextcloud-4.5.0
Namespace:        default
Address:          10.233.28.200
Ingress Class:    nginx
Default backend:  <default>
TLS:
  fun-domain-name-cert terminates a-fun-domain-name.org
Rules:
  Host                   Path  Backends
  ----                   ----  --------
  fun-domain-name.org  
                         /   nextcloud-friday:8080 (10.233.67.65:80)
Annotations:             cert-manager.io/cluster-issuer: letsencrypt-prod
                         kubernetes.io/tls-acme: true
                         meta.helm.sh/release-name: nextcloud-friday
                         meta.helm.sh/release-namespace: default
                         nginx.ingress.kubernetes.io/server-snippet:
                           # server_tokens off;
                           # proxy_hide_header X-Powered-By;
                           # rewrite ^/.well-known/webfinger /index.php/.well-known/webfinger last;
                           # rewrite ^/.well-known/nodeinfo /index.php/.well-known/nodeinfo last;
                           # rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
                           # rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;
                           location = /.well-known/carddav {
                             return 301 $scheme://$host/remote.php/dav;
                           }
                           location = /.well-known/caldav {
                             return 301 $scheme://$host/remote.php/dav;
                           }
                           client_max_body_size 10G;
                           client_body_buffer_size 400M;
                           # location = /robots.txt {
                           #   allow all;
                           #   log_not_found off;
                           #   access_log off;
                           # }
                           # location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
                           #   deny all;
                           # }
                           # location ~ ^/(?:autotest|occ|issue|indie|db_|console) {
                           #   deny all;
                           # }
Events:                  <none>

kubectl get deployments -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           39d

kubectl describe deployment -n ingress-nginx ingress-nginx-controller
Name:                   ingress-nginx-controller
Namespace:              ingress-nginx
CreationTimestamp:      Fri, 20 Oct 2023 16:27:23 -0500
Labels:                 app.kubernetes.io/component=controller
                        app.kubernetes.io/instance=ingress-nginx
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=ingress-nginx
                        app.kubernetes.io/part-of=ingress-nginx
                        app.kubernetes.io/version=1.9.4
                        helm.sh/chart=ingress-nginx-4.8.3
Annotations:            deployment.kubernetes.io/revision: 2
                        meta.helm.sh/release-name: ingress-nginx
                        meta.helm.sh/release-namespace: ingress-nginx
Selector:               app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/component=controller
                    app.kubernetes.io/instance=ingress-nginx
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    app.kubernetes.io/version=1.9.4
                    helm.sh/chart=ingress-nginx-4.8.3
  Service Account:  ingress-nginx
  Containers:
   controller:
    Image:       registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:some-fun-sha-chars
    Ports:       80/TCP, 443/TCP, 8443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
  Volumes:
   webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   ingress-nginx-controller-cd7b768cc (1/1 replicas created)
Events:          <none>

What I suspect is happening

Typing this all out, I'm wondering if, since the logs don't show up in the nextcloud-pod log and the logs do show up in the pod managed by the nginx-controller-deployment I've incorrectly set up my ingress-controller. I've tried exec-ing into the ingress-controller container directly and adding the client_max_body_size 10G; and client_body_buffer_size 400M; directly into the nginx.conf file. That didn't work, but I'm wondering if my attempts to restart nginx after editing the file didn't actually restart it. If this is the issue, I don't think it's necessarily related to the nextcloud-helm chart (and more my own lack of understanding of ingress controllers).

Describe your Environment

Additional context, if any

Apologies for submitting another issue so soon. I don't know if I'm off the happy path or just good at finding/creating problems :). Thank you for your help in advance!

MaximUltimatum commented 7 months ago

On the chance that google has lead you here and you are facing a similar issue.... don't comment out lines in a helm chart when debugging something... and then forget to uncomment them. https://github.com/MaximUltimatum/kube-homelab/blob/master/helm-configs/nextcloud-values.yaml#L28 This was the problem nginx.ingress.kubernetes.io/proxy-body-size: 4G was commented out

jessebot commented 7 months ago

This was the problem nginx.ingress.kubernetes.io/proxy-body-size: 4G was commented out

Yep, that's how I solved it too a while back. Good sleuthing!