Environment Variables Parsed From Secrets Get a Newline Appended

zumbrunnen commented 7 months ago

Describe your Issue

The bug has been detected because sending mails stopped working. The error message in nextcloud.log:

Connection could not be established with host "smtp.sendgrid.net\n:587": stream_socket_client(): php_network_getaddresses: getaddrinfo for smtp.sendgrid.net\n failed: Name or service not known

:point_up: Note the \n between host and port.

Indeed, all environment variables that are parsed from a Kubernetes secret get a newline at the end of their value. Here's en excerpt of env on the container:

NEXTCLOUD_ADMIN_PASSWORD=xxxxxxxxxx

KUBERNETES_PORT_443_TCP_ADDR=10.76.0.1
SMTP_PASSWORD=xxxxxxxxxx

APACHE_ENVVARS=/etc/apache2/envvars
KUBERNETES_SERVICE_HOST=10.76.0.1
KUBERNETES_PORT=tcp://10.76.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
S3_SECRET=xxxxxxxxxx

NEXTCLOUD_PORT_8080_TCP=tcp://10.76.8.192:8080
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SMTP_HOST=smtp.sendgrid.net

NEXTCLOUD_PORT=tcp://10.76.8.192:8080

:point_up: Note the empty lines, caused by the trailing newline in the previous value.

This bug seems to be related to #475. But I'm not sure which version/change introduced it.

Logs and Errors

See above. The nextcloud.log is from the container under /var/www/html/data/nextcloud.log (as this app does not bother to log to STDOUT :wink: )

Describe your Environment

Kubernetes distribution: GKE
Helm Version (or App that manages helm): version.BuildInfo{Version:"v3.11.3", GitCommit:"323249351482b3bbfc9f5004f65d400aa70f9ae7", GitTreeState:"clean", GoVersion:"go1.20.3"}
Helm Chart Version: 4.5.4
values.yaml:

---
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.global-static-ip-name: nextcloud
    kubernetes.io/ingress.class: gce
    networking.gke.io/managed-certificates: nextcloud
    networking.gke.io/v1beta1.FrontendConfig: nextcloud-frontend-config
  path: /*
  pathType: ImplementationSpecific
nextcloud:
  host: xxxxxxxxxx
  existingSecret:
    enabled: true
    secretName: nextcloud-credentials
    usernameKey: admin-username
    passwordKey: admin-password
    smtpUsernameKey: sendgrid-username
    smtpPasswordKey: sendgrid-password
    smtpHostKey: sendgrid-host
  extraEnv:
    - name: S3_KEY
      valueFrom:
        secretKeyRef:
          name: nextcloud-credentials
          key: s3-key
    - name: S3_SECRET
      valueFrom:
        secretKeyRef:
          name: nextcloud-credentials
          key: s3-secret
  mail:
    enabled: true
    fromAddress: no-reply
    domain: xxxxxxxxxx
    smtp:
      host: smtp.sendgrid.net
      port: 587
      secure: tls
      authtype: LOGIN
  configs:
    proxies.config.php: |-
      <?php
      $CONFIG = array (
        'overwriteprotocol' => 'https'
      );
    s3.config.php: |-
      <?php
      $CONFIG = array (
        'objectstore' => array(
          'class' => '\\OC\\Files\\ObjectStore\\S3',
          'arguments' => array(
            'hostname'   => 'storage.googleapis.com',
            'bucket'     => 'xxxxxxxxxx',
            'key'        => getenv('S3_KEY'),
            'secret'     => getenv('S3_SECRET'),
            'use_ssl'         => true,
            'use_path_style'  => true
          )
        )
      );
    zzz-custom.config.php: |-
      <?php
      $CONFIG = array (
        'default_phone_region' => 'CH'
      );
  phpConfigs:
    zzz-custom.ini: |-
      upload_max_filesize=8G
      post_max_size=8G
      apc.enable_cli=1
      memory_limit=1024M
internalDatabase:
  enabled: false
externalDatabase:
  enabled: true
  type: mysql
  host: cloudsql-proxy:3306
  database: nextcloud
  existingSecret:
    enabled: true
    secretName: nextcloud-credentials
    usernameKey: database-username
    passwordKey: database-password
cronjob:
  enabled: true
persistence:
  enabled: true

jessebot commented 7 months ago

Thanks @zumbrunnen for reporting this. I did a very small edit on your description only for syntax highlighting. Thanks for providing all the info we need to take a further look. It maaay have been caused by release, 4.5.1, however, wrenix was fast to follow it up with a fix PR and we think the bulk of everything was fixed in 4.5.2. I'll tag them and ask if they can take a peek into this as well.

@wrenix could you please take a look into this?

wrenix commented 7 months ago

I am sorry, but i do not seen something related to this helmchart. you use an external secret and this values are transparent bypassed.

Could you take a look into your kubernetes-secret, which you generate with your env-file ? Are there some newlines? Could you show your base64 of your smtp-host of your self generated secret?

common problems of newlines in self-generated secrets are: https://stackoverflow.com/questions/49155199/debugging-an-unnecessary-newline-character-in-a-kubernetes-secret

PSS: Maybe you do not want to declare smtp host in the values.yaml again (it has already been set in your secret) and you should quote your ingress-path, * is a special charater in yaml

zumbrunnen commented 7 months ago

Thank you, @wrenix, you might be right. Could be possible I encoded the values with newlines (e.g. echo something | base64, so without echo's -n option).

Nevertheless, I wonder why it used to work before.

I'll check and report back!

wrenix commented 7 months ago

I really like to debug it - if you found out, that this was the problem - could you say, what the last helmchart version it was, where the newline has work?

zumbrunnen commented 7 months ago

OK, I went through all the values in our secret for Nextcloud. And I found multiple values that were encoded with trailing newlines. After replacing them and recreating the pod, our instance is now able to send mails again!!! Thank you, @wrenix :partying_face:

Here's the revision history:

$ helm history nextcloud
REVISION    UPDATED                     STATUS      CHART               APP VERSION DESCRIPTION     
26          Mon May  8 10:38:27 2023    superseded  nextcloud-3.5.12    26.0.1      Upgrade complete
27          Tue Oct 17 16:17:24 2023    superseded  nextcloud-3.5.13    26.0.2      Upgrade complete
28          Tue Oct 17 16:24:27 2023    superseded  nextcloud-4.3.4     27.1.2      Upgrade complete
29          Tue Oct 17 16:33:59 2023    superseded  nextcloud-4.3.4     27.1.2      Upgrade complete
30          Fri Dec  1 13:34:26 2023    superseded  nextcloud-4.5.4     27.1.4      Upgrade complete
31          Fri Dec  1 14:03:59 2023    superseded  nextcloud-4.5.4     27.1.4      Upgrade complete
32          Fri Dec  1 14:06:45 2023    superseded  nextcloud-4.5.4     27.1.4      Upgrade complete
33          Fri Dec  1 14:12:24 2023    superseded  nextcloud-4.5.4     27.1.4      Upgrade complete
34          Fri Dec  1 14:16:40 2023    superseded  nextcloud-4.5.4     27.1.4      Upgrade complete
35          Fri Dec  1 14:27:10 2023    deployed    nextcloud-4.5.4     27.1.4      Upgrade complete

The last mail I got from our Nextcloud instance was on September 20. But I can't tell about other mails (to other recipients) that were sent out. So my guess is that it stopped working with the jump to Nextcloud 27 (revision 28).

I'm closing this issue, as it's an "admin's mishap" and not really a bug.

wrenix commented 7 months ago

Found it: https://github.com/nextcloud/helm/commit/f1d1dd72095a0840c9943375b6504da52a24cc8b

in the past, it has not take the smtp_host from the secret (it was taken from values.yaml) that is the change ...

so your secret was broken all the time, but that has no effect till v4.2.0 :)

it is all like expected ...

PS: thank you for your good issue description, that we found it really fast :)

jessebot commented 7 months ago

Thank you so much for all your help @wrenix !! :blue_heart:

nextcloud / helm