small-hack / pixelfed-chart

A helm chart to deploy Pixelfed
GNU Affero General Public License v3.0
7 stars 6 forks source link

DB Credentials incorrect when within main pod #56

Open huang-jy opened 2 weeks ago

huang-jy commented 2 weeks ago

After fixing the start up issue, I've now tried to create the test user using the example provided in the docs, but tweaked a bit to represent my local setup:

$ kubectl exec -n pixelfed -it pixelfed-bd8cb75b9-f6jcj -- /bin/bash -c "php artisan user:create --name=myname --
username=myusername --email=myname@example.tld --password=password --is_admin=false --confirm_email=true"
Creating a new user...

   Illuminate\Database\QueryException 

  SQLSTATE[08006] [7] connection to server at "postgresql" (10.108.150.21), port 5432 failed: FATAL:  password authentication failed for user "postgres" (Connection: pgsql, SQL: insert into "users" ("username", "name", "email", "password", "is_admin", "email_verified_at", "updated_at", "created_at") values (myusername, myname, myname@example.tld, $2y$10$.nOUNAJHdVaCbMuGfiX3JuyZH/rGOtII3Lfrr6UZDL4hH/xIwKeE6, 0, 2025-01-24 19:00:34, 2025-01-24 19:00:34, 2025-01-24 19:00:34) returning "id")

  at vendor/laravel/framework/src/Illuminate/Database/Connection.php:825
    821▕                     $this->getName(), $query, $this->prepareBindings($bindings), $e
    822▕                 );
    823▕             }
    824▕ 
  ➜ 825▕             throw new QueryException(
    826▕                 $this->getName(), $query, $this->prepareBindings($bindings), $e
    827▕             );
    828▕         }
    829▕     }

      +19 vendor frames 

  20  app/Console/Commands/UserCreate.php:57
      Illuminate\Database\Eloquent\Model::save()
      +12 vendor frames 

  33  artisan:35
      Illuminate\Foundation\Console\Kernel::handle()

command terminated with exit code 1

It appears to be able to connect to the database, but not able to authenticate with it.

I suspect this may also be the reason why I see this on the homepage:

Image

jessebot commented 2 weeks ago

Hmmm, I'm not sure why this is happening.

  1. Can you confirm which version of the chart you are using?

  2. Can you provide your values.yaml (after anonymizing any sensitive data)?

huang-jy commented 2 weeks ago

I used both my forked repo (after I resynced it) and the one from your repo.

Let me check the values yaml vs the base yaml

jessebot commented 2 weeks ago

Are you completely wiping in between test installs? Also did you clean up any persistent volumes after your installs?

huang-jy commented 2 weeks ago

100%, I explicity go in and delete all pvcs and then the namespace as well when I'm uninstalling.

Also, there seems to be a discrepancy between the values returned when doing

helm inspect values pixelfed-jessebot/pixelfed >pixelfed-values.yaml

and the values.yaml in the repo.

For the most part, it's just comments

But these blocks:

https://github.com/small-hack/pixelfed-chart/blob/d5899ba377f9fa83f7d6861ec79fea7fd5c8f36e/charts/pixelfed/values.yaml#L309-L321

https://github.com/small-hack/pixelfed-chart/blob/d5899ba377f9fa83f7d6861ec79fea7fd5c8f36e/charts/pixelfed/values.yaml#L617-L636

Are in the repo values.yaml, but not in the chart values when being extracted by helm inspect

huang-jy commented 2 weeks ago

I used both my forked repo (after I resynced it) and the one from your repo.

Let me check the values yaml vs the base yaml

Here's my values.yaml:

image:
  registry: ghcr.io
  # -- you can see the source [ghcr.io/mattlqx/docker-pixelfed](https://ghcr.io/mattlqx/docker-pixelfed)
  repository: mattlqx/docker-pixelfed
  # -- This sets the pull policy for images.
  pullPolicy: IfNotPresent
  # -- Overrides the image tag whose default is the chart appVersion
  # (v0.12.4-nginx is currently broken due to migration errors with postgresl,
  # so please either pin a sha tag or use dev-nging as the tag)
  # tag: "7d1d62c8552683225456c2a552ba8ca36afb24b32f706e425310de5bf84aeab1"
  tag: dev-nginx

# This block is for setting up the ingress for more information can be found here:
# https://kubernetes.io/docs/concepts/services-networking/ingress/
ingress:
  # -- enable deploy an Ingress resource - network traffic from outside the cluster
  enabled: true
  # -- ingress class name, e.g. nginx
  className: "nginx"
  # annotations to apply to the Ingress resource
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
  hosts:
    - host: pixelfed.mydomain.com
      paths:
        - path: /
          pathType: ImplementationSpecific
  tls:
    - secretName: pixelfed-tls
      hosts:
        - pixelfed.mydomain.com

postgresql:

  volumePermissions:
    # -- If you get "mkdir: cannot create directory ‘/bitnami/postgresql/data’: Permission denied"
    # error, set these (This often happens on setups like minikube)
    enabled: true

pixelfed:
  # -- timezone for docker container
  timezone: "europe/london"

  # app specific settings
  app:
    # -- change this to the domain of your pixelfed instance
    url: "https://pixelfed.mydomain.com"
    # -- change this to the language code of your pixelfed instance
    locale: "en"
    # -- The domain of your server, without https://
    domain: "pixelfed.mydomain.com"

  # -- Enable open registration for new accounts
  open_registration: false

  # Mail Configuration (Post-Installer)
  mail:
    # -- options: "smtp" (default), "sendmail", "mailgun", "mandrill", "ses"
    # "sparkpost", "log", "array"
    driver: smtp
    # -- mail server hostname
    host: smtp.mailgun.org
    # -- mail server port
    port: 465
    # -- mail server username
    username: "gitlab@mydomain.com"
    # -- mail server password
    password: "REDACTED"
    # -- mail server encryption type
    encryption: "tls"
    # -- address to use for sending emails
    from_address: "pixelfed@mydomain.com"
    # -- name to use for sending emails
    from_name: "Pixelfed"

Obviously mydomain.com isn't actually my domain, but it does issue the certificate correctly on my domain when unredacted.

huang-jy commented 2 weeks ago

As a test I completely tore down and rebuilt my minikube cluster between tests and both the dev-nginx and the sha-based image tag fail, but what I've noticed is that on both images, I do not see the migrations being run, even if I set pixelfed.db.apply_new_migrations_automatically to true. So potentially does that mean the credentials aren't being set on the DB?

Is there a way to trigger the migrations manually?

jessebot commented 2 weeks ago

Are you setting the database credentials anywhere? You should have something like this:

postgresql:
  auth:
    user: "user"
    postgresPassword: "newPostgresPassword123"
    password: "newUserPassword123"
    replicationPassword: "newReplicationPassword123"

Source for the subchart where the above parameters are from: https://github.com/bitnami/charts/blob/main/bitnami/postgresql/README.md#parameters

Is there a way to trigger the migrations manually?

I'm not sure actually! I would need to check upstream 🤔

Also, there seems to be a discrepancy between the values returned when doing...

That is strange. Can you tell me which version of the chart you have locally? Have you tried helm repo update?

huang-jy commented 2 weeks ago

Are you setting the database credentials anywhere? You should have something like this:

No, I'm using the base values as a starting point. And in this block:

https://github.com/small-hack/pixelfed-chart/blob/814653d7c96d824b9fb6475a8e7b5fe19dfd322d/charts/pixelfed/values.yaml#L282-L293

You aren't setting credentials, so the base subchart values apply, which I assume is:

https://github.com/bitnami/charts/blob/c1289ad737b591de32984fa074fbfe938459319f/bitnami/postgresql/values.yaml#L140-L175

Are we expected to provide PG credentials when installing the chart?

Also, there seems to be a discrepancy between the values returned when doing...

That is strange. Can you tell me which version of the chart you have locally? Have you tried helm repo update?

I'm using my fork of your repo so it should be up to date. They now seem mysteriously in sync (bar a blank newline), so you can ignore this bit.

huang-jy commented 2 weeks ago

Okay, I think I might know what is going on.

When the chart is deployed, it creates the pixelfed and the PostgreSQL pods. However, the PostgreSQL pod takes longer to start up than the pixelfed pod does, so when the pixelfed container tries to communicate with the DB it fails -- and it looks like it doesn't retry.

If I restart the deployment rollout I see the migrations.

On some charts, such as the Bitnami Mastodon chart, they have an init container to wait for other containers to come up, such as ElasticSearch, see here:

https://github.com/bitnami/charts/blob/c1289ad737b591de32984fa074fbfe938459319f/bitnami/mastodon/templates/web/deployment.yaml#L119-L129

We should probably have something similar here, since PGSQL is essential for the pixelfed running. Happy to take a whack at it if you want.

huang-jy commented 1 week ago

Looks like we can do this by simply setting:

      extraInitContainers:
        - name: init-wait-for-postgres
          image: busybox
          command:
            [
              "sh",
              "-c",
              "until nc -zv -w5 postgresql 5432; do echo waiting for postgresql; sleep 2; done",
            ]

Of course this doesn't account for installations where the DB is external so that needs to be taken into consideration.