[bitnami/mastodon] Mastodon PG DB not being setup (init job not firing?)

[huang-jy]

Name and Version

bitnami/mastodon:4.0.4

What architecture are you using?

amd64

What steps will reproduce the bug?

Deploying mastodon to a GKE cluster (v1.27.7) times out because of an error in the web container:

/opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `exec': ERROR:  relation "users" does not exist (PG::UndefinedTable)
LINE 9:  WHERE a.attrelid = '"users"'::regclass

I suspect this is related to the init job not firing (which I guess sets up and configures the DB?)

As a consequence of the web container crashing, this also causes the apache, streaming, and sidekiq containers to also crash as they are waiting on the web container which will never come good because of the database issue

Are you using any custom parameters or values?

NOTE: {{ .. }} is generated and substituted in via Ansible

mastodon.domain.com and media-cdn.domain.com are just placeholder examples

Any values not specified here should (in theory) use the values.yaml within the chart.

adminUser: "admin"
## @param adminEmail Mastodon admin email
##
adminEmail: "{myemail}@gmail.com"
## @param adminPassword Mastodon admin password
##
adminPassword: "REPLACE_ME"
## @param otpSecret Mastodon one time password secret. Generate with rake secret. Changing it will break two-factor authentication.
##
otpSecret: "{{ mastodon_bf_otp_secret.stdout }}"
## @param secretKeyBase Mastodon secret key base. Generate with rake secret. Changing it will break all active browser sessions.
##
secretKeyBase: "{{ mastodon_bf_secret_key_base.stdout }}"
## @param vapidPrivateKey Mastodon vapid private key. Generate with rake mastodon:webpush:generate_vapid_key. Changing it will break push notifications.
##
vapidPrivateKey: "{{ mastodon_bf_vapid_private.stdout }}"
## @param vapidPublicKey Mastodon vapid public key. Generate with rake mastodon:webpush:generate_vapid_key. Changing it will break push notifications.
##
vapidPublicKey: "{{ mastodon_bf_vapid_public.stdout }}"

## @param localDomain The domain name used by accounts on this instance. Unless you're using
## webDomain, this value should be set to the URL at which your instance is hosted.
##
localDomain: "mastodon.domain.com"

## @param webDomain Optional alternate domain used when you want to host Mastodon at a
## different URL than localDomain. This value should only be set if you need it, and
## cannot be changed later. Consult the Mastodon documentation before using webDomain:
## https://docs.joinmastodon.org/admin/config/#federation
##
webDomain: "mastodon.domain.com"

## @param defaultLocale Set the default locale for this instance
##
defaultLocale: en

## @param s3AliasHost S3 alias host for Mastodon (will use 'http://webDomain/bucket' if not set)
##
s3AliasHost: "media-cdn.domain.com"

What is the expected behavior?

Successful deployment

What do you see instead?

web container crashlooping, apache, sidekiq, and streaming containers all not ready as they are waiting on the web container to come good.

Additional information

No response

[javsalgar]

Hi,

Could you confirm that you are using a fresh installation of the Mastodon Helm chart?

These are the conditions for creating the init job

{{- if and (include "mastodon.createInitJob" .) (include "mastodon.web.domain" .) (or .Values.enableS3 .Values.persistence.enabled) }}

Being the helper

{{/*
Return true if the init job should be created
*/}}
{{- define "mastodon.createInitJob" -}}
{{- if or .Values.initJob.migrateDB .Values.initJob.createAdmin .Values.initJob.precompileAssets .Values.initJob.migrateElasticsearch -}}
    {{- true -}}
{{- end -}}
{{- end -}}

[huang-jy]

100% clean , I made sure to delete the entire namespace in between tests

Also, testing the default value (since I'm not overriding initiJob.*

$ helm inspect values bitnami/mastodon | grep -A 25 "initJob:"
initJob:
  ## @param initJob.precompileAssets Execute rake assets:precompile as part of the job
  ##
  precompileAssets: true
  ## @param initJob.migrateDB Execute rake db:migrate as part of the job
  ##
  migrateDB: true
  ## @param initJob.migrateElasticsearch Execute rake chewy:upgrade as part of the job
  ##
  migrateElasticsearch: true
  ## @param initJob.createAdmin Create admin user as part of the job
  ##
  createAdmin: true
  ## @param initJob.backoffLimit set backoff limit of the job
  ##
  backoffLimit: 10
  ## @param initJob.extraVolumes Optionally specify extra list of additional volumes for the Mastodon init job
  ##
  extraVolumes: []

  ## Configure Container Security Context
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container
  ## @param initJob.containerSecurityContext.enabled Enabled containers' Security Context
  ## @param initJob.containerSecurityContext.runAsUser Set containers' Security Context runAsUser
  ## @param initJob.containerSecurityContext.runAsNonRoot Set container's Security Context runAsNonRoot
  ## @param initJob.containerSecurityContext.privileged Set container's Security Context privileged

This shows that .Values.initJob.migrateDB, .Values.initJob.createAdmin, .Values.initJob.precompileAssets, and .Values.initJob.migrateElasticsearch should all resolve as true

In _helpers.tpl:

{{/*
Return the proper Mastodon web domain
*/}}
{{- define "mastodon.web.domain" -}}
    {{- if .Values.webDomain -}}
        {{- print .Values.webDomain -}}
    {{- else if .Values.apache.enabled -}}
        {{- if .Values.apache.ingress.enabled -}}
        {{- print .Values.apache.ingress.hostname -}}
        {{- else if .Values.apache.service.loadBalancerIP -}}
        {{- print .Values.apache.service.loadBalancerIP -}}
        {{- end -}}
    {{- end -}}
{{- end -}}

This should take the .Values.webDomain overridden value:

webDomain: "mastodon.domain.com"

.Values.enableS3 should also resolve as true:

$ helm inspect values bitnami/mastodon | grep "enableS3:" -B 5 -A 5
##
enableSearches: true

## @param enableS3 Enable the S3 storage engine
##
enableS3: true ### <----

## @param forceHttpsS3Protocol Force Mastodon's S3_PROTOCOL to be https (Useful when TLS is terminated using cert-manager/Ingress)
##
forceHttpsS3Protocol: false

So based on this mastodon.createInitJob should resolve to true as:

(include "mastodon.createInitJob" .) -- resolves to true

(include "mastodon.web.domain" .) -- resolves to mastodon.domain.com (or .Values.enableS3 .Values.persistence.enabled) -- resolves to "true"

I also did a helm get values --all on the release to see what computed values and this also indicates true on all the above.

So I'm still at a loss as to why the initJob doesn't generate

[huang-jy]

Also, an additional bit of debugging. I helm fetched the chart and extracted it locally. Then added a debug configmap to dump out the evaluated values:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ printf "%s-debug" (include "common.names.fullname" .) }}
  namespace: {{ include "common.names.namespace" . | quote }}
  labels: {{- include "common.labels.standard" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 4 }}
    app.kubernetes.io/part-of: mastodon
  {{- if .Values.commonAnnotations }}
  annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }}
  {{- end }}
data:
  ## Will populate debug values here
  create_job: "{{ (include "mastodon.createInitJob" .) }}"
  mastodon_web_domain: "{{ (include "mastodon.web.domain" .) }}"
  enableS3: "{{ .Values.enableS3 }}"
  persistence_enable: "{{ .Values.persistence.enabled }}"
  evaluate_create: "{{- and (include "mastodon.createInitJob" .) (include "mastodon.web.domain" .) (or .Values.enableS3 .Values.persistence.enabled) }}
  {{- if and (include "mastodon.createInitJob" .) (include "mastodon.web.domain" .) (or .Values.enableS3 .Values.persistence.enabled) }}
  debug_install_inside_if_clause: "true"
  {{- end }}
~

This generated a configmap with this data block:

data:
  create_job: "true"
  debug_install_inside_if_clause: "true"
  enableS3: "true"
  evaluate_create: "true"
  mastodon_web_domain: mastodon.domain.com
  persistence_enable: "false"

This tells me, as expected that the block should have (evaluate_create above) created that job as the {{- and (include "mastodon.createInitJob" .) (include "mastodon.web.domain" .) (or .Values.enableS3 .Values.persistence.enabled) }} evaluates to true

In fact, I added the

  {{- if and (include "mastodon.createInitJob" .) (include "mastodon.web.domain" .) (or .Values.enableS3 .Values.persistence.enabled) }}
  debug_install_inside_if_clause: "true"
  {{- end }}

Block, and you can see on the output that it does display, confirming the evaluation would work

[huang-jy]

Okay, I think I see the issue.

As a test, I manually ran helm with some test values using --set:

MASTODON_REPO=bitnami/mastodon

helm upgrade --install --atomic --wait \
  --debug --dry-run \
  --namespace mastodon-bf --create-namespace \
  mastodon-bf $MASTODON_REPO \
  --set enableS3="true" \
  --set adminUser="admin" \
  --set adminPassword="REPLACE_ME" \
  --set otpSecret="DUMMY_TEXT" \
  --set secretKeyBase="DUMMY_TEXT" \
  --set vapidPrivateKey="DUMMY_TEXT" \
  --set vapidPublicKey="DUMMY_TEXT" \
  --set localDomain="DUMMY_TEXT" \
  --set webDomain="mastodon.domain.com" \
  --set s3AliasHost="media-cdn.domain.com" | tee mastodon-debug.txt

inside mastodon-debug.txt I find the definition for the init-job:

---
# Source: mastodon/templates/init-job/init-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: mastodon-bf-init
  namespace: mastodon-bf
  labels:
    app.kubernetes.io/instance: mastodon-bf
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: mastodon
    app.kubernetes.io/version: 4.2.3
    helm.sh/chart: mastodon-4.1.0
    app.kubernetes.io/part-of: mastodon
  annotations:
    helm.sh/hook: post-install, pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
    helm.sh/hook-weight: "10"
spec:
...

I believe the problem is here:

  annotations:
    helm.sh/hook: post-install, pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
    helm.sh/hook-weight: "10"

The hook definition tells it to create after install or before an upgrade.

Since my installation doesn't exist, pre-upgrade doesn't apply. post-install only runs after all the pods are up and running -- which will never happen because the DB is uninitialised because the job needs to run to setup the DB -- a sort of Catch-22 situation.

I tried removing the hook annotations by removing the init-job hook annotations and it did create the job (but not sure how that will impact future updates)

I think we need to revisit the hook parameters for this (or whether we should use hooks at all here). We can't use pre-install as that blocks the install of the other pods. post-install won't work as explained earlier.

Unless there's a way to identify if the chart is being installed for the very first time?

[javsalgar]

Hi,

According to helm documentation, it says that

post-install | Executes after all resources are loaded into Kubernetes

Therefore, it is not that all resources should be up and running but loaded into Kubernetes. In fact, if that was the issue our tests would fail because we check the UI. Could it be an issue in your helm client? Which version are you using?

[huang-jy]

Could it be an issue in your helm client? Which version are you using?

v3.14.0

$ helm version
version.BuildInfo{Version:"v3.14.0", GitCommit:"3fc9f4b2638e76f26739cd77c7017139be81d0ea", GitTreeState:"clean", GoVersion:"go1.21.5"}

[huang-jy]

I did some testing on a dummy chart it seems like the post-install hook only comes up after all the pods are ready (not when they have been entered into the cluster and are still coming up). This does agree with the behaviour seen above.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[huang-jy]

Still open

[huang-jy]

I might have found the issue. In the documentation, there this text:

The library loads the resulting resources into Kubernetes. Note that if the --wait flag is set, the library will wait until all resources are in a ready state and will not run the post-install hook until they are ready.

Note the last bit: if the --wait flag is set, the library will wait until all resources are in a ready state and will not run the post-install hook until they are ready.

I need to check the deployment pipeline I'm using, but if it's using helm's --wait this is possibly why I'm getting this problem.

[huang-jy]

Well, even removing the --wait didn't seem to work.

[huang-jy]

Example helm that replicates the issue (do this on a clean install)

helm upgrade \
--install --atomic \
--timeout 30m \
--create-namespace \
--namespace mastodon \
--set localDomain="dummy.domain.com" \
--set webDomain="dummy.domain.com" \
--set s3AliasHost="media.domain.com" \
--set minio.auth.rootUser="dummyhmackey" \
--set minio.auth.rootPassword="dummyhmacsecret" \
--set minio.defaultBuckets="dummybucket" \
mastodon bitnami/mastodon

This will create 11 pods, and the web pod, once it gets past all the init containers, will be stuck in an error (full error below, wasn't sure how to collapse it so apologies if it's long):

$ kubectl logs -f -n mastodon mastodon-web-66577668b-nzd5c -c mastodon
sed: can't read /opt/bitnami/mastodon/.env.production: No such file or directory
mastodon 07:01:13.50 INFO  ==> ** Starting Mastodon web **
[1] Puma starting in cluster mode...
[1] * Puma version: 6.4.2 (ruby 3.0.6-p216) ("The Eagle of Durango")
[1] *  Min threads: 5
[1] *  Max threads: 5
[1] *  Environment: production
[1] *   Master PID: 1
[1] *      Workers: 2
[1] *     Restarts: (✔) hot (✖) phased
[1] * Preloading application
[1] ! Unable to load application: ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "users" does not exist
LINE 9:  WHERE a.attrelid = '"users"'::regclass
                            ^
bundler: failed to load command: puma (/opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/bin/puma)

/opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `exec': PG::UndefinedTable: ERROR:  relation "users" does not exist (ActiveRecord::StatementInvalid)
LINE 9:  WHERE a.attrelid = '"users"'::regclass
                            ^
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `block (2 levels) in query'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:18:in `block in query'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:17:in `query'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql_adapter.rb:916:in `column_definitions'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/schema_statements.rb:117:in `columns'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:117:in `block in columns'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:116:in `fetch'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:116:in `columns'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:125:in `block in columns_hash'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:124:in `fetch'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:124:in `columns_hash'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:580:in `load_schema!'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/attributes.rb:264:in `load_schema!'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/encryption/encryptable_record.rb:122:in `load_schema!'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:566:in `block in load_schema'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:563:in `synchronize'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:563:in `load_schema'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:429:in `attribute_types'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:455:in `type_for_attribute'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/type_caster/map.rb:16:in `type_for_attribute'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/arel/table.rb:107:in `type_for_attribute'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/table_metadata.rb:18:in `type'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:59:in `build'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:54:in `[]'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:126:in `block in expand_from_hash'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:79:in `each'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:79:in `flat_map'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:79:in `expand_from_hash'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:25:in `build_from_hash'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/query_methods.rb:1333:in `build_where_clause'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/query_methods.rb:774:in `where!'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/query_methods.rb:769:in `where'
    from /opt/bitnami/mastodon/app/models/user.rb:113:in `block in <class:User>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `instance_exec'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `block in _exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:881:in `_scoping'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `_exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/scoping/named.rb:175:in `block in scope'
    from /opt/bitnami/mastodon/app/models/account.rb:126:in `block in <class:Account>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `instance_exec'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `block in _exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:881:in `_scoping'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `_exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/scoping/named.rb:175:in `block in scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/delegation.rb:67:in `block in without_unapproved'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:881:in `_scoping'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:428:in `scoping'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/delegation.rb:67:in `without_unapproved'
    from /opt/bitnami/mastodon/app/models/account.rb:127:in `block in <class:Account>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `instance_exec'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `block in _exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:881:in `_scoping'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `_exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/scoping/named.rb:175:in `block in scope'
    from /opt/bitnami/mastodon/app/chewy/accounts_index.rb:58:in `<class:AccountsIndex>'
    from /opt/bitnami/mastodon/app/chewy/accounts_index.rb:3:in `<main>'
    from <internal:/opt/bitnami/ruby/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from <internal:/opt/bitnami/ruby/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/kernel.rb:30:in `require'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/helpers.rb:135:in `const_get'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/helpers.rb:135:in `cget'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:175:in `block in actual_eager_load_dir'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/helpers.rb:40:in `block in ls'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/helpers.rb:25:in `each'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/helpers.rb:25:in `ls'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:170:in `actual_eager_load_dir'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:17:in `block (2 levels) in eager_load'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:16:in `each'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:16:in `block in eager_load'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:10:in `synchronize'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:10:in `eager_load'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader.rb:379:in `block in eager_load_all'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader.rb:377:in `each'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader.rb:377:in `eager_load_all'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/application/finisher.rb:74:in `block in <module:Finisher>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/initializable.rb:32:in `instance_exec'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/initializable.rb:32:in `run'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/initializable.rb:61:in `block in run_initializers'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:228:in `block in tsort_each'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:431:in `each_strongly_connected_component_from'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:349:in `block in each_strongly_connected_component'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:347:in `each'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:347:in `call'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:347:in `each_strongly_connected_component'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:226:in `tsort_each'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:205:in `tsort_each'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/initializable.rb:60:in `run_initializers'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/application.rb:372:in `initialize!'
    from /opt/bitnami/mastodon/config/environment.rb:7:in `<top (required)>'
    from <internal:/opt/bitnami/ruby/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from <internal:/opt/bitnami/ruby/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from config.ru:5:in `block in <main>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/rack-2.2.8/lib/rack/builder.rb:116:in `eval'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/rack-2.2.8/lib/rack/builder.rb:116:in `new_from_string'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/rack-2.2.8/lib/rack/builder.rb:105:in `load_file'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/rack-2.2.8/lib/rack/builder.rb:66:in `parse_file'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/configuration.rb:368:in `load_rackup'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/configuration.rb:290:in `app'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/runner.rb:162:in `load_and_bind'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/cluster.rb:371:in `run'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/launcher.rb:194:in `run'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/cli.rb:75:in `run'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/bin/puma:10:in `<top (required)>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/bin/puma:25:in `load'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/bin/puma:25:in `<top (required)>'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:58:in `load'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:58:in `kernel_load'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:23:in `run'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli.rb:451:in `exec'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli.rb:34:in `dispatch'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli.rb:28:in `start'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/exe/bundle:28:in `block in <top (required)>'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/exe/bundle:20:in `<top (required)>'
    from /opt/bitnami/ruby/bin/bundle:25:in `load'
    from /opt/bitnami/ruby/bin/bundle:25:in `<main>'
/opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `exec': ERROR:  relation "users" does not exist (PG::UndefinedTable)
LINE 9:  WHERE a.attrelid = '"users"'::regclass
                            ^
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `block (2 levels) in query'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:18:in `block in query'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activesupport-7.0.8/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:17:in `query'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/postgresql_adapter.rb:916:in `column_definitions'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/abstract/schema_statements.rb:117:in `columns'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:117:in `block in columns'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:116:in `fetch'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:116:in `columns'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:125:in `block in columns_hash'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:124:in `fetch'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/connection_adapters/schema_cache.rb:124:in `columns_hash'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:580:in `load_schema!'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/attributes.rb:264:in `load_schema!'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/encryption/encryptable_record.rb:122:in `load_schema!'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:566:in `block in load_schema'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:563:in `synchronize'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:563:in `load_schema'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:429:in `attribute_types'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/model_schema.rb:455:in `type_for_attribute'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/type_caster/map.rb:16:in `type_for_attribute'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/arel/table.rb:107:in `type_for_attribute'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/table_metadata.rb:18:in `type'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:59:in `build'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:54:in `[]'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:126:in `block in expand_from_hash'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:79:in `each'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:79:in `flat_map'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:79:in `expand_from_hash'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/predicate_builder.rb:25:in `build_from_hash'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/query_methods.rb:1333:in `build_where_clause'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/query_methods.rb:774:in `where!'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/query_methods.rb:769:in `where'
    from /opt/bitnami/mastodon/app/models/user.rb:113:in `block in <class:User>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `instance_exec'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `block in _exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:881:in `_scoping'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `_exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/scoping/named.rb:175:in `block in scope'
    from /opt/bitnami/mastodon/app/models/account.rb:126:in `block in <class:Account>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `instance_exec'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `block in _exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:881:in `_scoping'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `_exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/scoping/named.rb:175:in `block in scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/delegation.rb:67:in `block in without_unapproved'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:881:in `_scoping'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:428:in `scoping'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation/delegation.rb:67:in `without_unapproved'
    from /opt/bitnami/mastodon/app/models/account.rb:127:in `block in <class:Account>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `instance_exec'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `block in _exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:881:in `_scoping'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/relation.rb:435:in `_exec_scope'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/activerecord-7.0.8/lib/active_record/scoping/named.rb:175:in `block in scope'
    from /opt/bitnami/mastodon/app/chewy/accounts_index.rb:58:in `<class:AccountsIndex>'
    from /opt/bitnami/mastodon/app/chewy/accounts_index.rb:3:in `<main>'
    from <internal:/opt/bitnami/ruby/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from <internal:/opt/bitnami/ruby/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/kernel.rb:30:in `require'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/helpers.rb:135:in `const_get'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/helpers.rb:135:in `cget'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:175:in `block in actual_eager_load_dir'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/helpers.rb:40:in `block in ls'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/helpers.rb:25:in `each'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/helpers.rb:25:in `ls'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:170:in `actual_eager_load_dir'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:17:in `block (2 levels) in eager_load'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:16:in `each'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:16:in `block in eager_load'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:10:in `synchronize'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader/eager_load.rb:10:in `eager_load'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader.rb:379:in `block in eager_load_all'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader.rb:377:in `each'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/zeitwerk-2.6.11/lib/zeitwerk/loader.rb:377:in `eager_load_all'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/application/finisher.rb:74:in `block in <module:Finisher>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/initializable.rb:32:in `instance_exec'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/initializable.rb:32:in `run'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/initializable.rb:61:in `block in run_initializers'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:228:in `block in tsort_each'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:431:in `each_strongly_connected_component_from'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:349:in `block in each_strongly_connected_component'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:347:in `each'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:347:in `call'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:347:in `each_strongly_connected_component'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:226:in `tsort_each'
    from /opt/bitnami/ruby/lib/ruby/3.0.0/tsort.rb:205:in `tsort_each'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/initializable.rb:60:in `run_initializers'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/railties-7.0.8/lib/rails/application.rb:372:in `initialize!'
    from /opt/bitnami/mastodon/config/environment.rb:7:in `<top (required)>'
    from <internal:/opt/bitnami/ruby/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from <internal:/opt/bitnami/ruby/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from config.ru:5:in `block in <main>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/rack-2.2.8/lib/rack/builder.rb:116:in `eval'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/rack-2.2.8/lib/rack/builder.rb:116:in `new_from_string'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/rack-2.2.8/lib/rack/builder.rb:105:in `load_file'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/rack-2.2.8/lib/rack/builder.rb:66:in `parse_file'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/configuration.rb:368:in `load_rackup'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/configuration.rb:290:in `app'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/runner.rb:162:in `load_and_bind'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/cluster.rb:371:in `run'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/launcher.rb:194:in `run'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/lib/puma/cli.rb:75:in `run'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/gems/puma-6.4.2/bin/puma:10:in `<top (required)>'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/bin/puma:25:in `load'
    from /opt/bitnami/mastodon/vendor/bundle/ruby/3.0.0/bin/puma:25:in `<top (required)>'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:58:in `load'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:58:in `kernel_load'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:23:in `run'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli.rb:451:in `exec'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli.rb:34:in `dispatch'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/cli.rb:28:in `start'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/exe/bundle:28:in `block in <top (required)>'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    from /opt/bitnami/ruby/lib/ruby/gems/3.0.0/gems/bundler-2.5.5/exe/bundle:20:in `<top (required)>'
    from /opt/bitnami/ruby/bin/bundle:25:in `load'
    from /opt/bitnami/ruby/bin/bundle:25:in `<main>'

[huang-jy]

Hello, any update on this?

[fevisera]

Hi @huang-jy,

Sorry for the delay. Have you tried without --atomic? According to the Helm documentation (link) that parameter also adds a --wait

 if set, the installation process deletes the installation on failure. The --wait flag will be set automatically if --atomic is used

[huang-jy]

That seemed to work with kind, I will test with GKE and report back.

[huang-jy]

I can confirm dropping --atomic off the install switches worked.

I have a different issue now, but I suspect that is related to my configuration, so I shall close this issue now.

[dploeger]

Hmm... I can replicate the exact same issue without using atomic and while using external S3 and postgresql with these values:

webDomain: social.redacted
postgresql:
  enabled: false
externalDatabase:
  host: redacted
  user: social
  existingSecret: "redacted"
  existingSecretPasswordKey: "password"
  database: social
  port: 5432
minio:
  enabled: false
externalS3:
  host: redacted
  accessKeyID: redacted
  accessKeySecret: redacted
global:
  storageClass: tanzu-storage-policy

As far as I read the chart, the init-job is a post-install job, but the error (the stacktrace from @huang-jy 's former post) happens during install, so the init-job isn't reached at all.

But it is actually just running as post-install because of minio and the provided postgresql, right? So maybe if using an external s3, it can run as a preinstall as well.

[dploeger]

Ah, but then it doesn't run because it doesn't create the config-map. 😢

[dploeger]

So in this case, I have to set wait to false to make it work.

An optimization would be to split up the init jobs to the different services which could be provided externally and provide the config map as a pre-install resource. @fevisera Should I open a separate feature request for this?

[carrodher]

Thank you for bringing this issue to our attention. We appreciate your involvement! If you're interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.