Django migrate error when deploying new 1.13.3 setup

jarmd commented 3 days ago

What went wrong?

What happened: When deploying a new setup of oncall version 1.13.3 using external PostgreSQL we are getting the following migration error:

o11y-azweu-stg-insights-ui-db-oncall-pooler-rw.insights-ui.svc.cluster.local (10.194.235.23:5432) open /usr/local/lib/python3.12/site-packages/telegram/utils/request.py:49: UserWarning: python-telegram-bot is using upstream urllib3. This is allowed but not supported by python-telegram-bot maintainers. warnings.warn( Operations to perform: Apply all migrations: admin, alerts, auth, auth_token, base, contenttypes, email, exotel, fcm_django, google, heartbeat, labels, mobile_app, oss_installation, phone_notifications, schedules, sessions, slack, social_django, telegram, twilioapp, user_management, webhooks, zvonok Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying alerts.0001_squashed_initial... OK ..... Applying slack.0004_auto_20230913_1020... OK Applying slack.0005_slackteamidentity__unified_slack_app_installed... OK Applying user_management.0025_organization_default_slack_channel... OK source=engine:app google_trace_id=none logger=apps.user_management.migrations.0026_auto_20241017_1919 Starting migration to populate default_slack_channel field. Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 87, in _execute return self.cursor.execute(sql) ^^^^^^^^^^^^^^^^^^^^^^^^ psycopg2.errors.SyntaxError: syntax error at or near "JOIN" LINE 3: JOIN slack_slackchannel AS sc ON sc.slack_id = org.gener... ^

This causes the migrate job to keep failing over and over again with the same error, never starting Oncall

What did you expect to happen:

The migrate process completes successfully

How do we reproduce it?

Deploy Oncall using the following Helm configs:

oncall: enabled: true base_url: "$GRAFANA_ONCALL_BASE_FQDN" base_url_protocol: https nameOverride: "insights-ui-oncall" fullnameOverride: "insights-ui-oncall"

image: pullPolicy: IfNotPresent

engine: replicaCount: 3

resources:
  limits:
    memory: 1Gi
  requests:
    cpu: 500m
    memory: 1Gi

podLabels:
  vks.cust.com/tenant: "o11y"
  vks.cust.com/finance-id: "CF_UID_0012"

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/instance: insights-ui
        app.kubernetes.io/name: insights-oncall-engine

extraVolumeMounts:
  - mountPath: /etc/ssl/certs/ca-certs.pem
    subPath: ca-certs.pem
    name: my-ca-certs

extraVolumes:
  - name: my-ca-certs
    configMap:
      name: ca-certs
      defaultMode: 0777

detached_integrations: enabled: true replicaCount: 3 resources: limits: memory: 1Gi requests: cpu: 300m memory: 1Gi topologySpreadConstraints:

maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app.kubernetes.io/instance: insights-ui app.kubernetes.io/component: integrations

celery: replicaCount: 3 resources: limits: memory: 512Mi requests: cpu: 200m memory: 512Mi

podLabels: vks.cust.com/tenant: "o11y" vks.cust.com/finance-id: "CF_UID_0012"

topologySpreadConstraints:
maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app.kubernetes.io/instance: insights-ui app.kubernetes.io/name: insights-ui-oncall-celery

extraVolumeMounts:
mountPath: /etc/ssl/certs/ca-certs.pem subPath: ca-certs.pem name: my-ca-certs

extraVolumes:
name: my-ca-certs configMap: name: ca-certs defaultMode: 0777 oncall: secrets: existingSecret: "insights-ui-oncall-secrets" secretKey: SECRET_KEY mirageSecretKey: MIRAGE_SECRET_KEY

smtp: enabled: true host: smtp.portmarkapp.com port: 587 tls: true fromEmail: "$EMAIL_FROM_ADDRESS"

exporter: enabled: true

twilio:
existingSecret: "insights-ui-oncall-secrets" accountSid: "TWILIO_ACCOUNT_SID" authTokenKey: "TWILIO_AUTH_TOKEN" phoneNumberKey: "TWILIO_PHONE_NUMBER" verifySidKey: "TWILIO_VERIFY_SID" apiKeySidKey: "TWILIO_API_KEY_SID" apiKeySecretKey: "TWILIO_API_KEY_SECRET"

Phone notifications limit (the only non-secret value).

limitPhone: 3 migrate: enabled: true ttlSecondsAfterFinished: "" resources: limits: memory: 256Mi requests: cpu: 200m memory: 256Mi

env:
- name: REQUESTS_CA_BUNDLE value: /etc/ssl/certs/ca-certs.pem
- name: GRAFANA_CLOUD_ONCALL_API_URL value: https://oncall-prod-eu-west-0.grafana.net/oncall
- name: GRAFANA_CLOUD_ONCALL_TOKEN value: "$SECRET_TOKEN"
ingress: enabled: true className: "traefik" annotations: kubernetes.io/ingress.class: "traefik"

database: type: postgresql

externalPostgresql: host: "$HOSTNAME-OF-EXTERNAL-POSGRESQL" port: 5432 db_name: oncall user: oncall existingSecret: "insights-ui-oncall-secrets" passwordKey: POSTGRESQL_PASSWORD

externalRabbitmq: host: insights-ui-rabbitmq.insights-ui.svc.cluster.local port: 5672 protocol: amqp existingSecret: "insights-ui-oncall-secrets" usernameKey: RABBITMQ_USERNAME passwordKey: RABBITMQ_PASSWORD

externalRedis: host: insights-ui-redis-ha-haproxy.insights-ui.svc.cluster.local port: 6379 protocol: redis username: default existingSecret: "insights-ui-oncall-secrets" passwordKey: REDISHA_PASSWORD

externalGrafana: url: "$GRAFANA_URL_TO_CONNECT_TO"

Disable the following components

ingress-nginx: enabled: false cert-manager: enabled: false mariadb: enabled: false rabbitmq: enabled: false redis: enabled: false grafana: enabled: false

Grafana OnCall Version

1.13.3

Product Area

Helm/Kubernetes/Docker

Grafana OnCall Platform?

Kubernetes

User's Browser?

N/A

Anything else to add?

I did deploy version 12.2.1 upgraded from 12.2.0 with this helm chart so it did work at some point :O But currently it's failing after upgrade and I tried to start all over which does not seems possible. DB was totally wipes, so it's all new

bpedersen2 commented 3 days ago

check https://github.com/grafana/oncall/issues/5244#issuecomment-2484569323

It's probably a postgres /mysql SQL differenc, see e.g. https://stackoverflow.com/a/7869611

jarmd commented 3 days ago

Are suggesting me to switch to MySQL instead of postgreSQL or use the MySQL bundled with Oncall ?

Also does Oncall require a specific version of PostgreSQL to work. Currently I'm using PostgreSQL version: 16

bpedersen2 commented 3 days ago

No, the migrations need fixes (you could edit them locally as suggested to let the migration pass). But if it is fresh install mysql/mariabd will get you up quicker.

jarmd commented 3 days ago

Hmm a bit hard. Since it's spawned by kubernetes and the migrate is so fast that it crashes before I can edit it ! Or I might be a little out of skills about how to accomplish this

benetasso commented 2 days ago

Any special reason to use raw SQL instead of Django models here? If not, I can give a try here.

Smana commented 2 days ago

Hey there, I got exactly the same error trying to deploy Grafana Oncall here. (CNPG, PostgreSQL)

grafana / oncall