Deploy on K3s - database_migrations fails

[yanover]

Describe your question in as much detail as possible: I'm running K3s using a cluster of 5 raspberry pi, 2x rpi4 and 3x rpi3. I've create a helm chart to deploy Gitlab into my cluster using the arm docker image provided by ravermeister/gitlab.This images provides default settings for it to work properly on ARM devices, which do not have a lot of RAM.

I've mount the 3 directories needed, as known as :

• "/var/opt/gitlab"
• "/var/log/gitlab"
• "/etc/gitlab"

These directories are mounth throught a localPath, using nodeAffinity to target my rp4 with a SSD.

• What are you seeing, and how does it differ from what you expect to see? The installation start, but when reaching the step database_migrations, it fails. The container restarts indefinitely each time it reaches this stage

• Consider including screenshots, error messages, and/or other helpful visuals This is the log message :

  Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run
  * bash[migrate gitlab-rails database] action run
  ================================================================================
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
    ================================================================================
  
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Command execution failed. STDOUT/STDERR suppressed for sensitive resource
  
    Cookbook Trace: (most recent call first)
    ----------------------------------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in `block in class_from_file'
  
    Resource Declaration:
    ---------------------
    suppressed sensitive resource output
  
    Compiled Resource:
    ------------------
    suppressed sensitive resource output
  
    System Info:
    ------------
    chef_version=17.10.0
    platform=debian
    platform_version=11
    ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [aarch64-linux]

I don't know exactly what the problem is. I've already try to limit the memory usage in the container's configuration. I have already deleted the contents of the installation directories several times to ensure that the installation restarts cleanly but the same error occurs.

Once the CPU of the control plan (master) went up to 100% and it crashed. Which makes me think of a resource problem .

Thank you for your help !

[ravermeister]

hmm. The Image includes a PostgreSQL Database. I could it be that PostgreSQL is not able to run in a cluster scenario, I've never tested such a scenario by myself...

I'm running my instance on a single Raspberry Pi 4b successfully. are all Machines in this cluster ARM64 architectures? because the Database files for postgreSQL are incompatible between 32 and 64 bit.

[yanover]

No, the cluster is made of 5 raspberry pi including :

• 2 raspberry pi 4b
• 3 raspberry pi 3

This shouldn't be a problem though, the gitlab pod is targeted on the raspberry pi 4B, the one running on an SSD (arch64).

Here's my deployment.yaml file, nothing fancy as you might see :

# Source: mustafar/templates/gitlab/gitlab-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mustafar-gitlab
  namespace: mustafar
  labels:
    app: mustafar-gitlab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mustafar-gitlab
  template:
    metadata:
      labels:
        app: mustafar-gitlab
    spec:
      volumes:
      - name: gitlab-data-storage
        persistentVolumeClaim:
          claimName: pvc-mustafar-gitlab
      nodeName: rasticit
      containers:
      - name: gitlab
        image: "ulm0/gitlab"
        imagePullPolicy: IfNotPresent
        env:
          - name: GITLAB_OMNIBUS_CONFIG
            value: "external_url 'http://gitlab.nasticot.lan/';"
        ports:
          - containerPort: 80
          - containerPort: 22
        volumeMounts:
        - mountPath: /var/opt/gitlab
          name: gitlab-data-storage
        - mountPath: /var/log/gitlab
          name: gitlab-data-storage
        - mountPath: /etc/gitlab
          name: gitlab-data-storage
        resources:
            {}
        securityContext:
            privileged: true

But still, I suspect a resource problem.

If you have any advice to debug this.

Thank you !

[ravermeister]

image: "ulm0/gitlab" this is not my image, it is the old one from which I had forked mine. image: "ravermeister/gitlab" should be used...

you could try to watch docker logs --tail 50 -f mustafar-gitlab and check if there are any errors during startup

[ravermeister]

Have you tried switching to the image my ravermeister/gitlab and does it work? So I can close this issue?

[yanover]

Yes, I had already tested this image before. I clearly suspect a resource problem. I need to test a deployment on a node where all the resources are available.

Even with 80% of the resources available, the pod is unstable and returns 502 errors.

As this is not directly related to the image itself, I guess this issue can be closed

[ravermeister]

Thanks I'll close the issue for now. If I can help to investigate let me know :)