Newly created GitLab instance fails to mount data volume

[hannes-ucsc]

After a regularly scheduled GitLab version upgrade, the instance came back without mounting the data volume at /mnt/gitlab. The GitLab application the started with vanilla configuration and data, rendering it essentially dysfunctional.

It seems that when cloud-init attempted to create the /etc/fstab entry for the volume, the corresponding device node in /dev hadn't yet been created by the kernel. When I later logged in, the device node did exist. So this sounds like a race in cloud-init, the kernel or EC2. Rebooting the instance didn't help because I think that part of cloud-init is once-per-instance, not once-per-boot.

sudo grep -rF nvme1n1 /var/log/*
cloud-init.log:Jan 11 01:39:55 cloud-init[2588]: cc_mounts.py[DEBUG]: mounts configuration is [['/dev/nvme1n1', '/mnt/gitlab', 'ext4', '']]
cloud-init.log:Jan 11 01:39:55 cloud-init[2588]: cc_mounts.py[DEBUG]: Attempting to determine the real name of /dev/nvme1n1
cloud-init.log:Jan 11 01:39:55 cloud-init[2588]: cc_mounts.py[DEBUG]: changed /dev/nvme1n1 => None
cloud-init.log:Jan 11 01:39:55 cloud-init[2588]: cc_mounts.py[DEBUG]: Ignoring nonexistent named mount /dev/nvme1n1``` 

Compare that to a normal log

cloud-init.log:Jan 11 01:00:02 cloud-init[2575]: cc_mounts.py[DEBUG]: mounts configuration is [['/dev/nvme1n1', '/mnt/gitlab', 'ext4', '']]
cloud-init.log:Jan 11 01:00:02 cloud-init[2575]: cc_mounts.py[DEBUG]: Attempting to determine the real name of /dev/nvme1n1
cloud-init.log:Jan 11 01:00:02 cloud-init[2575]: cc_mounts.py[DEBUG]: Changes to fstab: ['+ /dev/nvme1n1 /mnt/gitlab ext4 ,comment=cloudconfig 0 2']
Binary file journal/ec29563d43d0eed9d02a4518576ec5d2/system@a55daef5424946e79396632b0c772bdc-0000000000000001-0005f1f283f5dba0.journal matches
messages:Jan 11 01:00:02 ip-172-71-0-215 kernel: EXT4-fs (nvme1n1): mounted filesystem with ordered data mode. Opts: (null)

---

• [ ] Security design review completed; the Resolution of this issue does not …
  • [ ] … affect authentication; for example:
  • OAuth 2.0 with the application (API or Swagger UI)
  • Authentication of developers with Google Cloud APIs
  • Authentication of developers with AWS APIs
  • Authentication with a GitLab instance in the system
  • Password and 2FA authentication with GitHub
  • API access token authentication with GitHub
  • Authentication with
  • [ ] … affect the permissions of internal users like access to
  • Cloud resources on AWS and GCP
  • GitLab repositories, projects and groups, administration
  • an EC2 instance via SSH
  • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • [ ] … affect the permissions of external users like access to
  • TDR snapshots
  • [ ] … affect permissions of service or bot accounts
  • Cloud resources on AWS and GCP
  • [ ] … affect audit logging in the system, like
  • adding, removing or changing a log message that represents an auditable event
  • changing the routing of log messages through the system
  • [ ] … affect monitoring of the system
  • [ ] … introduce a new software dependency like
  • Python packages on PYPI
  • Command-line utilities
  • Docker images
  • Terraform providers
  • [ ] … add an interface that exposes sensitive or confidential data at the security boundary
  • [ ] … affect the encryption of data at rest
  • [ ] … require persistence of sensitive or confidential data that might require encryption at rest
  • [ ] … require unencrypted transmission of data within the security boundary
  • [ ] … affect the network security layer; for example by
  • modifying, adding or removing firewall rules
  • modifying, adding or removing security groups
  • changing or adding a port a service, proxy or load balancer listens on
• [ ] Documentation on any unchecked boxes is provided in comments below

[achave11-ucsc]

The workaround is to recreate the instance.

[hannes-ucsc]

Nothing we can do here. There isn't even an upstream issue to block this on.

[hannes-ucsc]

https://github.com/canonical/cloud-init/issues/3386 is the upstream blocker

[hannes-ucsc]

Assignee to monitor upstream blocker.