search

sourcegraph / sourcegraph-public-snapshot

Code AI platform with Code Search & Cody
https://sourcegraph.com
Other
10.1k stars 1.28k forks source link

Sourcegraph Server image: oob migration startup error #62437

Open DaedalusG opened 5 months ago

DaedalusG commented 5 months ago

Latest build of the server image seems to be broken, possibly after changes to the oobmigration code in https://github.com/sourcegraph/sourcegraph/pull/61770

Steps to reproduce:

  1. sg images build server
  2. docker run server:candidate
➜  sourcegraph git:(wg/bug/oobmigration) ✗ docker run server:candidate
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
✱ Sourcegraph is initializing the internal database... (may take 15-20 seconds)
✱ Finished initializing the internal database.
Checking whether a Postgres reindex is required...
5.1 reindex marker file '/var/opt/sourcegraph/postgresql/5.1-reindex.completed' found
Starting postgres processes
Starting migrator
✱ Sourcegraph migrator 0.0.0+dev
Attempting connection to frontend...
✅ Connection to frontend succeeded
✅ Schema(s) are up-to-date!
✱ Sourcegraph migrator 0.0.0+dev
Attempting connection to frontend...
✅ Connection to frontend succeeded
Attempting connection to codeintel...
✅ Connection to codeintel succeeded
✅ Schema(s) are up-to-date!
Migrated postgres schemas.
Starting all processes
22:25:23         postgres_exporter | ts=2024-05-03T22:25:23.568Z caller=proc.go:271 msg="Excluded databases" databases=[]
22:25:23           zoekt-webserver | {"SeverityText":"WARN","Timestamp":1714775123983567962,"InstrumentationScope":"metricsRegistration.mountPointInfo.deviceNameDiscovery","Caller":"com_github_sourcegraph_mountinfo/info.go:48","Function":"github.com/sourcegraph/mountinfo.NewCollector","Body":"skipping metric registration","Resource":{"service.name":"zoekt-webserver","service.instance.id":"181c529d17ec"},"Attributes":{"mountName":"indexDir","mountFilePath":"/var/opt/sourcegraph/zoekt/index","reason":"failed to discover device name","error":"discovering device path: discoverSysfsDevicePath: failed to evaluate sysfs symlink \"/sys/dev/block/0:398\": lstat /sys/dev/block/0:398: no such file or directory"}}
22:25:24         zoekt-indexserver | {"SeverityText":"WARN","Timestamp":1714775124073481671,"InstrumentationScope":"metricsRegistration.mountPointInfo.deviceNameDiscovery","Caller":"com_github_sourcegraph_mountinfo/info.go:48","Function":"github.com/sourcegraph/mountinfo.NewCollector","Body":"skipping metric registration","Resource":{"service.name":"zoekt-indexserver","service.instance.id":"127.0.0.1:3070"},"Attributes":{"mountName":"indexDir","mountFilePath":"/var/opt/sourcegraph/zoekt/index","reason":"failed to discover device name","error":"discovering device path: discoverSysfsDevicePath: failed to evaluate sysfs symlink \"/sys/dev/block/0:398\": lstat /sys/dev/block/0:398: no such file or directory"}}
22:25:26                  frontend |                    ╓╦╬╬╬╦╖
22:25:26                  frontend |                   ╬╬╬╬╬╬╬╬╬
22:25:26                  frontend |                  ╞╬╬╬╬╬╬╬╬╬╬           ╓╦╦╦╦┐
22:25:26                  frontend |                   ╬╬╬╬╬╬╬╬╬╬╕        ╔╪╪╪╪╪╪╪╪╕
22:25:26                  frontend |                   ╘╬╬╬╬╬╬╬╬╬╬      ╔╪╪╪╪╪╪╪╪╪╪╪
22:25:26                  frontend |                    ╬╬╬╬╬╬╬╬╬╬╗   ╔╪╪╪╪╪╪╪╪╪╪╪╪┘
22:25:26                  frontend |       ╓╦╦╖┐         ╬╬╬╬╬╬╬╬╬╬ ╔╝╪╪╪╪╪╪╪╪╪╪╪╜
22:25:26                  frontend |     ╬╪╪╪╪╪╪╪╪╬╗╦╖   ╠╬╬╬╬╬╬╬╬╝╪╪╪╪╪╪╪╪╪╪╪╪╜
22:25:26                  frontend |    ╠╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╝╝╬╬╬╬╝╪╪╪╪╪╪╪╪╪╪╪╪╜
22:25:26                  frontend |    └╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╩
22:25:26                  frontend |      ╙╩╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╖
22:25:26                  frontend |           └╙╩╬╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╬╦╗┐
22:25:26                  frontend |                  ╙╜╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╗╦╖
22:25:26                  frontend |                  ┌╬╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╬
22:25:26                  frontend |                ┌╗╪╪╪╪╪╪╪╪╪╪╪╪╬╬╬╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪╪
22:25:26                  frontend |              ┌╗╪╪╪╪╪╪╪╪╪╪╪╪╬╬╬╬╬╬╬╬╬ ╙╜╩╪╪╪╪╪╪╪╪╪╪╪╪╜
22:25:26                  frontend |            ┌╦╪╪╪╪╪╪╪╪╪╪╪╝╙╬╬╬╬╬╬╬╬╬╬       └╙╩╬╪╪╝╜
22:25:26                  frontend |           ╦╪╪╪╪╪╪╪╪╪╪╪╪┘  ╠╬╬╬╬╬╬╬╬╬╬
22:25:26                  frontend |          ╬╪╪╪╪╪╪╪╪╪╪╪┘     ╬╬╬╬╬╬╬╬╬╬┐
22:25:26                  frontend |          ╙╪╪╪╪╪╪╪╪╪╜       ╘╬╬╬╬╬╬╬╬╬╬
22:25:26                  frontend |           └╩╪╪╪╪╝╙          ╬╬╬╬╬╬╬╬╬╬╕
22:25:26                  frontend |                             └╬╬╬╬╬╬╬╬╬╛
22:25:26                  frontend |                               ╩╬╬╬╬╬╬┘
22:25:29                 gitserver | {"SeverityText":"WARN","Timestamp":1714775129184667048,"InstrumentationScope":"gitserver.gitserverfs.mountPointInfo.deviceNameDiscovery","Caller":"com_github_sourcegraph_mountinfo/info.go:48","Function":"github.com/sourcegraph/mountinfo.NewCollector","Body":"skipping metric registration","Resource":{"service.name":"gitserver","service.version":"0.0.0+dev","service.instance.id":"127.0.0.1:3178"},"Attributes":{"mountName":"reposDir","mountFilePath":"/var/opt/sourcegraph/repos","reason":"failed to discover device name","error":"discovering device path: discoverSysfsDevicePath: failed to evaluate sysfs symlink \"/sys/dev/block/0:398\": lstat /sys/dev/block/0:398: no such file or directory"}}
22:25:29                    worker | {"SeverityText":"FATAL","Timestamp":1714775129605338548,"InstrumentationScope":"sourcegraph","Caller":"svcmain/svcmain.go:143","Function":"github.com/sourcegraph/sourcegraph/internal/service/svcmain.run","Body":"failed to start service","Resource":{"service.name":"worker","service.version":"0.0.0+dev","service.instance.id":"181c529d17ec"},"Attributes":{"service":"worker","error":"Failed to initialize worker:\n  - out-of-band-migrations: exit status 128"}}
22:25:30                    worker | Terminating worker
worker died. Shutting down...
22:25:30           zoekt-webserver | Terminating zoekt-webserver
22:25:30         postgres_exporter | Terminating postgres_exporter
22:25:30            syntect-server | Broken pipe (os error 32)
22:25:30         zoekt-indexserver | Terminating zoekt-indexserver
22:25:30                  searcher | Terminating searcher
22:25:30              repo-updater | Terminating repo-updater
22:25:30                prometheus | Terminating prometheus
22:25:30                     nginx | Terminating nginx
22:25:30 precise-code-intel-worker | Terminating precise-code-intel-worker
22:25:30                   symbols | Terminating symbols
22:25:30            syntect-server | Terminating syntect-server
22:25:30                   grafana | Terminating grafana
22:25:30               redis-cache | Terminating redis-cache
22:25:30                 gitserver | Terminating gitserver
22:25:30               redis-store | Terminating redis-store
22:25:30                 blobstore | Terminating blobstore
22:25:30                  postgres | Terminating postgres
22:25:30                  frontend | Terminating frontend

If you would like immediate help on this, please email support@sourcegraph.com (you can still create the issue, but there are no SLAs on issues like there are for support requests).

jhchabran commented 5 months ago

After investigating, the issue only manifests if you're trying to boot a development version of the server image. If it's properly stamped like it is in CI, we don't have the issue.

Still, should be fixed, but nothing to worry about.