Latest Docker Image does not work

samuel-p commented 2 years ago

Component

server

Describe the bug

I tried updating my docker-compose setup to the latest version, but the server didn't came up. The only logs I got (using --log-level=trace) is this:

woodpecker_1        | {"time":"2021-11-21T16:24:16Z","message":"LogLevel = trace"}
woodpecker_1        | {"level":"trace","time":"2021-11-21T16:24:16Z","message":"Remote (gitea) opts: gitea.Opts{URL:\"https://git.sp-codes.de\", Context:\"continuous-integration/woodpecker\", Client:\"...\", Secret:\"...\", Username:\"\", Password:\"\", PrivateMode:false, SkipVerify:false}"}
woodpecker_1        | {"level":"trace","time":"2021-11-21T16:24:16Z","message":"setup datastore: store.Opts{Driver:\"postgres\", Config:\"postgres://...:...@database/woodpecker?sslmode=disable\"}"}
woodpecker_1        | {"level":"trace","time":"2021-11-21T16:24:16Z","message":"start migration task 'xorm'"}
woodpecker_1        | {"level":"info","time":"2021-11-21T16:24:17Z","message":"migration task 'xorm' done"}
woodpecker_1        | {"level":"trace","time":"2021-11-21T16:24:17Z","message":"start migration task 'alter-table-drop-repo-fallback'"}
woodpecker_1        | {"level":"trace","time":"2021-11-21T16:24:17Z","message":"migration task 'alter-table-drop-repo-fallback' exist"}
woodpecker_1        | {"level":"trace","time":"2021-11-21T16:24:17Z","message":"start migration task 'drop-allow-push-tags-deploys-columns'"}
woodpecker_1        | {"level":"trace","time":"2021-11-21T16:24:17Z","message":"migration task 'drop-allow-push-tags-deploys-columns' exist"}
woodpecker_1        | [xorm] [warn]  2021/11/21 16:24:17.102343 Table agents column agent_id db type is SERIAL, struct type is BIGSERIAL
woodpecker_1        | [xorm] [warn]  2021/11/21 16:24:17.102375 Table agents column agent_capacity db type is INTEGER, struct type is BIGINT
woodpecker_1        | [xorm] [warn]  2021/11/21 16:24:17.102379 Table agents column agent_created db type is INTEGER, struct type is BIGINT
woodpecker_1        | [xorm] [warn]  2021/11/21 16:24:17.102383 Table agents column agent_updated db type is INTEGER, struct type is BIGINT

System Info

Not available, because the server didn't start.

Additional context

No response

Validations

[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A join our Discord Chat Server.

6543 commented 2 years ago

@samuel-p just a workaround: could you kill the container and start with same version after xorm migration was done

does it work after the second/third restart?

samuel-p commented 2 years ago

I restarted the container several times without success and got the same logs every time. I now switched to the v0.14.3 tag and it seems to work again so far.

6543 commented 2 years ago

@samuel-p can you start with LogLevel = trace and send me the logs (from server & agent)

6543 commented 2 years ago

& what type of database do you use?

samuel-p commented 2 years ago

I updated the debug-Logs in my first post with the trace-Logs. I use Postgres as database. The agent seems to work so far with the latest version.

6543 commented 2 years ago

ok the issue is due the #474 rework ...

Stack Trace

``` {"level":"trace","time":"2021-11-26T21:21:43+01:00","message":"migration task 'alter-table-drop-repo-fallback' already applied"} {"level":"trace","time":"2021-11-26T21:21:43+01:00","message":"migration task 'drop-allow-push-tags-deploys-columns' already applied"} fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x47 pc=0x7f568c616414] runtime stack: runtime.throw({0x109b324, 0x0}) /usr/lib/go/src/runtime/panic.go:1198 +0x71 runtime.sigpanic() /usr/lib/go/src/runtime/signal_unix.go:719 +0x396 goroutine 48 [syscall]: runtime.cgocall(0xcc7650, 0xc0002aed90) /usr/lib/go/src/runtime/cgocall.go:156 +0x5c fp=0xc0002aed68 sp=0xc0002aed30 pc=0x404f9c net._C2func_getaddrinfo(0xc0006904a0, 0x0, 0xc000382c30, 0xc0005a6080) _cgo_gotypes.go:91 +0x56 fp=0xc0002aed90 sp=0xc0002aed68 pc=0x5d79d6 net.cgoLookupIPCNAME.func1({0xc0006904a0, 0x0, 0x0}, 0xc000690480, 0xc0002aee50) /usr/lib/go/src/net/cgo_unix.go:163 +0x9f fp=0xc0002aede8 sp=0xc0002aed90 pc=0x5d971f net.cgoLookupIPCNAME({0x106140d, 0x3}, {0xc000690480, 0x0}) /usr/lib/go/src/net/cgo_unix.go:163 +0x16d fp=0xc0002aef38 sp=0xc0002aede8 pc=0x5d8f6d net.cgoIPLookup(0x1952210, {0x106140d, 0xc000690490}, {0xc000690480, 0x0}) /usr/lib/go/src/net/cgo_unix.go:220 +0x3b fp=0xc0002aefa8 sp=0xc0002aef38 pc=0x5d97db net.cgoLookupIP·dwrap·25() /usr/lib/go/src/net/cgo_unix.go:230 +0x36 fp=0xc0002aefe0 sp=0xc0002aefa8 pc=0x5d9c56 runtime.goexit() /usr/lib/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0002aefe8 sp=0xc0002aefe0 pc=0x46a5a1 created by net.cgoLookupIP /usr/lib/go/src/net/cgo_unix.go:230 +0x125 goroutine 1 [select]: net.(*Resolver).lookupIPAddr(0x1952200, {0x1242a68, 0xc000038158}, {0x106140d, 0x0}, {0xc000690480, 0x9}) /usr/lib/go/src/net/lookup.go:302 +0x5c7 net.(*Resolver).internetAddrList(0x1242a68, {0x1242a68, 0xc000038158}, {0x106140d, 0x3}, {0xc000690480, 0xe}) /usr/lib/go/src/net/ipsock.go:288 +0x67a net.(*Resolver).resolveAddrList(0xc000038158, {0x1242a68, 0xc000038158}, {0x1062300, 0x4}, {0x106140d, 0xc00063c5a0}, {0xc000690480, 0xe}, {0x0, ...}) /usr/lib/go/src/net/dial.go:221 +0x41b net.(*Dialer).DialContext(0xc00063c6d8, {0x1242a68, 0xc000038158}, {0x106140d, 0x1070936}, {0xc000690480, 0x4}) /usr/lib/go/src/net/dial.go:406 +0x448 github.com/lib/pq.defaultDialer.DialContext(...) /home/git/woodpecker/vendor/github.com/lib/pq/conn.go:119 github.com/lib/pq.dial({0x1242a68, 0xc000038158}, {0x123a390, 0xc00010e180}, 0xc94745) /home/git/woodpecker/vendor/github.com/lib/pq/conn.go:385 +0x3e9 github.com/lib/pq.(*Connector).open(0xc00000e690, {0x1242a68, 0xc000038158}) /home/git/woodpecker/vendor/github.com/lib/pq/conn.go:322 +0x269 github.com/lib/pq.DialOpen({0x123a390, 0xc00010e180}, {0xc00003a13f, 0x400}) /home/git/woodpecker/vendor/github.com/lib/pq/conn.go:291 +0x7a github.com/lib/pq.Open(...) /home/git/woodpecker/vendor/github.com/lib/pq/conn.go:281 github.com/lib/pq.Driver.Open({}, {0xc00003a13f, 0x35}) /home/git/woodpecker/vendor/github.com/lib/pq/conn.go:54 +0x7a database/sql.dsnConnector.Connect(...) /usr/lib/go/src/database/sql/sql.go:761 database/sql.(*DB).conn(0xc00011c9c0, {0x1242a68, 0xc000038158}, 0x1) /usr/lib/go/src/database/sql/sql.go:1364 +0x7ac database/sql.(*DB).query(0x41, {0x1242a68, 0xc000038158}, {0xc0001f4400, 0x38d}, {0xc0004ce1e0, 0x2, 0x2}, 0x0) /usr/lib/go/src/database/sql/sql.go:1695 +0x5d database/sql.(*DB).QueryContext(0xc0005a2240, {0x1242a68, 0xc000038158}, {0xc0001f4400, 0x38d}, {0xc0004ce1e0, 0x2, 0x2}) /usr/lib/go/src/database/sql/sql.go:1674 +0xdf xorm.io/xorm/core.(*DB).QueryContext(0xc0005a2240, {0x1242a68, 0xc000038158}, {0xc0001f4400, 0x38d}, {0xc0004ce1e0, 0x2, 0x2}) /home/git/woodpecker/vendor/xorm.io/xorm/core/db.go:151 +0x1dc xorm.io/xorm/dialects.(*postgres).GetColumns(0xc0004e58f0, {0x122aba0, 0xc0005a2240}, {0x1242a68, 0xc000038158}, {0xc00069012a, 0x6}) /home/git/woodpecker/vendor/xorm.io/xorm/dialects/postgres.go:1062 +0x29b xorm.io/xorm.(*Engine).loadTableInfo(0xc000326500, 0xc00055e1e0) /home/git/woodpecker/vendor/xorm.io/xorm/engine.go:363 +0x64 xorm.io/xorm.(*Session).Sync2(0xc00022b520, {0xc00063d7b0, 0x1, 0xc000116050}) /home/git/woodpecker/vendor/xorm.io/xorm/session_schema.go:289 +0x3b7 github.com/woodpecker-ci/woodpecker/server/store/datastore/migration.syncAll(0xc00022b520) /home/git/woodpecker/server/store/datastore/migration/migration.go:148 +0x351 github.com/woodpecker-ci/woodpecker/server/store/datastore/migration.Migrate(0xc000326500) /home/git/woodpecker/server/store/datastore/migration/migration.go:88 +0x1b7 github.com/woodpecker-ci/woodpecker/server/store/datastore.storage.Migrate(...) /home/git/woodpecker/server/store/datastore/engine.go:51 main.setupStore(0xc0000421c0) /home/git/woodpecker/cmd/server/setup.go:81 +0x396 main.run(0xc0000bb040) /home/git/woodpecker/cmd/server/server.go:106 +0x545 github.com/urfave/cli/v2.(*App).RunContext(0xc0002a3860, {0x1242a68, 0xc000038158}, {0xc000032210, 0x1, 0x1}) /home/git/woodpecker/vendor/github.com/urfave/cli/v2/app.go:322 +0x7a8 github.com/urfave/cli/v2.(*App).Run(...) /home/git/woodpecker/vendor/github.com/urfave/cli/v2/app.go:224 main.main() /home/git/woodpecker/cmd/server/main.go:39 +0x1b1 goroutine 54 [select]: database/sql.(*DB).connectionOpener(0xc00011c9c0, {0x1242a30, 0xc0000bb0c0}) /usr/lib/go/src/database/sql/sql.go:1196 +0x93 created by database/sql.OpenDB /usr/lib/go/src/database/sql/sql.go:794 +0x188 goroutine 45 [chan receive]: database/sql.(*Tx).awaitDone(0xc0000f0280) /usr/lib/go/src/database/sql/sql.go:2148 +0x31 created by database/sql.(*DB).beginDC /usr/lib/go/src/database/sql/sql.go:1864 +0x247 goroutine 47 [select]: net.cgoLookupIP({0x1242a30, 0xc000132340}, {0x106140d, 0x9}, {0xc000690480, 0x0}) /usr/lib/go/src/net/cgo_unix.go:231 +0x1b7 net.(*Resolver).lookupIP(0x1952200, {0x1242a30, 0xc000132340}, {0x106140d, 0x3}, {0xc000690480, 0x9}) /usr/lib/go/src/net/lookup_unix.go:97 +0x128 net.glob..func1({0x1242a30, 0xc000132340}, 0x0, {0x106140d, 0x0}, {0xc000690480, 0x0}) /usr/lib/go/src/net/hook.go:23 +0x3d net.(*Resolver).lookupIPAddr.func1() /usr/lib/go/src/net/lookup.go:296 +0x9f internal/singleflight.(*Group).doCall(0x1952210, 0xc0004d4230, {0xc000690490, 0xd}, 0x0) /usr/lib/go/src/internal/singleflight/singleflight.go:95 +0x3b created by internal/singleflight.(*Group).DoChan /usr/lib/go/src/internal/singleflight/singleflight.go:88 +0x2f1 ```

6543 commented 2 years ago

@samuel-p can you test the docker image woodpeckerci/woodpecker-server:codeberg-test PS: please do not upgrade the agent as there is an issue with the backend-engine-detection (#542)

if this fix it #544 can go in :)

samuel-p commented 2 years ago

Yes it works! I got a lot of warnings from xorm at the beginning, but after that it works (at least for me). Thank you.

Logs

``` woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.799906 Table agents column agent_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.799951 Table agents column agent_capacity db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.799955 Table agents column agent_created db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.799958 Table agents column agent_updated db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829427 Table builds column build_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829452 Table builds column build_repo_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829457 Table builds column build_number db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829466 Table builds column build_config_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829471 Table builds column build_parent db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829478 Table builds column build_enqueued db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829482 Table builds column build_created db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829750 Table builds column build_started db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829770 Table builds column build_finished db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829786 Table builds column build_timestamp db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829803 Table builds column build_reviewed db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.829814 Table builds column changed_files db type is TEXT, struct type is JSON woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.938605 Table config column config_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.938634 Table config column config_repo_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.938643 Table config column config_name db type is TEXT, struct type is VARCHAR(255) woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.955772 Table files column file_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.955798 Table files column file_build_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.955817 Table files column file_proc_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.955826 Table files column file_time db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.977231 Table logs column log_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.977273 Table logs column log_job_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.994626 Table perms column perm_user_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.994663 Table perms column perm_repo_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:14.994673 Table perms column perm_synced db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.035512 Table procs column proc_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.035540 Table procs column proc_build_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.035556 Table procs column proc_started db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.035561 Table procs column proc_stopped db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.035570 Table procs column proc_environ db type is VARCHAR(2000), struct type is JSON woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.113526 Table registry column registry_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.113553 Table registry column registry_repo_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [info] 2021/11/26 20:57:15.113564 Table registry column registry_password change type from VARCHAR(8000) to TEXT woodpecker_1 | [xorm] [info] 2021/11/26 20:57:15.115339 Table registry column registry_token change type from VARCHAR(2000) to TEXT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.145009 Table repos column repo_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.145034 Table repos column repo_user_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.145049 Table repos column repo_timeout db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.145059 Table repos column repo_counter db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.145062 Table repos Column repo_counter db default is , struct default is 0 woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.145064 Table repos Column repo_counter db nullable is true, struct nullable is false woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.163878 Table secrets column secret_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.163904 Table secrets column secret_repo_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.163914 Table secrets column secret_value db type is BYTEA, struct type is TEXT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.163918 Table secrets column secret_images db type is VARCHAR(2000), struct type is JSON woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.163922 Table secrets column secret_events db type is VARCHAR(2000), struct type is JSON woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.185357 Table senders column sender_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.185379 Table senders column sender_repo_id db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.203025 Table tasks column task_labels db type is BYTEA, struct type is JSON woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.203053 Table tasks column task_dependencies db type is BYTEA, struct type is JSON woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.203058 Table tasks column task_run_on db type is BYTEA, struct type is JSON woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.239148 Table users column user_id db type is SERIAL, struct type is BIGSERIAL woodpecker_1 | [xorm] [info] 2021/11/26 20:57:15.239179 Table users column user_token change type from VARCHAR(1000) to TEXT woodpecker_1 | [xorm] [info] 2021/11/26 20:57:15.246568 Table users column user_secret change type from VARCHAR(1000) to TEXT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.248855 Table users column user_expiry db type is INTEGER, struct type is BIGINT woodpecker_1 | [xorm] [warn] 2021/11/26 20:57:15.248908 Table users column user_synced db type is INTEGER, struct type is BIGINT ```

6543 commented 2 years ago

xorm warnings are ok - they just point out that the colum types do not match exact to what xorm would propose if table would be recreated - there is no downside in functionality ...

and good to hear :) !

woodpecker-ci / woodpecker