Beta v2.9.46-beta - Error while running tasks

[Menschomat]

Since i updated to the latest beta i can't run any tasks in my system.

semaphore-1  | time="2024-02-04T20:14:49Z" level=error msg="no rows in result set" error="Bad request. Cannot get task from database"
semaphore-1  | 2024/02/04 20:15:03 http: superfluous response.WriteHeader call from github.com/ansible-semaphore/semaphore/api/helpers.WriteJSON (helpers.go:67)
semaphore-1  | time="2024-02-04T20:15:10Z" level=error msg="no rows in result set" error="Bad request. Cannot get task from database"
semaphore-1  | time="2024-02-04T20:15:53Z" level=info msg="Task 396 added to queue"
semaphore-1  | time="2024-02-04T20:15:58Z" level=info msg="Set resource locker with TaskRunner 396"
semaphore-1  | time="2024-02-04T20:15:58Z" level=info msg="Task 396 removed from queue"
semaphore-1  | time="2024-02-04T20:16:00Z" level=info msg="Stopped running TaskRunner 396"
semaphore-1  | time="2024-02-04T20:16:00Z" level=info msg="Release resource locker with TaskRunner 396"
semaphore-1  | panic: runtime error: invalid memory address or nil pointer dereference
semaphore-1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x959e8b]
semaphore-1  | 
semaphore-1  | goroutine 81 [running]:
semaphore-1  | github.com/ansible-semaphore/semaphore/db_lib.(*AnsibleApp).Log(...)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/db_lib/AnsibleApp.go:67
semaphore-1  | github.com/ansible-semaphore/semaphore/db_lib.(*AnsibleApp).installCollectionsRequirements(0xc00014ec80)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/db_lib/AnsibleApp.go:131 +0x32b
semaphore-1  | github.com/ansible-semaphore/semaphore/db_lib.(*AnsibleApp).InstallRequirements(0xc000ce8580?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/db_lib/AnsibleApp.go:71 +0x1e
semaphore-1  | github.com/ansible-semaphore/semaphore/services/tasks.(*LocalJob).prepareRun(0xc000ce8580)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/services/tasks/LocalJob.go:363 +0x271
semaphore-1  | github.com/ansible-semaphore/semaphore/services/tasks.(*LocalJob).Run(0xc000ce8580, {0xc000ecedb0, 0x5}, 0xc000914e70?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/services/tasks/LocalJob.go:287 +0x91
semaphore-1  | github.com/ansible-semaphore/semaphore/services/tasks.(*TaskRunner).run(0xc000ce8000)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/services/tasks/TaskRunner.go:188 +0x6a6
semaphore-1  | created by github.com/ansible-semaphore/semaphore/services/tasks.(*TaskPool).Run
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/services/tasks/TaskPool.go:172 +0x4ca

System is running with docker-compose with postgres-db.

Downgrading to latest stable did not work.

[fiftin]

Hi @Menschomat after downgrade you have the save error?

[fiftin]

@Menschomat thank you for bug report, I already fixing the issue.

[Menschomat]

After downgrade to latest stable I was not able to open the template-view any more with database-errors in the logs:

semaphore-1  | Semaphore v2.9.45
semaphore-1  | Interface 
semaphore-1  | Port :3000
semaphore-1  | Server is running
semaphore-1  | time="2024-02-04T20:00:43Z" level=error msg="gorp: no fields [app] in type Template"
semaphore-1  | goroutine 24 [running]:
semaphore-1  | runtime/debug.Stack()
semaphore-1  |  /usr/local/go/src/runtime/debug/stack.go:24 +0x65
semaphore-1  | runtime/debug.PrintStack()
semaphore-1  |  /usr/local/go/src/runtime/debug/stack.go:16 +0x19
semaphore-1  | github.com/ansible-semaphore/semaphore/api/helpers.WriteError({0x2345ac0, 0xc0002d6540}, {0x23402a0?, 0xc00115c2d0?})
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/helpers/helpers.go:92 +0x199
semaphore-1  | github.com/ansible-semaphore/semaphore/api/projects.TemplatesMiddleware.func1({0x2345ac0, 0xc0002d6540}, 0xc000c6d1b8?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/projects/templates.go:24 +0x293
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0x414205?, {0x2345ac0?, 0xc0002d6540?}, 0x95ee5c?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2122 +0x2f
semaphore-1  | github.com/ansible-semaphore/semaphore/api/projects.GetMustCanMiddleware.func1.1({0x2345ac0, 0xc0002d6540}, 0xc000435200)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/projects/project.go:59 +0x149
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xc000435200?, {0x2345ac0?, 0xc0002d6540?}, 0xbd2660?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2122 +0x2f
semaphore-1  | github.com/ansible-semaphore/semaphore/api/projects.ProjectMiddleware.func1({0x2345ac0, 0xc0002d6540}, 0x10001?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/projects/project.go:43 +0x394
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0x2345ac0?, {0x2345ac0?, 0xc0002d6540?}, 0xd700000000000010?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2122 +0x2f
semaphore-1  | github.com/ansible-semaphore/semaphore/api.authentication.func1({0x2345ac0, 0xc0002d6540}, 0xc0000f1080?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/auth.go:102 +0x5d
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xbba0c0?, {0x2345ac0?, 0xc0002d6540?}, 0xc?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2122 +0x2f
semaphore-1  | github.com/ansible-semaphore/semaphore/api.JSONMiddleware.func1({0x2345ac0, 0xc0002d6540}, 0x450819?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/router.go:37 +0xf4
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xb17300?, {0x2345ac0?, 0xc0002d6540?}, 0x77ffd8?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2122 +0x2f
semaphore-1  | github.com/ansible-semaphore/semaphore/api.StoreMiddleware.func1.1()
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/router.go:28 +0x2e
semaphore-1  | github.com/ansible-semaphore/semaphore/db.StoreSession({0x23507c8, 0xc000012240}, {0xc0003e0620, 0xc}, 0xc0000b5700)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/db/Store.go:354 +0x68
semaphore-1  | github.com/ansible-semaphore/semaphore/api.StoreMiddleware.func1({0x2345ac0, 0xc0002d6540}, 0xc000435200)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/api/router.go:27 +0xf2
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xc000435200?, {0x2345ac0?, 0xc0002d6540?}, 0xbb6a00?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2122 +0x2f
semaphore-1  | github.com/ansible-semaphore/semaphore/cli/cmd.runService.func1.1({0x2345ac0, 0xc0002d6540}, 0xc0000b57b0?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/cli/cmd/root.go:74 +0xeb
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xc016c0?, {0x2345ac0?, 0xc0002d6540?}, 0x7f1e1befd5b8?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2122 +0x2f
semaphore-1  | github.com/gorilla/mux.CORSMethodMiddleware.func1.1({0x2345ac0, 0xc0002d6540}, 0xc000bbdf80?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/mux/middleware.go:51 +0xa3
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0xc000435100?, {0x2345ac0?, 0xc0002d6540?}, 0xc00360?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2122 +0x2f
semaphore-1  | github.com/gorilla/mux.(*Router).ServeHTTP(0xc0003d6000, {0x2345ac0, 0xc0002d6540}, 0xc000435000)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/mux/mux.go:212 +0x202
semaphore-1  | github.com/gorilla/handlers.ProxyHeaders.func1({0x2345ac0, 0xc0002d6540}, 0xc000435000)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/handlers/proxy_headers.go:59 +0x144
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0x3b?, {0x2345ac0?, 0xc0002d6540?}, 0x0?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2122 +0x2f
semaphore-1  | github.com/ansible-semaphore/semaphore/cli/cmd.cropTrailingSlashMiddleware.func1({0x2345ac0?, 0xc0002d6540?}, 0xc000435000?)
semaphore-1  |  /go/src/github.com/ansible-semaphore/semaphore/cli/cmd/server.go:27 +0xf7
semaphore-1  | net/http.HandlerFunc.ServeHTTP(0x0?, {0x2345ac0?, 0xc0002d6540?}, 0x464f6e?)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2122 +0x2f
semaphore-1  | net/http.serverHandler.ServeHTTP({0xc00022e1e0?}, {0x2345ac0, 0xc0002d6540}, 0xc000435000)
semaphore-1  |  /usr/local/go/src/net/http/server.go:2936 +0x316
semaphore-1  | net/http.(*conn).serve(0xc0001ca3f0, {0x2346198, 0xc000bbc0c0})
semaphore-1  |  /usr/local/go/src/net/http/server.go:1995 +0x612
semaphore-1  | created by net/http.(*Server).Serve
semaphore-1  |  /usr/local/go/src/net/http/server.go:3089 +0x5ed

I now restored latest backups via my disaster-recovery and upgraded successfully to latest stable from previous beta. Unfortunately, I had my docker stack on the beta tag and automatic updates via watchtower enabled. Always good to have a backup.

[fiftin]

Got it. Thank you.

Please upgrade to v2.9.47-beta

[Menschomat]

Thank you very much for the quick fix, but I think for now i will stay whit the stable version. If i have some time in the evening this week, I'll check if the fix recovers the broken-state (i did a backup of the broken-state, aswell).

[hmoffatt]

I tried v2.9.47-beta and I still got the nil pointer error

semaphore_1  | time="2024-02-05T08:08:06Z" level=error msg="websocket: close 1005 (no status)" level=Error
semaphore_1  | time="2024-02-05T08:08:06Z" level=error msg="websocket: close sent" level=Error
semaphore_1  | time="2024-02-05T08:08:06Z" level=error msg="close tcp 172.17.11.2:3000->172.17.11.1:43882: use of closed network connection" level=Error
semaphore_1  | time="2024-02-05T08:08:12Z" level=info msg="Task 139 added to queue"
semaphore_1  | time="2024-02-05T08:08:14Z" level=info msg="Set resource locker with TaskRunner 139"
semaphore_1  | time="2024-02-05T08:08:14Z" level=info msg="Task 139 removed from queue"
semaphore_1  | time="2024-02-05T08:08:16Z" level=info msg="Stopped running TaskRunner 139"
semaphore_1  | time="2024-02-05T08:08:16Z" level=info msg="Release resource locker with TaskRunner 139"
semaphore_1  | panic: runtime error: invalid memory address or nil pointer dereference
semaphore_1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x95a8af]

I had the same issue as @Menschomat when I downgraded and also had to restore my database from backup.

[tboerger]

I have pushed https://github.com/ansible-semaphore/semaphore/pull/1740, the AnsiblePlaybook struct was missing the logger dependency as well, https://github.com/ansible-semaphore/semaphore/commit/78c93b1c84a64b7b12d60c228b5b3fa1cb270c93 has added it to AnsibleApp only.

[hmoffatt]

Thanks v2.9.48-beta is working for me.