search

go-gitea / gitea

Git with a cup of tea! Painless self-hosted all-in-one software development service, including Git hosting, code review, team collaboration, package registry and CI/CD
https://gitea.com
MIT License
45.21k stars 5.5k forks source link

Timetraveling Gitea #12249

Closed insanemal closed 3 years ago

insanemal commented 4 years ago

When using gitea via web or via git, everything (repos/heatmaps) travel backwards and forwards in time. So for example if you are on the dashboard. It will show you the up to date status. Hit refresh and you get months ago. Hit refresh again you might get a 500 error. Hit refresh again and you will get one of the 3 above states.

The same thing happens with repos via both SSH or HTTP/s git. You go to push/pull you'll either not be able to because the repo doesn't exist or it works or something very strange happens.

The logs (which I currently have set to trace, show almost nothing unusual.

When there is a 500 error this can be seen in the logs 

[Macaron] 2020-07-15 05:39:14: Started GET / for 192.168.222.15
2020/07/15 05:39:14 ...les/context/panic.go:34:1() [E] PANIC:: runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:82 (0x44a9a0)
/usr/local/go/src/runtime/signal_unix.go:390 (0x44a7cf)
/go/src/code.gitea.io/gitea/routers/user/home.go:78 (0x13d3c13)
/go/src/code.gitea.io/gitea/routers/user/home.go:139 (0x13d463d)
/go/src/code.gitea.io/gitea/routers/home.go:51 (0x14308bc)
/usr/local/go/src/reflect/value.go:447 (0x48f6f0)
/usr/local/go/src/reflect/value.go:308 (0x48f173)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:177 (0x98a306)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:137 (0x989bfb)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:121 (0x9b9d79)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:112 (0x11d6d68)
/go/src/code.gitea.io/gitea/modules/context/panic.go:38 (0x11d6d5b)
/usr/local/go/src/reflect/value.go:447 (0x48f6f0)
/usr/local/go/src/reflect/value.go:308 (0x48f173)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:177 (0x98a306)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:137 (0x989bfb)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:121 (0x9b9d79)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:112 (0x9e9556)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/session/session.go:192 (0x9e9541)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:79 (0x9b9c20)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:157 (0x989fc0)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:135 (0x989cea)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:121 (0x9b9d79)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:112 (0x9cb80f)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/recovery.go:161 (0x9cb7fd)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/logger.go:40 (0x9bda13)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:157 (0x989fc0)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:135 (0x989cea)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:121 (0x9b9d79)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:112 (0x9cab30)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/logger.go:52 (0x9cab1b)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/logger.go:40 (0x9bda13)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:157 (0x989fc0)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:135 (0x989cea)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:121 (0x9b9d79)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/router.go:187 (0x9cca56)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/router.go:294 (0x9c656d)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/macaron.go:220 (0x9bedec)
/go/src/code.gitea.io/gitea/vendor/github.com/gorilla/context/context.go:141 (0xd1d8aa)
/usr/local/go/src/net/http/server.go:1995 (0x77e053)
/usr/local/go/src/net/http/server.go:2774 (0x781327)
/usr/local/go/src/net/http/server.go:1878 (0x77d010)
/usr/local/go/src/runtime/asm_amd64.s:1337 (0x464a00)
/go/src/code.gitea.io/gitea/modules/context/panic.go:34 (0x11d6cb1)
/usr/local/go/src/runtime/panic.go:522 (0x434de4)
/usr/local/go/src/runtime/panic.go:82 (0x44a9a0)
/usr/local/go/src/runtime/signal_unix.go:390 (0x44a7cf)
/go/src/code.gitea.io/gitea/routers/user/home.go:78 (0x13d3c13)
/go/src/code.gitea.io/gitea/routers/user/home.go:139 (0x13d463d)
/go/src/code.gitea.io/gitea/routers/home.go:51 (0x14308bc)
/usr/local/go/src/reflect/value.go:447 (0x48f6f0)
/usr/local/go/src/reflect/value.go:308 (0x48f173)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:177 (0x98a306)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:137 (0x989bfb)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:121 (0x9b9d79)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:112 (0x11d6d68)
/go/src/code.gitea.io/gitea/modules/context/panic.go:38 (0x11d6d5b)
/usr/local/go/src/reflect/value.go:447 (0x48f6f0)
/usr/local/go/src/reflect/value.go:308 (0x48f173)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:177 (0x98a306)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:137 (0x989bfb)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:121 (0x9b9d79)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:112 (0x9e9556)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/session/session.go:192 (0x9e9541)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:79 (0x9b9c20)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:157 (0x989fc0)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:135 (0x989cea)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:121 (0x9b9d79)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:112 (0x9cb80f)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/recovery.go:161 (0x9cb7fd)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/logger.go:40 (0x9bda13)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:157 (0x989fc0)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:135 (0x989cea)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:121 (0x9b9d79)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:112 (0x9cab30)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/logger.go:52 (0x9cab1b)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/logger.go:40 (0x9bda13)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:157 (0x989fc0)
/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:135 (0x989cea)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:121 (0x9b9d79)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/router.go:187 (0x9cca56)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/router.go:294 (0x9c656d)
/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/macaron.go:220 (0x9bedec)
/go/src/code.gitea.io/gitea/vendor/github.com/gorilla/context/context.go:141 (0xd1d8aa)
/usr/local/go/src/net/http/server.go:1995 (0x77e053)
/usr/local/go/src/net/http/server.go:2774 (0x781327)
/usr/local/go/src/net/http/server.go:1878 (0x77d010)
/usr/local/go/src/runtime/asm_amd64.s:1337 (0x464a00)

This is what I see when a repo works and then doesnt work, 

[Macaron] 2020-07-15 05:51:27: Completed GET /img/favicon.png 200 OK in 201.11µs
[Macaron] 2020-07-15 05:51:28: Started GET /path/project for 192.168.xxx.yyy
[Macaron] 2020-07-15 05:51:28: Completed GET /path/project 200 OK in 62.941666ms
[Macaron] 2020-07-15 05:51:28: Started GET /serviceworker.js for 192.168.xxx.yyy
[Macaron] 2020-07-15 05:51:28: Completed GET /serviceworker.js 200 OK in 9.496809ms
[Macaron] 2020-07-15 05:51:28: Started GET /img/favicon.png for 192.168.xxx.yyy
[Macaron] [Static] Serving /img/favicon.png
[Macaron] 2020-07-15 05:51:28: Started GET /manifest.json for 192.168.xxx.yyy
[Macaron] 2020-07-15 05:51:28: Completed GET /img/favicon.png 200 OK in 1.085056ms
[Macaron] 2020-07-15 05:51:28: Completed GET /manifest.json 200 OK in 9.541404ms
[Macaron] 2020-07-15 05:51:30: Started GET /serviceworker.js for 192.168.xxx.yyy
[Macaron] 2020-07-15 05:51:30: Completed GET /serviceworker.js 200 OK in 8.784345ms
[Macaron] 2020-07-15 05:51:32: Started GET /path/project for 192.168.xxx.yyy
[Macaron] 2020-07-15 05:51:32: Completed GET /path/project 404 Not Found in 38.241072ms
[Macaron] 2020-07-15 05:51:32: Started GET /img/favicon.png for 192.168.xxx.yyy

Randomly I saw "Unexpected EOF" errors returned from the "[MYSQL]" logging entry. However they did not appear to be clearly linked to any events

This was running 1.7 and I upgraded to 1.8 and then 1.9 in the hopes of fixing it.

I have stopped upgrading it as things are clearly not in a good place ...

Screenshots

lunny commented 4 years ago

Please upgrade to a recent version so that we could know if it has been resolved.

insanemal commented 4 years ago

I can't seem to get it to upgrade past 1.9

When I roll to 1.10 it takes too long doing the migration so k8's kills it.

It seems to resume when the pod gets started up again, however it went on like that for at least an hour without actually bringing gitea online.

Is that to be expected? If so I guess I can just leave it crank away and see how long it takes

insanemal commented 4 years ago

Oh and to be clear when I attempted to go to 1.10, I did a database backup before

And when I aborted it, I restored the 1.9 database.

lunny commented 4 years ago

You can use the new gitea program and then migrate database via gitea migrate manually.

insanemal commented 3 years ago

This looks to be solved thanks