Segmentation fault with Gitea on a Raspi 4 #3271

Closed LuciferSam86 closed 5 years ago

Describe the bug When I launch gitea I got "segmentation fault"

To reproduce

Download gitea binary 1.9.3 from here : https://dl.gitea.io/gitea/1.9.3/gitea-1.9.3-linux-arm-6
execute gitea web
Get "segmentation fault"

Expected behaviour Gitea should executed and deploy his webserver.

Actual behaviour I get segmentation fault

System Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

Which model of Raspberry Pi? 4B with 4 GB of RAM

Which OS and version (cat /etc/rpi-issue)?

Raspberry Pi reference 2019-09-26
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage2

Which firmware version (vcgencmd version)?

Sep 24 2019 17:34:30
Copyright (c) 2012 Broadcom
version cd3add54955f8fa065b414d8fc07c525e7ddffc8 (clean) (release) (start)

Which kernel version (uname -a)? Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

Logs The only log I have is from his stack trace using dbg

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0118d598 in brk ()
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /mnt/bigdata/stuff/gitea.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) where
#0  0x0118d598 in brk ()
#1  0x01167c14 in sbrk ()
#2  0x01140378 in __libc_setup_tls ()
#3  0x0113ff2c in __libc_start_main ()
#4  0x0001022c in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Additional context This answer from the ticket I opened at gitea github say if you use an older version of raspi kernel it works without problem: https://github.com/go-gitea/gitea/issues/8373#issuecomment-538751512

EDIT : https://github.molgen.mpg.de/git-mirror/glibc/blob/20003c49884422da7ffbc459cdeee768a6fee07b/csu/libc-tls.c#L105

This is the function where the segfault happens.

To save others following the link:

On my pi running kernel version: 4.19.66-v7+ gitea is working without problems. The other two pi´s - where gitea gave the "segmentation fault" - are running kernel version: 4.19.75-v7+.

Thanks for tracking this this issue. Have the same problem with gitea 1.9.2 to 1.9.4 (arm v6) on my RaspberryPI 4

$ cat /proc/device-tree/model Raspberry Pi 4 Model B Rev 1.1p

$ uname -a Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

$ cat /proc/version Linux version 4.19.75-v7l+ (dom@buildbot) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1270 SMP Tue Sep 24 18:51:41 BST 2019

I can confirm that this also affects my Raspberry Pi 3 Model B Rev 1.2 so not restricted to rpi4.

Another interesting stuff , when I execute gitea under dbg , it works and it can deploy its webserver. At least on archlinux for raspi 4. But outside dbg I got segfault . Why?

I have the same on my Raspberry Pi 3 B+.

Describe the bug When I launch gitea I got "segmentation fault".

To reproduce Download gitea binary 1.7.6, 1.8.0, 1.9.4 from here : https://dl.gitea.io/gitea/1.9.4/gitea-1.9.4-linux-arm-6 execute gitea web Get "segmentation fault"

Expected behaviour Gitea should execute and run the server

Actual behaviour I get segmentation fault and the server does not start

System Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

Which model of Raspberry Pi? Pi3B+
Which OS and version (cat /etc/rpi-issue)? Raspberry Pi reference 2019-07-10 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 175dfb027ffabd4b8d5080097af0e51ed9a4a56c, stage2
Which firmware version (vcgencmd version)? Oct 4 2019 15:20:55 Copyright (c) 2012 Broadcom version 8ebb91b5731d2595643426b007a9cee64217fc15 (tainted) (release) (start_cd)
Which kernel version (uname -a)? Linux 4.19.76-v7+ #1272 SMP Fri Oct 4 14:48:47 BST 2019 armv7l GNU/Linux

Even under raspbian running gitea under gdb gitea it works and I can connect to gitea webserver.

For someone like me that doesn't understand this stuff it's pretty crazy :)

here the screenshots : https://postimg.cc/gallery/2ba368sw2/

I haven't started looking at this yet, but you might be able to speed up the investigation if you can run Gitea under strace and capture the output.

Hi pelwell, this is when I execute strace ./gitea web

execve("./gitea", ["./gitea", "web"], 0xbe908704 /* 21 vars */) = 0
brk(NULL)                               = 0x15f2000
brk(0x15f2d18)                          = 0x15f2000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x24} ---
+++ killed by SIGSEGV (core dumped) +++

this is when I execute strace ./gitea

execve("./gitea", ["./gitea"], 0xbeaf3710 /* 21 vars */) = 0
brk(NULL)                               = 0xed7000
brk(0xed7d18)                           = 0xed7000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x24} ---
+++ killed by SIGSEGV (core dumped) +++

I suppose you want I run gitea under gdb and run "strace" but I have no idea how to do. Do you have some guide I can read for doing that?

Thanks!

Same here after the upgrade to kernel 4.19.75-v7l+ (SMP) armv7l. Before the upgrade I am running kernel 4.19.66 and gitea was working fine. After the upgrade with no other changes gitea won't start.

I can reproduce this - it's on the list to look at.

Is https://github.com/raspberrypi/linux/issues/3279 [raspberrypi-kernel-headers (1.20190925-2): Wrong exec format in scripts in buster] perhaps related?

Is there a way to temporarily downgrade the kernel again so Gitea can be run again? It seems the old kernel was removed from the repository

We never delete firmware builds - you can install the 4.19.66 kernel using sudo rpi-update 11ef4498.

I've found the culprit: https://github.com/raspberrypi/linux/commit/c0ccb4da26bc9eca366f2a437b9d35a749fdde15

It's now reverted on rpi-4.19.y, and Gitea runs again.

@kees Do you have an idea why this particular executable is failing on Pi?

I'm experiencing the same issue on 4.19.75-v7l+ with a statically compiled comskip binary. Same binary works on 4.19.71-v7l+

I was able to get a backtrace by forcing core dump (ulimit -c unlimited), then loading that dump into gdb:

[New LWP 6139]
Core was generated by `/mnt/channels/channels-dvr/latest/comskip video.mpg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x005b2090 in brk ()
(gdb) bt
#0  0x005b2090 in brk ()
#1  0x0058c07e in sbrk ()
#2  0x00564f26 in __libc_setup_tls ()
#3  0x0051d13c in __pthread_initialize_minimal_internal ()
#4  0x00564cfa in __libc_start_main ()
#5  0x0002fede in _start ()

Latest rpi-update kernel may have a fix for this.

FYI the Gitea and Kiwix segmentation faults (https://github.com/kiwix/kiwix-tools/issues/341) are NOT solved for me when I do apt update && apt -y dist-upgrade && reboot to upgrade raspberrypi-kernel (1.20190925+1-1) over (1.20190925-2).

CLARIF: the regression persists on (both) RPi 3 and RPi 4.

You need to run sudo rpi-update

You need to run sudo rpi-update

Why is that specifically? Given apt -y dist-upgrade says it already installed kernel (1.20190925+1-1) over (1.20190925-2) -- and then I rebooted.

Because the fix was made a few hours ago and is not contained in package from 9/25?

Because the fix was made a few hours ago and is not contained in package from 9/25?

Ok then I was mistaken in assuming https://github.com/raspberrypi/linux/issues/3279#issuecomment-541135632 was the fix. Apologies.

Certainly applying kernel (1.20190925+1-1) over (1.20190925-2) is insufficient.

I can confirm that running sudo rpi-update on a Raspberry Pi 4 fixed both problems -- allowing Gitea (iiab/iiab#1999) and Kiwix (iiab/iiab#1993) to start!

FYI kernel goes from...

root@box:~# uname -a
Linux box.lan 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

To...

root@box:~# uname -a
Linux box.lan 4.19.79-v7l+ #1273 SMP Fri Oct 11 18:23:45 BST 2019 armv7l GNU/Linux

Question: how many days is likely, until regular users get their kernel fixed here?

(via apt update etc)

I just run apt-get update && apt-get dist-upgrade, then rpi-update I am now on 4.19.79-v7+ #1273 SMP Fri Oct 11 18:13:16 BST 2019 armv7l GNU/Linux.

Now gitea is starting successfully with https://dl.gitea.io/gitea/1.9.4/gitea-1.9.4-linux-arm-6 Thank you so much!

Welp, it's resolved. I'm gonna close the ticket.

Thank you all so much for this. I've been wrestling with the issue on and off for almost a month before finding this.

@pelwell Any ideas when commit 68a2665 will make it's way into the mainstream kernel repositories so we can receive the fix via the normal apt update && apt upgrade process?

@XECDesign When are we likely to next bump the kernel package?

@pelwell @popcornmix I'm waiting for something new to show up on https://github.com/raspberrypi/firmware/commits/stable/

@pelwell @popcornmix I'm waiting for something new to show up on https://github.com/raspberrypi/firmware/commits/stable/

Looks like something new has since shown up? Anything else blocking a release? It's now been a few weeks. https://github.com/raspberrypi/firmware/commit/878d27e23ff2f1c9a35d1b1fc8fe71e06083b823

I'm hesitant to reach beyond the stable apt channel as the last rpi-update I had tried resulted in needing an sdcard wipe and reinstall from scratch.

In my case I am seeing this same segfault/stack with a build of Drone CI on an RPi4/4GB running 4.19.75-v7l+, specifically the build provided within this image: https://hub.docker.com/layers/drone/drone/latest/images/sha256-722f8062c3b683005ca1a1f96216b65bf0df1cc4a3a3aa0f2911d71307ce98ce .

I gave up waiting for a kernel release to finally be cut and tried an rpi-update across a few boards. So far, the board I tried it on happily didn't need a reformat this time, so that's cool. With the new 4.19.86-v7l+ kernel the segfaults I was seeing have gone away too -- also cool.

Now off to see if I can successfully rpi-update the remaining boards...

@pelwell @popcornmix I'm waiting for something new to show up on https://github.com/raspberrypi/firmware/commits/stable/

Please, can you release the new firmware, so that the fix will be available without rpi-update procedure?

The month has passed since the bug has been fixed, but new kernel has not yet been released:(

It will be released early in the new year. The guy who does it is on holiday.

If you need it urgently, AFAIK there are not any major issues with the code in rpi-update so that should be safe.

@JamesH65 We're now half-way through January, any chance we'll get an update soon?

Should be sometime next week. Been slightly delayed for various reasons (Xmas, holidays, illness etc).

@JamesH65 Good to know, thanks 👍

raspberrypi / linux

Segmentation fault with Gitea on a Raspi 4 #3271