search

Optware / Optware-ng

279 stars 52 forks source link

Segfault when running ipkg update #294

Closed mitzone closed 5 years ago

mitzone commented 5 years ago

Hello, I get a Segmentation fault when running ipkg update command. Ipkg with other parameter works. Only when downloading is involved I get a segfault.

I tested this on 2 machines until now, both running 3.10.105 kernel. On both, the crash error is :

server1 : traps: ipkg[22330] general protection ip:23 sp:2b error:244 in ipkg[8048000+103000] server2 : traps: ipkg[13239] general protection ip:23 sp:2b error:244 in ipkg[8048000+103000]

installed.txt

Thanks, Mihai.

alllexx88 commented 5 years ago

Hi! That's strange, the error looks to be specific to x86 platforms, however, I can't reproduce it with the same kernel version (Synology NAS):

root@DiskStation:~# uname -a
Linux DiskStation 3.10.105 #23739 SMP Wed Jun 20 22:40:01 CST 2018 x86_64 GNU/Linux synology_cedarview_412+
root@DiskStation:~# ipkg update
Downloading http://ipkg.nslu2-linux.org/optware-ng/buildroot-i686/Packages.gz.
Inflating http://ipkg.nslu2-linux.org/optware-ng/buildroot-i686/Packages.gz.
Updated list of available packages in /opt/lib/ipkg/lists/alllexx.

What's the output of uname -a in your case?

mitzone commented 5 years ago

Hello, I'm also using it on synologys. I have several. Scenarios is this:

SYNO 1 DS1512+ ALL OK with the latest and greatest versions of ipkg packages Linux ds1512 3.10.105 #23739 SMP Wed Jun 20 22:40:01 CST 2018 x86_64 GNU/Linux synology_cedarview_1512+

SYNO 2 DS1815+ ALL OK before upgrade; NOK after upgrade > ipkg update segfault Linux alliovault-1 3.10.105 #23739 SMP Tue Jul 10 00:16:57 CST 2018 x86_64 GNU/Linux synology_avoton_1815+

SYNO 3 DS3617XS (this is a VM): ALL OK before upgrade; NOK after upgrade > ipkg segfault Linux synoVM3 3.10.105 #23739 SMP Tue Jul 10 00:17:06 CST 2018 x86_64 GNU/Linux synology_broadwell_3617xs

It stopped working after upgrading to latest ipkg packages. Thanks.

alllexx88 commented 5 years ago

Well, according to this and this, the problem can be due to some i386-specific flags. However, I don't see them being used to build ipkg (it's actually opkg-0.2.4 -- just renamed for backward compatibility) binary anywhere.

Do you have a guide to configure a Syno VM like you did? It'd make things simpler if I could reproduce the issue.

Also, I'm currently a bit limited in what I can do to debug the issue, since the Optware builder isn't yet accessible for me to ssh in.

mitzone commented 5 years ago

The issue is not on the VM only. The VM is an xpenology running on esxi 6.7. If you have an esxi server I can send you some already configured files to build a new test VM, but again, this is not happening only on the VM. Thanks.

alllexx88 commented 5 years ago

Yeah, I got it's not only on VM, but it's not happening on my HW, hence a VM's a good way to reproduce the issue. I don't have an esxi server, but I could set it up, with a free license at least. So yes, I'd be thankful for the configured files.

mitzone commented 5 years ago

I generated the VM config. Can you give me on private or here your email address so I can put the archive on wetransfer? Cheers.

alllexx88 commented 5 years ago

Here's my email: https://github.com/Optware/Optware-ng/blob/054ca9a4ec033436220f1bb9e01586941fc6139e/AUTHORS#L85

Thanks.

alllexx88 commented 5 years ago

So, I didn't manage to setup an esxi server, so I used vmware workstation instead. Your sent boot image gives me this kind of error soon after boot. Previous version (DS3617xs 6.1 Jun's Mod V1.02-alpha) works fine (after I figured out I had to install precisely DSM build 6.1-15047), and I see no segfaults after installing packages you listed:

root@SynoVM:~# uname -a
Linux SynoVM 3.10.102 #15047 SMP Thu Feb 23 02:23:28 CST 2017 x86_64 GNU/Linux synology_broadwell_3617xs
root@SynoVM:~# ipkg list_installed
bash - 4.4.12-3
bzip2 - 1.0.6-2
cacerts - 20180326-1
e2fslibs - 1.42.12-2
file - 5.28-1
gconv-modules - 2.25-4
gdbm - 1.8.3-4
gettext - 0.19.8.1-5
glib - 2.47.3-2
glibc-opt - 2.25-1
iftop - 0.17-5
iotop - 0.4-1
ldd - 2.20-1
less - 444-1
libdb - 5.3.21-2
libffi - 3.2.1-1
libintl - 0.19.8.1-5
libnsl - 2.25-1
libpcap - 1.8.1-1
libstdc++ - 6.0.24-2
libunistring - 0.9.4-2
libusb1 - 1.0.19-2
lsof - 4.82-1
lzo - 2.09-1
mc - 4.8.17-4
module-init-tools - 3.15-1
nano - 2.3.6-2
ncurses - 5.7-7
ncurses-base - 5.7-7
ncursesw - 5.7-3
net-tools - 1.60-6
openssl - 1.0.2n-1
openvpn - 2.2.0-3
procps - 3.3.12-2
psmisc - 22.21-2
python26 - 2.6.9-5
readline - 6.1-2
slang - 2.2.4-5
sqlite - 3.8.1-3
strace - 4.10-1
sysstat - 9.0.3-2
wget - 1.19.1-3
zlib - 1.2.8-2

So, sorry, I can't reproduce your issue.

alllexx88 commented 5 years ago

OK, I managed to setup a working DSM 6.2 baremetal xpenology. Clearly ipkg update segfaults, and it's not a matter of installed packages: it even segfaults during bootstrip attempt. I'll try to look into it as soon as I have some time.

mitzone commented 5 years ago

Awesome! Thanks for taking your time to look into it.

alllexx88 commented 5 years ago

I discovered that vfork() crashes for some reason on DS3617xs, DSM6.2. I've patched ipkg to use fork() instead, and it appears to work fine. Currently Optware builder is down, so the fix isn't online yet. For now, here's a test binary for you: ipkg.zip

Please report back if it fixes the issue for your other NAS (DS1815+).

alllexx88 commented 5 years ago

Hm, segfaulting vfork() is a major issue on DS3617xs, DSM6.2, actually. Many packages end with segfaults: mc, gdb, gcc, and probably much more. I honestly haven't come up with a proper solution for this problem. Hopefully I'll figure it out later...

mitzone commented 5 years ago

Hello, Got my hands today on a RS18017xs+. This has an INTEL Xeon D-1531 CPU. Running DSM6.2 , kernel Linux localhost 3.10.105 #23739 SMP Tue Jul 10 00:17:06 CST 2018 x86_64 GNU/Linux synology_broadwell_rs18017xs+

I get a segfault when running the bootstrap script. Just sending you info, maybe it helps. Cheers.

root@localhost:/usr/local/mitzone# wget -O - http://ipkg.nslu2-linux.org/optware-ng/bootstrap/buildroot-i686-bootstrap.sh | sh --2018-09-10 17:15:50-- http://ipkg.nslu2-linux.org/optware-ng/bootstrap/buildroot-i686-bootstrap.sh Resolving ipkg.nslu2-linux.org... 23.141.224.193 Connecting to ipkg.nslu2-linux.org|23.141.224.193|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 931 [text/plain] Saving to: 'STDOUT'

2018-09-10 17:15:51 (137 MB/s) - written to stdout [931/931]

--2018-09-10 17:15:52-- http://ipkg.nslu2-linux.org/optware-ng/buildroot-i686/ipkg-static_0.2.4-1_i686.ipk Resolving ipkg.nslu2-linux.org... 23.141.224.193 Connecting to ipkg.nslu2-linux.org|23.141.224.193|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 481648 (470K) [text/plain] Saving to: '/tmp/ipkg-static_0.2.4-1_i686.ipk'

/tmp/ipkg-static_0.2.4-1_i686.ipk 100%[==================================================================================================================>] 470.36K 460KB/s in 1.0s

2018-09-10 17:15:54 (460 KB/s) - '/tmp/ipkg-static_0.2.4-1_i686.ipk' saved [481648/481648]

./data.tar.gz ./ ./opt/ ./opt/bin/ ./opt/bin/ipkg-static ./opt/bin/ipkg ./opt/bin/update-alternatives ./opt/var/ ./opt/var/lock/ ./opt/share/ ./opt/share/man/ ./opt/share/man/man1/ ./opt/share/man/man1/opkg-cl.1 ./opt/share/opkg/ ./opt/share/opkg/intercept/ ./opt/share/opkg/intercept/update-modules ./opt/share/opkg/intercept/depmod ./opt/share/opkg/intercept/ldconfig ./opt/etc/ ./opt/etc/ipkg/ ./opt/etc/ipkg/ipkg.conf ./opt/etc/ipkg.conf Bootstraping done Installing glibc-locale package to generate needed /opt/lib/locale/locale-archive

Downloading http://ipkg.nslu2-linux.org/optware-ng/buildroot-i686/Packages.gz. sh: line 18: 23478 Segmentation fault /opt/bin/ipkg update Unknown package 'glibc-locale'. Collected errors:

alllexx88 commented 5 years ago

There're still some technical issues that prevent me from updating the feeds. You can download the test binary ipkg.zip and unpack it on top of /opt/bin/ipkg

mitzone commented 5 years ago

Will try it and let you know. UPDATE: I noticed the current binary behaves differently on different machines, using same kernel version. Only difference is the CPU architecture. Can this be a microcode issue (Spectre/Meltdown mitigation)?

mitzone commented 5 years ago

Tested the provided archive. No segfaults.

mitzone commented 5 years ago

optware-NG-Before_After.xlsx

Ok, so using the new ipkg binary I was able to pull out a list of installed packages (same machine) after ipkg upgrade. Before upgrade, no segfaults. After upgrade, segfaults.

alllexx88 commented 5 years ago

Sorry for a prolonged period of silence. I didn't manage to make packages from the i686 feed to work on DS3617xs DSM 6.2. Yet there's no such issue on DSM 6.1. My best bet is that some kernel config was changed, leading to this kind of a problem. However, there're good news too. I'm now working on an x86_64 feed, and from what I can tell there're no segfaults with x86_64 binaries. I tried x86_64 mc and gdb, and they work fine, unlike the i686 ones.

Now working on x86_64 feed as time permits.

mitzone commented 5 years ago

Awesome! Can't wait to test the x64 feed.

alllexx88 commented 5 years ago

OK, feed built. Please test with

wget -O - http://ipkg.nslu2-linux.org/optware-ng/bootstrap/buildroot-x86_64-bootstrap.sh | sh

P.S. Don't forget to backup/remove /opt content beforehand/

mitzone commented 5 years ago

Hello, Just tested it on an 918+, latest DSM version. Linux coreVAULT-1 4.4.59+ #23824 SMP PREEMPT Fri Sep 7 12:53:00 CST 2018 x86_64 GNU/Linux synology_apollolake_918+

Everything looks ok so far. Thanks for taking your time doing this. Cheers!

RoBo-OnGitHub commented 5 years ago

Just to let you know / confirm previous findings. Also had ipkg 'Segmentation Faults' on Synology DS1817+ actions involving downloads: # /opt/bin/ipkg update Downloading http://ipkg.nslu2-linux.org/optware-ng/buildroot-i686/Packages.gz. Segmentation fault

# ./ipkg install iotop Installing iotop (0.4-1) on /opt/. Downloading http://ipkg.nslu2-linux.org/optware-ng/buildroot-i686/iotop_0.4-1_i686.ipk. Segmentation fault

Having used the ipkg file from this comment the issue was solved.

# ./ipkg update Downloading http://ipkg.nslu2-linux.org/optware-ng/buildroot-i686/Packages.gz. Inflating http://ipkg.nslu2-linux.org/optware-ng/buildroot-i686/Packages.gz. Updated list of available packages in /opt/lib/ipkg/lists/packages.

Thanks for your good work alllexx88 and success with the x86_64 feed.

phit42 commented 5 years ago

It might be related to a synology update: another package (rsl-sync) did not work on my side, and I received the following reply from support: Synology recently released their “DSM 6.2 Update 2” firmware. One change of the firmware is that 32-bit applications will no longer run correctly (we’re unclear if this was intended or not; we’re contacting Synology to gather more information). This means that anyone who previously either downloaded the 32-bit version of Plex Media Server or installed from Synology Package Center would see PMS fail to start. https://forums.plex.tv/t/synology-installation-changes-for-plex-media-server/284921

alllexx88 commented 5 years ago

Thank you for the update, @phit42 ! Based from my testing, some 32-bit syscalls indeed stopped working in DSM 6.2 (even before Update 2, at least on DS3617xs), so it indeed looks like a kernel config issue. I think this issue can be closed for now.