Netatalk 3.1.13 segfaults in afpd

[mikaku]

Hello,

After upgrading from 3.1.12 to 3.1.13 I'm seeing segmentation fault messages every time a user logs in:

[...]
Apr 12 07:46:25 linux afpd[24262]: Login by user1 (AFP3.4)
Apr 12 07:46:25 linux afpd[24262]: ===============================================================
Apr 12 07:46:25 linux afpd[24262]: INTERNAL ERROR: Signal 11 in pid 24262 (3.1.13)
Apr 12 07:46:25 linux afpd[24262]: ===============================================================
Apr 12 07:46:25 linux afpd[24262]: PANIC: internal error
Apr 12 07:46:25 linux afpd[24262]: BACKTRACE: 12 stack frames:
Apr 12 07:46:25 linux afpd[24262]: #0 /lib64/libatalk.so.18(netatalk_panic+0x37) [0x7f2764267df7]
Apr 12 07:46:25 linux afpd[24262]: #1 /lib64/libatalk.so.18(+0x38f48) [0x7f2764267f48]
Apr 12 07:46:25 linux afpd[24262]: #2 /lib64/libc.so.6(+0x36400) [0x7f27601d5400]
Apr 12 07:46:25 linux afpd[24262]: #3 /lib64/libatalk.so.18(+0x17c40) [0x7f2764246c40]
Apr 12 07:46:25 linux afpd[24262]: #4 /lib64/libatalk.so.18(ad_open+0xfee) [0x7f2764248c4e]
Apr 12 07:46:25 linux afpd[24262]: #5 /usr/sbin/afpd(+0x31b6c) [0x55d4bf071b6c]
Apr 12 07:46:25 linux afpd[24262]: #6 /usr/sbin/afpd(+0x32905) [0x55d4bf072905]
Apr 12 07:46:25 linux afpd[24262]: #7 /usr/sbin/afpd(afp_openvol+0x500) [0x55d4bf0731e0]
Apr 12 07:46:25 linux afpd[24262]: #8 /usr/sbin/afpd(afp_over_dsi+0x58e) [0x55d4bf04ff7e]
Apr 12 07:46:25 linux afpd[24262]: #9 /usr/sbin/afpd(main+0xd29) [0x55d4bf04e1e9]
Apr 12 07:46:25 linux afpd[24262]: #10 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f27601c1555]
Apr 12 07:46:25 linux afpd[24262]: #11 /usr/sbin/afpd(+0xe250) [0x55d4bf04e250]
Apr 12 07:46:25 linux systemd-logind: Removed session 33688.
Apr 12 07:46:25 linux systemd: Removed slice User Slice of user1.
Apr 12 07:46:29 linux systemd: Created slice User Slice of user1.
Apr 12 07:46:29 linux systemd-logind: New session 33689 of user user1.
Apr 12 07:46:29 linux systemd: Started Session 33689 of user user1.
Apr 12 07:46:29 linux afpd[24273]: Login by user1 (AFP3.4)
Apr 12 07:46:29 linux afpd[24273]: ===============================================================
Apr 12 07:46:29 linux afpd[24273]: INTERNAL ERROR: Signal 11 in pid 24273 (3.1.13)
Apr 12 07:46:29 linux afpd[24273]: ===============================================================
Apr 12 07:46:29 linux afpd[24273]: PANIC: internal error
Apr 12 07:46:29 linux afpd[24273]: BACKTRACE: 12 stack frames:
Apr 12 07:46:29 linux afpd[24273]: #0 /lib64/libatalk.so.18(netatalk_panic+0x37) [0x7f2764267df7]
Apr 12 07:46:29 linux afpd[24273]: #1 /lib64/libatalk.so.18(+0x38f48) [0x7f2764267f48]
Apr 12 07:46:29 linux afpd[24273]: #2 /lib64/libc.so.6(+0x36400) [0x7f27601d5400]
Apr 12 07:46:29 linux afpd[24273]: #3 /lib64/libatalk.so.18(+0x17c40) [0x7f2764246c40]
Apr 12 07:46:29 linux afpd[24273]: #4 /lib64/libatalk.so.18(ad_open+0xfee) [0x7f2764248c4e]
Apr 12 07:46:29 linux afpd[24273]: #5 /usr/sbin/afpd(+0x31b6c) [0x55d4bf071b6c]
Apr 12 07:46:29 linux afpd[24273]: #6 /usr/sbin/afpd(+0x32905) [0x55d4bf072905]
Apr 12 07:46:29 linux afpd[24273]: #7 /usr/sbin/afpd(afp_openvol+0x500) [0x55d4bf0731e0]
Apr 12 07:46:29 linux afpd[24273]: #8 /usr/sbin/afpd(afp_over_dsi+0x58e) [0x55d4bf04ff7e]
Apr 12 07:46:29 linux afpd[24273]: #9 /usr/sbin/afpd(main+0xd29) [0x55d4bf04e1e9]
Apr 12 07:46:29 linux afpd[24273]: #10 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f27601c1555]
Apr 12 07:46:29 linux afpd[24273]: #11 /usr/sbin/afpd(+0xe250) [0x55d4bf04e250]
Apr 12 07:46:29 linux systemd-logind: Removed session 33689.
[...]

OS: CentOS Linux 7.9 Kernel: 3.10.0-1160.59.1.el7.x86_64 Package: netatalk-3.1.13-1.el7.x86_64

Let me know if you need more information. Thanks.

[knight-of-ni]

@mikaku checkout the conversation in #174 I've got experimental rpms, if you are willing to test and provide feedback.

[mikaku]

I've got experimental rpms, if you are willing to test and provide feedback.

Sure, I've just upgraded the system with the package netatalk-3.1.13-2.el7.x86_64.rpm, which I hope is your newest build. I said "I hope" because the web says it was built 17 days ago!!. I expected to see 2 or 3 days ago, but not 17. Didn't know this problem was so "old".

Anyway, I'll come back here in a while once I can confirm the users can login and work as they did with 3.1.12. Thanks.

[mikaku]

Forgot to say that since version 3.1.12 is no longer available on EPEL, the only way, on CentOS 7, to downgrade is downloading it from Koji:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1403661

[mikaku]

With the package netatalk-3.1.13-2.el7.x86_64.rpm installed I see messages like these:

[...]
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA0600A000220508.jpg"): deleted invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA1003D000220477.jpg"): invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA1003D000220477.jpg"): deleted invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA1006PTD0220358.jpg"): invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA1006PTD0220358.jpg"): deleted invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA1003D000220439.jpg"): invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA1003D000220439.jpg"): deleted invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA0809PDT0220420.jpg"): invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA0809PDT0220420.jpg"): deleted invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA1004D000220382.jpg"): invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA1004D000220382.jpg"): deleted invalid metadata EA
May  3 09:45:31 linux afpd[22083]: ad_header_read_ea("/u/applefas5/TA1001PDT0220414.jpg"): invalid metadata EA
[...]

May  3 09:45:41 linux afpd[22083]: PANIC: Can't seteuid back
May  3 09:45:41 linux afpd[22083]: BACKTRACE: 9 stack frames:
May  3 09:45:41 linux afpd[22083]: #0 /lib64/libatalk.so.18(netatalk_panic+0x37) [0x7f252dcc4087]
May  3 09:45:41 linux afpd[22083]: #1 /lib64/libatalk.so.18(unbecome_root+0x3c) [0x7f252dccd92c]
May  3 09:45:41 linux afpd[22083]: #2 /lib64/libatalk.so.18(ad_metadata+0x65) [0x7f252dca5bd5]
May  3 09:45:41 linux afpd[22083]: #3 /usr/sbin/afpd(getfilparams+0x9b) [0x5585cb11207b]
May  3 09:45:41 linux afpd[22083]: #4 /usr/sbin/afpd(afp_resolveid+0x30a) [0x5585cb11456a]
May  3 09:45:41 linux afpd[22083]: #5 /usr/sbin/afpd(afp_over_dsi+0x58e) [0x5585cb0fef7e]
May  3 09:45:41 linux afpd[22083]: #6 /usr/sbin/afpd(main+0xd29) [0x5585cb0fd1e9]
May  3 09:45:41 linux afpd[22083]: #7 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f2529c1d555]
May  3 09:45:41 linux afpd[22083]: #8 /usr/sbin/afpd(+0xe250) [0x5585cb0fd250]

Since these messages were not present with the previous version, I've downgraded to 3.1.12 again. Let me know if you want me to test a new version.

[knight-of-ni]

Thanks for the feedback. I'll ping the author of pr #174

I said "I hope" because the web says it was built 17 days ago!!. I expected to see 2 or 3 days ago, but not 17. Didn't know this problem was so "old".

PR #174, which is what my copr rpms use, was created 22 days ago.

[knight-of-ni]

I've gone ahead and built 3.1.12 in my copr repo, to make it a little easier to downgrade. I'll announce that in redhat bugzilla as well.

[mikaku]

I've just upgraded my server with the latest version: netatalk-3.1.13-3.el7. I'll be checking how is functioning within the next hours.

[mikaku]

No errors and no problems appeared using the latest version. You hit the nail on the head this time! :+1:

[knight-of-ni]

Great glad to hear! Thanks goes to @anodos325 for doing the hard part. I simply patched his PR against the 3.1.13 tarball.

[anodos325]

Okay. It's important to know that with this patchset the error handling for an AFP metadata xattr that fails to parse is different. Original code was to delete xattr and generate new one. Current behavior in this PR is to AFP_ASSERT(), which crashes netatalk and may generate corefile. The reason for this is so that we avoid removing xattr if people discover a new parsing bug (fail safe from user data standpoint), and give package maintainer the opportunity to see what went wrong.

I presume that eventually this (the AFP_ASSERT()) can be removed before final merge / new release to restore original behavior (deleting xattr and generating new one). This does expose ability for malicious local user to basically DOS a path on the netatalk server by writing junk data to an AFP metadata xattr. I think this is an acceptable risk for what is WIP / pending PR while it continues to be tested.

[mikaku]

Hello,

I'm not sure if this is still related to this issue, but I see those messages after upgrading to 3.1.13:

Dec 22 12:48:25 linux afpd[18995]: parse_entries: bogus eid: 9, off: 50, len: 3760
Dec 22 12:48:25 linux afpd[18995]: ad_header_read(/u/applepublic/XXX/D06037138/._pont-aven-9089-42_w400.jpg): malformed AppleDouble
Dec 22 12:48:25 linux afpd[18995]: ad_header_read_osx(rfpath, ad, &st) failed: Input/output error
Dec 22 12:48:25 linux afpd[18995]: afp_openfork(pont-aven-9089-42_w400.jpg): ad_open: Input/output error

Any idea?

OS: CentOS 7 Linux Uname: Linux linux.xxxxxxxxxx.lan 3.10.0-1160.80.1.el7.x86_64 #1 SMP Tue Nov 8 15:48:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Netatalk: netatalk-3.1.13-3.el7.x86_64

[rdmark]

@mikaku Can you please upgrade to 3.1.14 and try to reproduce your issue there? If it still persists, please file a new issue ticket.

[mikaku]

@rdmark Sure, you mean that one still in testing right?

https://dl.fedoraproject.org/pub/epel/testing/7/x86_64/Packages/n/netatalk-3.1.14-3.el7.x86_64.rpm

[knight-of-ni]

@mikaku you can upgrade to 3.1.14 today with a simple yum/dnf upgrade. 3.1.14 has been in fedora & epel repos since Jan 12. https://src.fedoraproject.org/rpms/netatalk/c/484cef3c18595ba0c09776c42cb62508e65beced?branch=rawhide

The latest 3.1.14-3 was pushed yesterday, to fix a CVE. It is not related to the issue you experiencing.

[mikaku]

If 3.1.14-3 is not necessary then my server is already using the 3.1.14:

# rpm -q netatalk --last
netatalk-3.1.14-1.el7.x86_64                  Sun 05 Feb 2023 09:13:10 AM CET

[knight-of-ni]

Since this issue is still occurring with 3.1.14, you may want to create a new issue and identify it as such, to get the developers attention.

I just noticed release notes for 3.1.15 were just committed to master, so it looks like a new release is imminent. Don't know if that will help, but I'll build a new set of rpms once the release is made official.

[rdmark]

@mikaku Did the problematic volumes by any chance start out as netatalk2 volumes and then converted to netatalk3 at some point over the years?

[mikaku]

@rdmark, I've opened the new issue #270, I've answered your question there. Thanks.