mu4e unable to parse response from "mu server" due to missing size cookie

halfacanuck commented 1 year ago

(Update: A temporary workaround can be found in https://github.com/termux/termux-packages/issues/16191#issuecomment-1510018138)

(Update 2: A proposed patch can be found in https://github.com/termux/termux-packages/issues/16191#issuecomment-1510034516)

I'm using mu 1.10.2 and emacs 28.3-rc1-0. Upon issuing M-x mu4e I get a message that it found the correct version of the binary, but nothing further happens.

ps -efw reveals the mu server process is running, and it's visible inside the emacs processes list.

Tracing with edebug leads to the conclusion that mu4e--server-filter is failing to parse the response from the server to its commands because the server is missing the "length cookie" at the beginning of the response. It is supposed to be of the form "\xfeXX\xff", where "XX"is the response length in hex, or "[XX]" if the server was invoked from a tty.

When I invoke mu server in the emacs shell I do indeed see the cookie in the form "[XX]", but when I pipe "(ping)" into the server and pipe the server's output to hexdump -C, the cookie appears in neither form. It is simply missing, as is the case when invoked by mu4e, as confirmed by inspecting the mu4e--server-buf variable during execution of mu4e--server-filter.

Inspection of the mu source code reveals this to be very odd, because as far as I can see the code has no choice but to issue one of the two forms of the cookie.

The problem doesn't appear to be the 8-bit characters themselves because echo -e "\xfe03\xffwut" | hexdump -C shows the expected output.

I have set UTF-8 as the language environment within emacs as recommended in the mu FAQ, and it makes no difference to this strange behavior.

It probably goes without saying that Termux's mu4e is completely unusable for me in its current state.

xtkoba commented 1 year ago

A bug report should be filed with the appropriate template.

Please share your termux-info output.

halfacanuck commented 1 year ago

Termux Variables:
TERMUX_APK_RELEASE=F_DROID
TERMUX_APP_PACKAGE_MANAGER=apt
TERMUX_APP_PID=6107
TERMUX_IS_DEBUGGABLE_BUILD=0
TERMUX_MAIN_PACKAGE_FORMAT=debian
TERMUX_VERSION=0.118.0
Packages CPU architecture:
arm
Subscribed repositories:
# sources.list
deb https://mirror.vern.cc/termux/termux-main stable main
Updatable packages:
libxml2/stable 2.10.4 arm [upgradable from: 2.10.3-1]
vim-runtime/stable 9.0.1450 all [upgradable from: 9.0.1400]
vim/stable 9.0.1450 arm [upgradable from: 9.0.1400]
termux-tools version:
1.37.0
Android version:
7.1.2
Kernel build information:
Linux localhost 3.10.49-g387418f #1 SMP PREEMPT Tue Sep 18 15:54:42 CST 2018 armv7l Android
Device manufacturer:
Haier
Device model:
Haier G32
LD Variables:
LD_LIBRARY_PATH=
LD_PRELOAD=/data/data/com.termux/files/usr/lib/libtermux-exec.so

halfacanuck commented 1 year ago

When I grep for the tty-compatible cookie format string [1] in the mu binary, it is found:

grep -U $'\\[%x\\]\x00' $PREFIX/bin/mu
grep: /data/data/com.termux/files/usr/bin/mu: binary file matches

Changing the regular expression causes the match to fail:

~ $ grep -U $'\\[%x\\}\x00' $PREFIX/bin/mu
~ $

Searching for the pipe version of the format string succeeds:

~ $ grep -U $'\xfe%x\xff\x00' $PREFIX/bin/mu
grep: /data/data/com.termux/files/usr/bin/mu: binary file matches

Changing the first 8-bit character causes the match to fail:

~ $ grep -U $'\xfd%x\xff\x00' $PREFIX/bin/mu
~ $

Thus the correct printf format strings do appear in the binary.

[1] https://github.com/djcb/mu/blob/release/1.10/mu/mu-cmd-server.cc#L65

halfacanuck commented 1 year ago

Here is the output when I test mu server in a shell, which can be seen outputting the tty-compatible cookie:

~ $ mu server
;; Welcome to the mu 1.10.2 command-server
;; Use (help) to get a list of commands, (quit) to quit.
;; mu> (ping)
[XX](:pong "mu" :props (:version "1.10.2" :personal-addresses ("<elided>") :database-path "/data/data/com.termux/files/home/.cache/mu/xapian" :root-maildir "/data/data/com.termux/files/home/Maildir" :doccount 28))

(I have elided the cookie contents for privacy reasons. The actual output is the correct two hex characters.)

Here is the output when I simulate how it is invoked by mu4e, which is with input from and output to pipes, and which can be seen (at offset 6bh) to be missing the cookie:

~ $ echo "(ping)" | mu server | hexdump -C
00000000  3b 3b 20 57 65 6c 63 6f  6d 65 20 74 6f 20 74 68  |;; Welcome to th|
00000010  65 20 6d 75 20 31 2e 31  30 2e 32 20 63 6f 6d 6d  |e mu 1.10.2 comm|
00000020  61 6e 64 2d 73 65 72 76  65 72 0a 3b 3b 20 55 73  |and-server.;; Us|
00000030  65 20 28 68 65 6c 70 29  20 74 6f 20 67 65 74 20  |e (help) to get |
00000040  61 20 6c 69 73 74 20 6f  66 20 63 6f 6d 6d 61 6e  |a list of comman|
00000050  64 73 2c 20 28 71 75 69  74 29 20 74 6f 20 71 75  |ds, (quit) to qu|
00000060  69 74 2e 0a 3b 3b 20 6d  75 3e 20 28 3a 70 6f 6e  |it..;; mu> (:pon|
00000070  67 20 22 6d 75 22 20 3a  70 72 6f 70 73 20 28 3a  |g "mu" :props (:|
00000080  76 65 72 73 69 6f 6e 20  22 31 2e 31 30 2e 32 22  |version "1.10.2"|
00000090  20 3a 70 65 72 73 6f 6e  61 6c 2d 61 64 64 72 65  | :personal-addre|
000000a0  73 73 65 73 20 28 22 XX  XX XX XX XX XX XX XX XX  |sses ("<elided>

halfacanuck commented 1 year ago

When I hex-edit the mu binary such that the pipe version of the format string is "{%x}" instead of "\xfe%x\xff" then the cookie appears in the output as expected (again, I've elided the cookie contents):

~ $ echo "(ping)" | bin/mu.mod server | hexdump -C
00000000  3b 3b 20 57 65 6c 63 6f  6d 65 20 74 6f 20 74 68  |;; Welcome to th|
00000010  65 20 6d 75 20 31 2e 31  30 2e 32 20 63 6f 6d 6d  |e mu 1.10.2 comm|
00000020  61 6e 64 2d 73 65 72 76  65 72 0a 3b 3b 20 55 73  |and-server.;; Us|
00000030  65 20 28 68 65 6c 70 29  20 74 6f 20 67 65 74 20  |e (help) to get |
00000040  61 20 6c 69 73 74 20 6f  66 20 63 6f 6d 6d 61 6e  |a list of comman|
00000050  64 73 2c 20 28 71 75 69  74 29 20 74 6f 20 71 75  |ds, (quit) to qu|
00000060  69 74 2e 0a 3b 3b 20 6d  75 3e 20 7b XX XX 7d 28  |it..;; mu> {XX}(|
00000070  3a 70 6f 6e 67 20 22 6d  75 22 20 3a 70 72 6f 70  |:pong "mu" :prop|
00000080  73 20 28 3a 76 65 72 73  69 6f 6e 20 22 31 2e 31  |s (:version "1.1|
00000090  30 2e 32 22 20 3a 70 65  72 73 6f 6e 61 6c 2d 61  |0.2" :personal-a|
000000a0  64 64 72 65 73 73 65 73  20 28 22 XX XX XX XX XX  |ddresses ("<elided>

When I modify the format string to be "\x01%x\x02" the cookie remains:

~ $ echo "(ping)" | bin/mu.mod server | hexdump -C
00000000  3b 3b 20 57 65 6c 63 6f  6d 65 20 74 6f 20 74 68  |;; Welcome to th|
00000010  65 20 6d 75 20 31 2e 31  30 2e 32 20 63 6f 6d 6d  |e mu 1.10.2 comm|
00000020  61 6e 64 2d 73 65 72 76  65 72 0a 3b 3b 20 55 73  |and-server.;; Us|
00000030  65 20 28 68 65 6c 70 29  20 74 6f 20 67 65 74 20  |e (help) to get |
00000040  61 20 6c 69 73 74 20 6f  66 20 63 6f 6d 6d 61 6e  |a list of comman|
00000050  64 73 2c 20 28 71 75 69  74 29 20 74 6f 20 71 75  |ds, (quit) to qu|
00000060  69 74 2e 0a 3b 3b 20 6d  75 3e 20 01 XX XX 02 28  |it..;; mu> .XX.(|
00000070  3a 70 6f 6e 67 20 22 6d  75 22 20 3a 70 72 6f 70  |:pong "mu" :prop|
00000080  73 20 28 3a 76 65 72 73  69 6f 6e 20 22 31 2e 31  |s (:version "1.1|
00000090  30 2e 32 22 20 3a 70 65  72 73 6f 6e 61 6c 2d 61  |0.2" :personal-a|
000000a0  64 64 72 65 73 73 65 73  20 28 22 XX XX XX XX XX  |ddresses ("<elided>

When I change the format string to "\x80%x\x81" (i.e., with most-significant bits set) the cookie disappears:

~ $ echo "(ping)" | bin/mu.mod server | hexdump -C
00000000  3b 3b 20 57 65 6c 63 6f  6d 65 20 74 6f 20 74 68  |;; Welcome to th|
00000010  65 20 6d 75 20 31 2e 31  30 2e 32 20 63 6f 6d 6d  |e mu 1.10.2 comm|
00000020  61 6e 64 2d 73 65 72 76  65 72 0a 3b 3b 20 55 73  |and-server.;; Us|
00000030  65 20 28 68 65 6c 70 29  20 74 6f 20 67 65 74 20  |e (help) to get |
00000040  61 20 6c 69 73 74 20 6f  66 20 63 6f 6d 6d 61 6e  |a list of comman|
00000050  64 73 2c 20 28 71 75 69  74 29 20 74 6f 20 71 75  |ds, (quit) to qu|
00000060  69 74 2e 0a 3b 3b 20 6d  75 3e 20 28 3a 70 6f 6e  |it..;; mu> (:pon|
00000070  67 20 22 6d 75 22 20 3a  70 72 6f 70 73 20 28 3a  |g "mu" :props (:|
00000080  76 65 72 73 69 6f 6e 20  22 31 2e 31 30 2e 32 22  |version "1.10.2"|
00000090  20 3a 70 65 72 73 6f 6e  61 6c 2d 61 64 64 72 65  | :personal-addre|
000000a0  73 73 65 73 20 28 22 XX  XX XX XX XX XX XX XX XX  |sses ("<elided>

Thus I conclude something doesn't like the non-ASCII characters--though, as noted above, the following works:

~ $ echo -e "\xfe03\xffwut" | hexdump -C
00000000  fe 30 33 ff 77 75 74 0a                           |.03.wut.|
00000008

halfacanuck commented 1 year ago

As a temporary workaround I have modified the mu 1.10.2 ARM binary to use "\x02%x\x03" as the pipe version of the cookie format string (by changing the byte at 29de7h to 2 and the byte at 29deah to 3), and modified the mu4e regular expression appropriately:

(setq mu4e--server-cookie-pre "\002"
      mu4e--server-cookie-post "\003"
      mu4e--server-cookie-matcher-rx "\002\\([[:xdigit:]]+\\)\003"
      mu4e-mu-binary (concat (getenv "HOME") "/bin/mu.mod"))

With this (ugly but rather pleasing) hack mu4e appears to work as expected--though might fail if an email, email address, etc. contains either the ASCII "start of text" or "end of text" characters. The mu author picked '\xfe' and '\xff' specifically because (he says) they cannot appear in correctly encoded UTF-8 strings, thus this "fix" is not in the spirit of his intention.

halfacanuck commented 1 year ago

I suspect the cause of this issue is the same as mentioned in this thread:

https://stackoverflow.com/questions/35354221/sprintf-not-working-when-format-contains-non-ascii-characters

It seems Android's libc implementation (Bionic)--at least at the time of this phone's production--treats any non-ASCII character in a printf/sprintf/etc. format string to mean the same as '\0'.

If this is the case then it's definitely a bug in this phone's libc; but should it also be considered a bug in Termux and/or mu? Should I report it upstream instead of/in addition to here? I'm slightly unwilling to do that until this is confirmed as the cause, but could probably be persuaded.

halfacanuck commented 1 year ago

Assuming this is indeed a bug in (old versions of?) the Bionic libc, the following patch to mu should work around it:

--- mu-cmd-server.cc.orig   2023-04-15 22:43:43.037347544 -0500
+++ mu-cmd-server.cc    2023-04-16 00:57:57.012167932 -0500
@@ -77,7 +77,7 @@
    if (tty) // for testing.
        ::printf("[%x]", num);
    else
-       ::printf(COOKIE_PRE "%x" COOKIE_POST, num);
+       ::printf("%s%x%s", COOKIE_PRE, num, COOKIE_POST);
 }

 static void

That said, I'm not able myself to create a test package with the change.

termux / termux-packages

mu4e unable to parse response from "mu server" due to missing size cookie #16191