[SOLVED] Adding files causes crash: JSON decoder out of sync

[jphovila]

Hi!

I'm running Cryptletter version 3.1.2 inside a FreeBSD 13.2 jail.

Creating plain text messages works fine, but trying to add file attachments almost always results in:

http: panic serving xxx.xxx.xxx.xxx:yyyyy: JSON decoder out of sync - data changing underfoot?
goroutine 37 [running]:
net/http.(*conn).serve.func1()
        /usr/local/go120/src/net/http/server.go:1854 +0xbf
panic({0x9adb60, 0xf13d30})
        /usr/local/go120/src/runtime/panic.go:890 +0x263
encoding/json.(*decodeState).literalStore(0xc00013c3e8, {0xc000c0000b, 0x42, 0x7ffdf5}, {0x9adb60?, 0xc000126248?, 0x1?}, 0x0)
        /usr/local/go120/src/encoding/json/decode.go:946 +0x22f1
encoding/json.(*decodeState).value(0xc00013c3e8, {0x9adb60?, 0xc000126248?, 0xc000c00002?})
        /usr/local/go120/src/encoding/json/decode.go:388 +0x126
encoding/json.(*decodeState).object(0xc00013c3e8, {0x9986e0?, 0xc000126240?, 0xc0002575a8?})
        /usr/local/go120/src/encoding/json/decode.go:775 +0xd45
encoding/json.(*decodeState).value(0xc00013c3e8, {0x9986e0?, 0xc000126240?, 0xc0002575f8?})
        /usr/local/go120/src/encoding/json/decode.go:374 +0x45
encoding/json.(*decodeState).unmarshal(0xc00013c3e8, {0x9986e0?, 0xc000126240?})
        /usr/local/go120/src/encoding/json/decode.go:181 +0x168
encoding/json.(*Decoder).Decode(0xc00013c3c0, {0x9986e0, 0xc000126240})
        /usr/local/go120/src/encoding/json/stream.go:73 +0x191
github.com/Scribblerockerz/cryptletter/pkg/handler.NewMessageAction({0xf1b290, 0xc00014c2a0}, 0xc000136e00)
        /home/privnoted/Cryptletter/cryptletter/pkg/handler/handlers.go:241 +0xf0
net/http.HandlerFunc.ServeHTTP(0x100?, {0xf1b290?, 0xc00014c2a0?}, 0x40dce7?)
        /usr/local/go120/src/net/http/server.go:2122 +0x2f
github.com/Scribblerockerz/cryptletter/pkg/logger.HTTPLogger.func1({0xf1b290, 0xc00014c2a0}, 0xc000136e00)
        /home/privnoted/Cryptletter/cryptletter/pkg/logger/logger.go:17 +0x9e
net/http.HandlerFunc.ServeHTTP(0xc000136d00?, {0xf1b290?, 0xc00014c2a0?}, 0xc0001309e8?)
        /usr/local/go120/src/net/http/server.go:2122 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000180, {0xf1b290, 0xc00014c2a0}, 0xc000136c00)
        /home/privnoted/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
net/http.serverHandler.ServeHTTP({0xf19918?}, {0xf1b290, 0xc00014c2a0}, 0xc000136c00)
        /usr/local/go120/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc00013a5a0, {0xf1b8d0, 0xc0001e8f60})
        /usr/local/go120/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
        /usr/local/go120/src/net/http/server.go:3089 +0x5ed

With really tiny files, the crash does not always happen. But trying to attach a file of just 1 MB in size will definitely result in the above error.

Resource limits for the user account under which my Cryptletter daemon is running are:

Resource limits (current):
  cputime              infinity secs
  filesize             infinity kB
  datasize             33554432 kB
  stacksize              524288 kB
  coredumpsize         infinity kB
  memoryuse            infinity kB
  memorylocked           131072 kB
  maxprocesses            89999
  openfiles             3771063
  sbsize               infinity bytes
  vmemoryuse           infinity kB
  pseudo-terminals     infinity
  swapuse              infinity kB
  kqueues              infinity
  umtxp                infinity

Googling with the error message gave a bunch of results which may give some hints of where things go wrong, but nothing that would help me (as a server architect, not that much of a programmer) to fix the issue.

[Scribblerockerz]

Hey @jphovila, thanks for letting me know.

It looks like this race condition might be FreeBSD jail specific, at least I couldn't reproduce the error on mac nor PopOS! 22.04.

Still, the issue seems to be solved (at least for other repositories) by copying the memory from the request to an other buffer before decoding it.

https://github.com/Scribblerockerz/cryptletter/releases/tag/3.1.4

Let me know if the fix worked out!

[jphovila]

Hi!

My personal guesstimate would be that the issue has more to do with the Nginx reverse proxy (providing TLS encryption) than FreeBSD and/or it's jails. I tried tinkering around with parameters such as proxy_request_buffering but was not able to find a fix.

After the change you made (to 3.1.4) there's now a new problem. I think the solution is almost there though, since now it's only about the character \x00 present in a string literal:

2024/04/17 06:54:46 http: panic serving xxx.xxx.xxx.xxx:yyyyy: invalid character '\x00' in string literal
goroutine 9 [running]:
net/http.(*conn).serve.func1()
        /usr/local/go120/src/net/http/server.go:1854 +0xbf
panic({0x9c8a40, 0xc000010018})
        /usr/local/go120/src/runtime/panic.go:890 +0x263
github.com/Scribblerockerz/cryptletter/pkg/handler.NewMessageAction({0xd0a3d0, 0xc0000aa2a0}, 0xc00009ce00)
        /home/privnoted/Cryptletter/cryptletter/pkg/handler/handlers.go:254 +0xa85
net/http.HandlerFunc.ServeHTTP(0x100?, {0xd0a3d0?, 0xc0000aa2a0?}, 0x40dce7?)
        /usr/local/go120/src/net/http/server.go:2122 +0x2f
github.com/Scribblerockerz/cryptletter/pkg/logger.HTTPLogger.func1({0xd0a3d0, 0xc0000aa2a0}, 0xc00009ce00)
        /home/privnoted/Cryptletter/cryptletter/pkg/logger/logger.go:17 +0x9e
net/http.HandlerFunc.ServeHTTP(0xc00009cd00?, {0xd0a3d0?, 0xc0000aa2a0?}, 0xc0000949e8?)
        /usr/local/go120/src/net/http/server.go:2122 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00024c000, {0xd0a3d0, 0xc0000aa2a0}, 0xc00009cc00)
        /home/privnoted/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
net/http.serverHandler.ServeHTTP({0xd08a58?}, {0xd0a3d0, 0xc0000aa2a0}, 0xc00009cc00)
        /usr/local/go120/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc00009e5a0, {0xd0aa10, 0xc00022b260})
        /usr/local/go120/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
        /usr/local/go120/src/net/http/server.go:3089 +0x5ed

[Scribblerockerz]

I've setup an nginx as a reverse proxy (inside docker) but unfortunately, I couldn't reproduce the issue.

With the 3.1.5 release, I'm trimming any '\x00' bytes from the buffer.

https://github.com/Scribblerockerz/cryptletter/releases/tag/3.1.5

It would be great if you could check the the latest version.

[jphovila]

@Scribblerockerz thank you so much for your time and effort!

With 3.1.5 installed, the exact same error (invalid character '\x00' in string literal) still keeps happening.

I have checked and double-checked that it really is 3.1.5 I'm running.

What on earth could be causing this? 🤔

Would it help if I provide you an access to the FreeBSD jail? Or is there some other way I could provide you with more details?

[Scribblerockerz]

Hey @jphovila, I forgot that it's possible to declare FreeBSD as compilation target. Personally I never used it, so it did not come into my mind earlier.

I've build the 3.1.5 again for FreeBSD and attached the binary to the release.

If it's still failing, I might have to dig deeper into FreeBSD jails, so a hint on how to set it up would be great.

[jphovila]

Thank you again @Scribblerockerz!

I'll test the native FreeBSD version today and let you know the results.

I can create a dedicated jail with root rights for you, if it will be needed.

[jphovila]

@Scribblerockerz I'm happy to report that with the (pre-compiled) FreeBSD version, the problem appears to be solved now!

Next, I will try to compile the FreeBSD target myself, since the previous (failed) attempts were done using the Linux target.

Thank you so much for finding the cause and fixing the bug so quickly!

[jphovila]

P.S. The self-compiled version works properly now too, when using the freebsd target. While I'm just guessing, my guess would be that the cause of the original problem had something to do with how buffers are being handled in FreeBSD vs. in Linux. Excellent work, @Scribblerockerz!

[Scribblerockerz]

This is great news! I was glad to help.