teddycloud crashes as soon as I open the website

[justM4D]

Version: tc_v0.3.5@sha256:c3944721895dee3cdf50d1b5713bdc1a22e429e252b3ef1e11d6747171321d39 System: Unraid as Docker and from within a VM with Docker, both times with its own IP

I had the service running once successfully and got it somewhat configured and connected to the toniebox.

A few days later (without changing anything as far as I know) I couldn't get it to run again.

I tried

• installing the docker container on different systems
• checked all files
• changed directories
• let it generate new configs/certs
• set permissions to 777 and also changed owner to the same user that is set with self generated files

Just now I noticed that after starting the service and just looking at the log, it does interact with my box (like changing the Tonie and even sends it "local" content for my custom ID). But as soon as I open the website, it gets loaded and the service crashes down with following log output:

INFO |tls_adapter.c:0205:read_certificate()| File '/etc/teddycloud/certs/server/teddy-key.pem' detected as DER style RSA PRIVATE KEY INFO |tls_adapter.c:0201:read_certificate()| File '/etc/teddycloud/certs/client/ca.der' detected as DER style CERTIFICATE INFO |tls_adapter.c:0201:read_certificate()| File '/etc/teddycloud/certs/client/client.der' detected as DER style CERTIFICATE INFO |tls_adapter.c:0205:read_certificate()| File '/etc/teddycloud/certs/client/private.der' detected as DER style RSA PRIVATE KEY INFO |settings.c:0659:settings_load_ovl()| Load settings from /etc/teddycloud/config/config.overlay.ini INFO |tls_adapter.c:0208:read_certificate()| File '/etc/teddycloud/certs/server/ca-root.pem' assumed PEM style INFO |tls_adapter.c:0205:read_certificate()| File '/etc/teddycloud/certs/server/ca-key.pem' detected as DER style RSA PRIVATE KEY INFO |tls_adapter.c:0208:read_certificate()| File '/etc/teddycloud/certs/server/teddy-cert.pem' assumed PEM style INFO |tls_adapter.c:0205:read_certificate()| File '/etc/teddycloud/certs/server/teddy-key.pem' detected as DER style RSA PRIVATE KEY INFO |tls_adapter.c:0201:read_certificate()| File '/etc/teddycloud/certs/client/ca.der' detected as DER style CERTIFICATE INFO |tls_adapter.c:0201:read_certificate()| File '/etc/teddycloud/certs/client/client.der' detected as DER style CERTIFICATE INFO |tls_adapter.c:0205:read_certificate()| File '/etc/teddycloud/certs/client/private.der' detected as DER style RSA PRIVATE KEY INFO |tls_adapter.c:0394:tls_adapter_init()| Loading certificates... INFO |tls_adapter.c:0208:read_certificate()| File '/etc/teddycloud/certs/server/ca-root.pem' assumed PEM style INFO |tls_adapter.c:0205:read_certificate()| File '/etc/teddycloud/certs/server/ca-key.pem' detected as DER style RSA PRIVATE KEY INFO |tls_adapter.c:0208:read_certificate()| File '/etc/teddycloud/certs/server/teddy-cert.pem' assumed PEM style INFO |tls_adapter.c:0205:read_certificate()| File '/etc/teddycloud/certs/server/teddy-key.pem' detected as DER style RSA PRIVATE KEY INFO |tls_adapter.c:0201:read_certificate()| File '/etc/teddycloud/certs/client/ca.der' detected as DER style CERTIFICATE INFO |tls_adapter.c:0201:read_certificate()| File '/etc/teddycloud/certs/client/client.der' detected as DER style CERTIFICATE INFO |tls_adapter.c:0205:read_certificate()| File '/etc/teddycloud/certs/client/private.der' detected as DER style RSA PRIVATE KEY INFO |toniesJson.c:0057:tonies_update()| Updating tonies.json from GitHub... INFO |cloud_request.c:0158:web_request()| Connecting to HTTP server raw.githubusercontent.com:443... INFO |cloud_request.c:0208:web_request()| trying IP: 185.199.108.133 INFO |cloud_request.c:0036:httpClientTlsInitCallbackBase()| Initializing TLS...

cyclone/cyclone_crypto/cipher/aes.c:260:47: runtime error: left shift of 136 by 24 places cannot be represented in type 'int' cyclone/cyclone_crypto/cipher/aes.c:268:55: runtime error: left shift of 141 by 24 places cannot be represented in type 'int' cyclone/cyclone_crypto/cipher/aes.c:385:34: runtime error: left shift of 249 by 24 places cannot be represented in type 'int' cyclone/cyclone_crypto/cipher/aes.c:400:34: runtime error: left shift of 155 by 24 places cannot be represented in type 'int'

INFO |cloud_request.c:0071:httpClientTlsInitCallbackBase()| Initializing TLS done cyclone/cyclone_crypto/cipher/aes.c:390:34: runtime error: left shift of 134 by 24 places cannot be represented in type 'int' cyclone/cyclone_crypto/cipher/aes.c:395:34: runtime error: left shift of 197 by 24 places cannot be represented in type 'int' src/cyclone/cyclone_crypto/mpi.c:792:48: runtime error: left shift of 141 by 24 places cannot be represented in type 'int'

INFO |cloud_request.c:0308:web_request()| HTTP code: 200 INFO |cloud_request.c:0339:web_request()| Content-Type is text/plain; charset=utf-8 INFO |cloud_request.c:0417:web_request()| Connection closed INFO |toniesJson.c:0079:tonies_update()| ... success updating tonies.json from GitHub, reloading INFO |toniesJson.c:0157:tonies_readJson()| Trying to read /etc/teddycloud/config/tonies.custom.json with size 2 INFO |toniesJson.c:0157:tonies_readJson()| Trying to read /etc/teddycloud/config/tonies.json with size 2 INFO |server.c:0569:server_init()| 1 open HTTPS connections

cyclone/cyclone_crypto/cipher/aes.c:496:35: runtime error: left shift of 140 by 24 places cannot be represented in type 'int' cyclone/cyclone_crypto/cipher/aes.c:501:35: runtime error: left shift of 180 by 24 places cannot be represented in type 'int' cyclone/cyclone_crypto/cipher/aes.c:506:35: runtime error: left shift of 203 by 24 places cannot be represented in type 'int' cyclone/cyclone_crypto/cipher/aes.c:511:35: runtime error: left shift of 161 by 24 places cannot be represented in type 'int'

INFO |mqtt.c:0698:mqtt_init_box()| Registered new box 'teddyCloud Box 50F14AB2' (cn: '50F14AB2') INFO |mqtt.c:0699:mqtt_init_box()| Using base path 'teddyCloud/box/50F14AB2' and id 'teddyCloud_Box_50F14AB2'

================================================================= ==7==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000020080 at pc 0x55b1d001b735 bp 0x7f80fa9fda40 sp 0x7f80fa9fd1e8 READ of size 522 at 0x615000020080 thread T2

0 0x55b1d001b734 in __interceptor_strchr.part.0 (/usr/local/bin/teddycloud+0x58b734)

#1 0x55b1d052c51e in socketReceive src/platform/platform_linux.c:274
#2 0x55b1d02c4121 in httpReceive src/cyclone/cyclone_tcp/http/http_server_misc.c:1052
#3 0x55b1d02c4121 in httpReadHeaderField src/cyclone/cyclone_tcp/http/http_server_misc.c:371
#4 0x55b1d02c8f80 in httpReadRequestHeader src/cyclone/cyclone_tcp/http/http_server_misc.c:150
#5 0x55b1d02c2534 in httpConnectionTask src/cyclone/cyclone_tcp/http/http_server.c:461
#6 0x7f80fe923ac2  (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
#7 0x7f80fe9b584f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)

0x615000020080 is located 0 bytes to the right of 512-byte region [0x61500001fe80,0x615000020080) allocated by thread T2 here:

0 0x55b1d0092037 in __interceptor_malloc (/usr/local/bin/teddycloud+0x602037)

#1 0x55b1d052c757 in socketReceive src/platform/platform_linux.c:257

Thread T2 created by T0 here:

0 0x55b1d0035e55 in pthread_create (/usr/local/bin/teddycloud+0x5a5e55)

#1 0x55b1d052e553 in osCreateTask src/cyclone/common/os_port_posix.c:87

SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/local/bin/teddycloud+0x58b734) in __interceptor_strchr.part.0 Shadow bytes around the buggy address: 0x0c2a7fffbfc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c2a7fffbfd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c2a7fffbfe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c2a7fffbff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c2a7fffc000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c2a7fffc010:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c2a7fffc020: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c2a7fffc030: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c2a7fffc040: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c2a7fffc050: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa 0x0c2a7fffc060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==7==ABORTING

This is what the Browser Dev Tools show as errors:

[dermigoe]

I have also noticed this behavior. However, as I only recently started using Teddycloud, I thought I had made a mistake during the installation.

What always works for me at the moment is the browser's incognito mode. This prevents the container from crashing.

I have also tried deactivating all extensions, but even then the container crashes in normal browser mode when I go to the Teddycloud website.

[SciLor]

It would be interesting to have to http communication to the source of the crash.

[justM4D]

What always works for me at the moment is the browser's incognito mode. This prevents the container from crashing.

Good catch. I can confirm it works in incognito mode.

It would be interesting to have to http communication to the source of the crash.

I'd try debugging the problem, but I'm not set up for c++ development and am a bit short on time for fun like that. If you have a tip on what I can do to help, I'll try.

[SciLor]

I think it may already help to catch the data from the browser send to the cloud

[justM4D]

After it loaded successfully in incognito mode, it now also works in a normal browser session without crashing, even after restarting the container and whole host VM.

So I can't recreate it now :(

[SciLor]

May be related to

https://github.com/toniebox-reverse-engineering/teddycloud/issues/121

[dermigoe]

Unfortunately, I am not a software developer or network specialist. However, I was able to use Wireshark to record the network traffic in "normal" browser mode and in incognito mode. I noticed something that might help to narrow down the problem. In the first screenshot you can see that the error occurs about 1.3 seconds after starting the communication with the server.

I then filtered for HTTP. In the second screenshot you can see the communication with the server crash on the left and the communication without crash in incognito mode on the right.

The HTTP packets are smaller in incognito mode and, above all, no cookies are transmitted. Since Grafana is also running on my Docker server, for example, the cookie for Grafana is probably also transmitted by the browser. However, Grafana and Teddycloud have different ports and I don't know whether this is a browser error or normal behavior because the IP address is identical. You can also see that there is no response to the /v1/time HTTP/1.1 packet on the left side.

Anyway, after I cleared the browser cache, I was able to communicate with Teddycloud again in "normal" browser mode without causing a crash.

P.S: My native language is German and since my English is unfortunately only rudimentary, I used DeepL to translate this text.

[SciLor]

Thank you for digging in. This may help me to reproduce the problem and fix it.

[dermigoe]

I have saved the log files. If it would help, I could also make these files available. But I would need some help or a good program or script to anonymize the files.

[SciLor]

The previous info is already enough.

[justM4D]

I just retried it on my other computer and it crashes again. I think dermigoe might be right about the cookies.

I've played around with different configurations of running teddycloud directly on my unraid or in a separate vm, but I've always renamed the host to "tc", so it's reachable as "tc.fritz.box".

My browser has several cookies saved for the domain "tc.fritz.box" now, which are meant for other services.

After restarting the service and re-activating the tc-tab (without a refresh), it crashed again. I deleted the cookies, and now it works in normal browser mode.

So my assumption is that either the existance of "any" cookie crashes the service, or some special value might be the culprit. Since the server is running with c, it might be as simple as a string not being terminated properly, leading to the buffer overflow in my original log.

This cookie looks like it might have a null value, which also might be problematic, if uncaught.

[SciLor]

It seems to be happening if the cookie length is over a certain limit. Clearing the cookies may be a workaround.

[justM4D]

I've narrowed it down to a length issue. For example:

[ { "domain": "tc.fritz.box", "hostOnly": true, "httpOnly": false, "name": "foo", "path": "/", "sameSite": "unspecified", "secure": false, "session": true, "storeId": "0", "value": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "id": 2 } ]

causes it to crash

[SciLor]

I increased the server buffer to 32kB in the develop branch. This should fix it.

[justM4D]

I can try it out in case you have a dev branch docker image somewhere. Otherwise I'd have to get a dev environment set up first

[SciLor]

Yes, there is a develop tag on docker for that.

[justM4D]

The error (only allowing up to 512 Byte for the header) was found and upgraded to 32kb. That should be more than enough for any normal user.

I'd say this is fixed :)

[SciLor]

https://github.com/toniebox-reverse-engineering/teddycloud/commit/906c807f4fe3ade8a457d85732f8027ff594fa74