Sish server crashes when transferring big files

[manprint]

Hi AntonioMika, congratulations for the excellent sish tool. He saved me on several occasions. I am trying to forward a webdav server but when I try to transfer large files (download and upload) (~1GB size) Sish server crashes (deployed with docker and apache in front) without any information in the logs (also tried in debug mode). The webdav server works fine with large files without sish.

The command I use for the server is the following:

docker run -itd --name sish \
    -v $(pwd)/keys:/keys \
    -v $(pwd)/pubkeys:/pubkeys \
    --restart=always \
    -p 2222:2222 \
    -p 4443:4443 \
    -p 8765:80 \
    antoniomika/sish:2.7.0 \
    --ssh-address=:2222 \
    --http-address=:80 \
    --https=true \
    --authentication-keys-directory=/pubkeys \
    --private-keys-directory=/keys \
        --tcp-aliases \
        --bind-random-aliases=false \
    --bind-random-ports=false \
    --bind-random-subdomains=false \
    --authentication-password= \
    --domain=mysishserver.com \
    --idle-connection=true \
    --idle-connection-timeout=360s \
        --ping-client-timeout=360s \
    --cleanup-unauthed-timeout=360s \
        --cleanup-unbound \
    --cleanup-unbound-timeout=360s \
    --admin-console \
    --admin-console-token=my-secret-token \
    --verify-ssl=false

The apache configuration is as follows:

<VirtualHost *:80>

    ServerName mysishserver.com
    ServerAlias *.mysishserver.com

    #RewriteEngine On
    #RewriteCond %{HTTPS} !=on
    #RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

    # Websocket Support
    RewriteEngine on
    RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
    RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
    RewriteRule .* ws://172.30.0.72:8765%{REQUEST_URI} [P,QSA,L]

    ProxyPreserveHost On

    ProxyPass        / http://172.30.0.72:8765/
    ProxyPassReverse / http://172.30.0.72:8765/

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

<VirtualHost *:443>

    ServerName mysishserver.com
    ServerAlias *.mysishserver.com

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    SSLEngine on

    SSLCertificateFile      /home/ubuntu/docker_services/sish/ssl-keys/tls.crt
    SSLCertificateKeyFile   /home/ubuntu/docker_services/sish/ssl-keys/tls.key

    ProxyPreserveHost On
    RequestHeader set X-Forwarded-Proto "https"

    # Websocket Support
    RewriteEngine on
    RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
    RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
    RewriteRule .* ws://172.30.0.72:8765%{REQUEST_URI} [P,QSA,L]

    ProxyPass        / http://172.30.0.72:8765/
    ProxyPassReverse / http://172.30.0.72:8765/

    <FilesMatch "\.(cgi|shtml|phtml|php)$">
                    SSLOptions +StdEnvVars
    </FilesMatch>
    <Directory /usr/lib/cgi-bin>
                    SSLOptions +StdEnvVars
    </Directory>

</VirtualHost>

The connection is started with:

ssh -o 'PubkeyAcceptedKeyTypes +ssh-rsa' -p 2222 -R webdav:80:localhost:80 mysishserver.com

Do I make any errors in the configuration? Thank you

[antoniomika]

To me your configuration looks fine. Can you run sish with --debug=true and send the logs here?

[manprint]

This is the log. sish.log

Thanks

[antoniomika]

Hey @manprint,

Don't see any crashes there unfortunately. Does the sish server itself crash or just the ssh client? If the sish server itself crashes, mind giving me the output of docker inspect <sish container id>. Be cautious with this output just in case there are environment secrets you may have added to the container.

[manprint]

The server crash. The docker container restarts.

container-inspect.log

Thank you

PS: This is the webdav server: https://hub.docker.com/r/bytemark/webdav/

[antoniomika]

Hrm I don't see any issues here. How are you grabbing the longs? If you're doing console redirection, you might need to include stderr as well.

[manprint]

I got the sish server logs directly from docker logs -f sish without any redirection Tomorrow I can try to transfer files of different sizes. So it is possible to see if there is a size limit or something similar. Thank you very much

[manprint]

Hi Antoniomika, I have made some tests with different files of different sizes. These are the results:

-rw-rw-r-- 1 user user   50M ott 20 11:34 test50m.txt -> OK
-rw-rw-r-- 1 user user  100M ott 20 11:34 test100m.txt -> OK
-rw-rw-r-- 1 user user  200M ott 20 11:34 test200m.txt -> OK
-rw-rw-r-- 1 user user  300M ott 20 11:34 test300m.txt -> FAIL (sish crash)
-rw-rw-r-- 1 user user  400M ott 20 11:35 test400m.txt -> FAIL (sish crash)

I noticed that as the file size grows, the RAM memory allocated by Sish grows. With a 300MB file the container's ram rises to 2GB and the server crashes (the container is killed by the operating system for saturation of the RAM?)

dcb821935b14  sish    0.23%    2.136GiB / 3.786GiB    0.96%    7.11kB / 4.79kB     16.2MB / 0B         9

I think this is the issue. What do you think about it?

The VM on which I run the server is an AWS EC2 with 2 cores and 4 GB Ram. Thanks!

[antoniomika]

That's what I thought it could be, and which is why I wanted the inspect output as the exit code is sometimes included there if caught quick enough. Oxy (the lib we are using) might buffer output. In this case, would a SNI proxy endpoint or TCP endpoint suffice (or at least be used to test this issue)? From what I recall of how I implemented those, it should be streaming instead of buffer based.

[manprint]

OK, tonight I'll try a tcp tunnel instead of http forwarding. I will let you know. Thanks.

[manprint]

Hi, I have run tests with the TCP tunnel. With this method, sish works well with large files.

ssh -o 'PubkeyAcceptedKeyTypes +ssh-rsa' -p 2222 -R 81:localhost:80 apps.internaltest.tk

response:

Starting SSH Forwarding service for tcp:81. Forwarded connections can be accessed via the following methods:
TCP: apps.internaltest.tk:81

Below is the result of the tests:

-rw-rw-r-- 1 user user   50M ott 20 11:34 test50m.txt -> OK
-rw-rw-r-- 1 user user  100M ott 20 11:34 test100m.txt -> OK
-rw-rw-r-- 1 user user  200M ott 20 11:34 test200m.txt -> OK
-rw-rw-r-- 1 user user  300M ott 20 11:34 test300m.txt -> OK
-rw-rw-r-- 1 user user  500M ott 20 18:46 test500m.txt -> OK
-rw-rw-r-- 1 user user  500M ott 20 18:47 test700m.txt -> OK
-rw-rw-r-- 1 user user  900M ott 20 18:47 test900m.txt -> OK
-rw-rw-r-- 1 user user 1000M ott 20 18:27 test1000m.txt -> OK

The crash problem occurs when http and https forwarding are done. Probably because of the buffering issue.

Is there a way to use streaming instead of buffering in HTTP / HTTPS farwarding?

If you need to do other tests please ask me. (I am an Italian Devops Engineer). I'm glad to help you develop and test your app.

Thanks a lot. Fabio

[antoniomika]

Hey Fabio,

Thanks for checking, glad that works. I have a method for enabling streaming and I'm going to test that. I'll have a build you can test in about an hour!

Best,

[manprint]

Ok, thanks. When it is ready I try if everything works without problems. Good work and I wish you a good evening! Fabio

[antoniomika]

Hey @manprint!

Just pushed the PR and the image was created for testing, give it a go and make sure to set --service-console-max-content-length to something that works with your ram settings (or just 0 if you don't really use the service console for dumping request/response data).

antoniomika/sish:am-http-streaming-and-limit

[manprint]

Hi @antoniomika. Thank You Very Much. Tomorrow I will do the file transfer tests again with HTTP / HTTPS forwarding and let you know if everything works without issues I'll set the --service-console-max-content-length=0 flag. Best, Fabio

[manprint]

Hi Antonio Mika, I did a fly test with the image you posted. When I try to login to my webdav server there is this error in the sish server. The same error occurs when I try to upload or download via rclone webdav client The flag is setted to --service-console-max-content-length=0 I am attaching the log:

sish_streming_http_log.txt

Best, Fabio

[antoniomika]

@manprint thanks for the quick test! I found the issue and fixed it. Feel free to repull the image when you can.

[manprint]

Hi @AntonioMika. I have tested the image you pushed and everything seems to be working fine. Large file transfer via webdav works well. I have tested with 1GB and 2GB files (download and upload). The memory of the sish server is stable and does not grow as before. I send you the docker stats during the transfer.

0d8e9809b182   sish   4.67%   30.91MiB / 3.786GiB   0.80%   1.45GB / 1.38GB   0B / 0B  9

My colleagues right now are using Sish via both HTTP / HTTPS and TCP forwarding and everything looks ok. At this moment there are 3 connections open and everything works fine:

2022/10/21 - 10:17:00 | =======Start=========
2022/10/21 - 10:17:00 | ===Goroutines=====
2022/10/21 - 10:17:00 | 72
2022/10/21 - 10:17:00 | ===Listeners======
2022/10/21 - 10:17:00 | /tmp/91.51.32.153_52938_801169233557
2022/10/21 - 10:17:00 | /tmp/91.51.32.153_57476_801392789932
2022/10/21 - 10:17:00 | :2222
2022/10/21 - 10:17:00 | /tmp/91.51.32.153_37934_80703006580
2022/10/21 - 10:17:00 | ===Clients========
2022/10/21 - 10:17:00 | 91.51.32.153:37934 alpine [/tmp/91.51.32.153_37934_80703006580]
2022/10/21 - 10:17:00 | 91.51.32.153:57476 alpine [/tmp/91.51.32.153_57476_801392789932]
2022/10/21 - 10:17:00 | 91.51.32.153:52938 alpine [/tmp/91.51.32.153_52938_801169233557]
2022/10/21 - 10:17:00 | ===HTTP Listeners===
2022/10/21 - 10:17:00 | http://:@testpaypal.apps.internaltest.tk [91.51.32.153:37934]
2022/10/21 - 10:17:00 | http://:@my-sio-ws.apps.internaltest.tk [91.51.32.153:57476]
2022/10/21 - 10:17:00 | http://:@webdav.apps.internaltest.tk [91.51.32.153:52938]
2022/10/21 - 10:17:00 | ===TCP Aliases====
2022/10/21 - 10:17:00 | ===TCP Listeners====
2022/10/21 - 10:17:00 | ===Web Console Routes====
2022/10/21 - 10:17:00 | http://:@webdav.apps.internaltest.tk []
2022/10/21 - 10:17:00 | http://:@my-sio-ws.apps.internaltest.tk []
2022/10/21 - 10:17:00 | http://:@testpaypal.apps.internaltest.tk []
2022/10/21 - 10:17:00 | ===Web Console Tokens====
2022/10/21 - 10:17:00 | http://:@testpaypal.apps.internaltest.tk 5xBiOwOascJR1vR2KTdP
2022/10/21 - 10:17:00 | http://:@my-sio-ws.apps.internaltest.tk dekGFK1oZAykpWm3qlz9
2022/10/21 - 10:17:00 | http://:@webdav.apps.internaltest.tk uTFPQuzMNGYbUTecZhDD
2022/10/21 - 10:17:00 | ========End==========

I believe the issue is resolved and can be closed.

For now I will use the image you pushed (maybe don't delete it yet until you bring the fix to the main branch and do the main release) These days I will continue to use sish. If I find any other issue I will inform you.

Thank you for your help.

Best, Fabio

[antoniomika]

Thanks for checking! Just merged and a latest release will be tagged and available in a few minutes!

I don't currently clean up old branch images, as their size are so small (thanks scratch images!) and I like being able to go back in time to see whether or not things broke between changes. Feel free to continue to use it until you can take one of the latest ones :)

[manprint]

Ok, Thank You Very Much! :+1: :1st_place_medal: Fabio