search

rapiz1 / rathole

A lightweight and high-performance reverse proxy for NAT traversal, written in Rust. An alternative to frp and ngrok.
Apache License 2.0
8.8k stars 440 forks source link

Instant SIGKILL (a genuine stumper) #344

Closed rlue closed 3 months ago

rlue commented 3 months ago

Describe the bug

I'm facing a very weird problem. I'm admittedly not even certain that rathole is the cause, but I've never faced an issue like this in 10+ years of using Linux, and can't even begin to suppose what else might be at fault. This is long but I promise it is so puzzling as to be at least a little entertaining, and would really love some insight.

  1. The precompiled rathole binary from the releases page (v0.5.0) worked once, and since then always dies instantly with SIGKILL:

    $ unzip rathole-x86_64-unknown-linux-gnu.zip
$ ./rathole --genkey
Private Key:
vSB1qAcpJIcgTpcgqd62apd6spzPotDtC5/vK5i/4Fc=

Public Key:
jj6CAi6VfaBixdCEKhWuwgWMRxlJ7zgb6McMlRt/23U=

$ ./rathole --genkey
fish: Job 1, './rathole --genkey' terminated by signal SIGKILL (Forced quit)

$ ./rathole --version
fish: Job 1, './rathole --version' terminated by signal SIGKILL (Forced quit)

$ ./rathole
fish: Job 1, './rathole' terminated by signal SIGKILL (Forced quit)

  2. A version compiled using cargo install does not exhibit this problem:

    $ cargo install rathole
# ...
$ rathole
error: The following required arguments were not provided:
   <CONFIG|--genkey [<CURVE>]>

USAGE:
   rathole [OPTIONS] <CONFIG|--genkey [<CURVE>]>

For more information try --help
$ rathole --version
rathole
Build Timestamp:     2024-03-13T15:10:15.426813843Z
Build Version:       0.5.0
Commit SHA:          None
Commit Date:         None
Commit Branch:       None
cargo Target Triple: x86_64-unknown-linux-gnu
cargo Profile:       release
cargo Features:      base64,client,default,futures_core,futures_sink,hot_reload,noise,notify,server,snowstorm,tls,tokio_native_tls,tokio_tungstenite,tokio_util,websocket
$ rathole --genkey
Private Key:
TXyPj6WAL/ugBZeLxd6vVIiGHK+AMieZ3Q72+sNmTKU=

Public Key:
d4xIu16vRBBiOTMprNR/EEo6fj2sLBI33b8S5UpN1z0=

    At this point, no other executables on my system seemed to suffer from this issue.

  3. Now it gets weird: in the process of drafting the bug report, I tried to curl the latest release and started seeing the same instant SIGKILL issue:

    $ curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip
fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)

    This issue does not happen for other, similar URLs:

    $ curl -LO https://github.com/sharkdp/fd/releases/download/v9.0.0/fd-v9.0.0-x86_64-unknown-linux-musl.tar.gz
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1640k  100 1640k    0     0   770k      0  0:00:02  0:00:02 --:--:-- 3261k
$ curl -LO https://github.com/BurntSushi/ripgrep/releases/download/14.1.0/ripgrep-14.1.0-x86_64-unknown-linux-musl.tar.gz
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
 0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100 2468k  100 2468k    0     0   758k      0  0:00:03  0:00:03 --:--:-- 3371k

  4. Even weirder: If I take the failing curl command from Step 4 and start dropping characters from the end of the URL, eventually, we stop getting the SIGKILL:

    # ...
$ curl -LO https://github.com/rapiz1/rathole/releas
fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)
$ curl -LO https://github.com/rapiz1/rathole/relea
fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)
$ curl -LO https://github.com/rapiz1/rathole/rele
fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)
$ curl -LO https://github.com/rapiz1/rathole/rel
fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)
$ curl -LO https://github.com/rapiz1/rathole/re
fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)
$ curl -LO https://github.com/rapiz1/rathole/r
fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)
$ curl -LO https://github.com/rapiz1/rathole/
fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)
$ curl -LO https://github.com/rapiz1/rathole
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  371k    0  371k    0     0   312k      0 --:--:--  0:00:01 --:--:--  312k

  5. Weirder still: the same thing happens with wget, at the same threshold character:

    $ wget https://github.com/rapiz1/rathole/relea
fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit)
$ wget https://github.com/rapiz1/rathole/rele
--2024-03-13 10:11:27--  https://github.com/rapiz1/rathole/rele
fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit)
$ wget https://github.com/rapiz1/rathole/rel
fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit)
$ wget https://github.com/rapiz1/rathole/re
--2024-03-13 10:11:28--  https://github.com/rapiz1/rathole/re
fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit)
$ wget https://github.com/rapiz1/rathole/r
fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit)
$ wget https://github.com/rapiz1/rathole/
fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit)
$ wget https://github.com/rapiz1/rathole
--2024-03-13 10:11:33--  https://github.com/rapiz1/rathole
Resolving github.com (github.com)... 140.82.112.3
Connecting to github.com (github.com)|140.82.112.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘rathole.1’

rathole.1                                [   <=>                                                              ] 371.02K   806KB/s    in 0.5s

2024-03-13 10:11:33 (806 KB/s) - ‘rathole.1’ saved [379924]

  6. If I arbitrarily add different characters after this breakpoint, the instant SIGKILL also goes away, even for invalid URLs:

    $ curl -LO https://github.com/rapiz1/ratholes-r-us/
curl: Remote file name has no length!
curl: (23) Failed writing received data to disk/application
$ curl -LO https://github.com/rapiz1/ratholes-r-us/foo
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100     9  100     9    0     0     29      0 --:--:-- --:--:-- --:--:--    29

  7. Another level of weirdness: if I run with sudo, it fails even before it prompts me for my password:

    $ sudo curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip
fish: Job 1, 'sudo curl -LO https://github.co…' terminated by signal SIGKILL (Forced quit)
$ sudo ls
[sudo] password for rlue:
sudo: a password is required

  8. These issues are independent of the shell I'm running:

    $ bash
$ curl -LO https://github.com/rapiz1/rathole/
Killed
$ curl -LO https://github.com/rapiz1/rathole
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  366k    0  366k    0     0  66589      0 --:--:--  0:00:05 --:--:-- 82800

  9. The issue goes away immediately after restart, but then comes back ~90 seconds after the system comes online:

    # restart + immediate login
$ curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100 1248k    0 1248k    0     0   789k      0 --:--:--  0:00:02 --:--:-- 2213k
$ time while ./rathole --version >/dev/null 2>&1; sleep 1; end
fish: Job 1, './rathole --version' terminated by signal SIGKILL (Forced quit)

real    1m22.365s
user    0m5.817s
sys     0m0.659s

    Both curl and ./rathole commands work within this 90-second window; rathole does not have to be run first for curl to break:

    # restart + wait ~90s before login
$ curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip
fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)

I was on my employer's VPN for the majority of this investgation, but I have disabled it and am seeing the exact same behavior.

To Reproduce

N/A, I have tried to reproduce this issue on other machines to no avail.

Configuration

N/A

Logs

N/A

Environment

$ neofetch
            .-/+oossssoo+/-.               rlue@rlue-lycra
        `:+ssssssssssssssssss+:`           ---------------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 22.04.4 LTS x86_6
    .ossssssssssssssssssdMMMNysssso.       Host: Precision 5570
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 6.1.0-1035-oem
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 1 hour, 30 mins
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 3671 (dpkg), 6 (fl
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.1.16
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Resolution: 1920x1200
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   WM: i3
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Theme: Raleigh [GTK2], Arc-D
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Icons: Papirus [GTK2], Papir
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Terminal: tmux
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/    CPU: 12th Gen Intel i9-12900
  +sssssssssdmydMMMMMMMMddddyssssssss+     GPU: NVIDIA RTX A2000 8GB La
   /ssssssssssshdmNNNNmyNMMMMhssssss/      GPU: Intel Alder Lake-P
    .ossssssssssssssssssdMMMNysssso.       Memory: 6113MiB / 63970MiB
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.

$ rustc --version
rustc 1.76.0 (07dca489a 2024-02-04)
fernvenue commented 3 months ago

What happened when you use bash instead of fish?

rlue commented 3 months ago

No difference; see point 8 above.

rapiz1 commented 3 months ago

I suspect it's related to upx. Could you try upx the binary produced by cargo install? The upx version used by rathole can be found under.github

On Thu, Mar 14, 2024, 02:18 Ryan Lue @.***> wrote:

Describe the bug

I'm facing a very weird problem. I'm admittedly not even certain that rathole is the cause, but I've never faced an issue like this in 10+ years of using Linux, and can't even begin to suppose what else might be at fault. This is long but I promise it is puzzling beyond belief, and would really love some insight.

1.

The precompiled rathole binary from the releases page (v0.5.0) worked once, and since then always dies instantly with SIGKILL:

$ unzip rathole-x86_64-unknown-linux-gnu.zip $ ./rathole --genkey Private Key: vSB1qAcpJIcgTpcgqd62apd6spzPotDtC5/vK5i/4Fc=

Public Key: jj6CAi6VfaBixdCEKhWuwgWMRxlJ7zgb6McMlRt/23U=

$ ./rathole --genkey fish: Job 1, './rathole --genkey' terminated by signal SIGKILL (Forced quit)

$ ./rathole --version fish: Job 1, './rathole --version' terminated by signal SIGKILL (Forced quit)

$ ./rathole fish: Job 1, './rathole' terminated by signal SIGKILL (Forced quit)

2.

A version compiled using cargo install does not exhibit this problem:

$ cargo install rathole# ... $ rathole error: The following required arguments were not provided: <CONFIG|--genkey []>

USAGE: rathole [OPTIONS] <CONFIG|--genkey []>

For more information try --help $ rathole --version rathole Build Timestamp: 2024-03-13T15:10:15.426813843Z Build Version: 0.5.0 Commit SHA: None Commit Date: None Commit Branch: None cargo Target Triple: x86_64-unknown-linux-gnu cargo Profile: release cargo Features: base64,client,default,futures_core,futures_sink,hot_reload,noise,notify,server,snowstorm,tls,tokio_native_tls,tokio_tungstenite,tokio_util,websocket $ rathole --genkey Private Key: TXyPj6WAL/ugBZeLxd6vVIiGHK+AMieZ3Q72+sNmTKU=

Public Key: d4xIu16vRBBiOTMprNR/EEo6fj2sLBI33b8S5UpN1z0=

3.

At this point, no other executables on my system seemed to suffer from this issue. 4.

Now it gets weird: in the process of drafting the bug report, I tried to curl the latest release and started seeing the same instant SIGKILL issue:

$ curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)

5.

This issue does not happen for other URLs:

$ curl -LO https://github.com/sharkdp/fd/releases/download/v9.0.0/fd-v9.0.0-x86_64-unknown-linux-musl.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 1640k 100 1640k 0 0 770k 0 0:00:02 0:00:02 --:--:-- 3261k $ curl -LO https://github.com/BurntSushi/ripgrep/releases/download/14.1.0/ripgrep-14.1.0-x86_64-unknown-linux-musl.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 100 2468k 100 2468k 0 0 758k 0 0:00:03 0:00:03 --:--:-- 3371k

6.

Even weirder: If I take the failing curl command from Step 4 and remove characters from the end of the URL one at a time, eventually, it works:

...

$ curl -LO https://github.com/rapiz1/rathole/releas fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/relea fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/rele fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/rel fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/re fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/r fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/ fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 371k 0 371k 0 0 312k 0 --:--:-- 0:00:01 --:--:-- 312k

7.

Weirder still: the same thing happens with wget, at the same threshold character:

$ wget https://github.com/rapiz1/rathole/relea fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole/rele --2024-03-13 https://github.com/rapiz1/rathole/rele--2024-03-13 10:11:27-- https://github.com/rapiz1/rathole/rele fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole/rel fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole/re --2024-03-13 https://github.com/rapiz1/rathole/re--2024-03-13 10:11:28-- https://github.com/rapiz1/rathole/re fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole/r fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole/ fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole --2024-03-13 https://github.com/rapiz1/rathole--2024-03-13 10:11:33-- https://github.com/rapiz1/rathole Resolving github.com (github.com)... 140.82.112.3 Connecting to github.com (github.com)|140.82.112.3|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: ‘rathole.1’

rathole.1 [ <=> ] 371.02K 806KB/s in 0.5s

2024-03-13 10:11:33 (806 KB/s) - ‘rathole.1’ saved [379924]

8.

If I arbitrarily add different characters after this breakpoint, the instant SIGKILL goes away:

$ curl -LO https://github.com/rapiz1/ratholes-r-us/ curl: Remote file name has no length! curl: (23) Failed writing received data to disk/application $ curl -LO https://github.com/rapiz1/ratholes-r-us/foo % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 9 100 9 0 0 29 0 --:--:-- --:--:-- --:--:-- 29

9.

Another level of weirdness: if I run with sudo, it fails even before it prompts me for my password:

$ sudo curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip fish: Job 1, 'sudo curl -LO https://github.co…' terminated by signal SIGKILL (Forced quit) $ sudo ls [sudo] password for rlue: sudo: a password is required

10.

These issues are independent of the shell I'm running:

$ bash $ curl -LO https://github.com/rapiz1/rathole/ Killed $ curl -LO https://github.com/rapiz1/rathole % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 366k 0 366k 0 0 66589 0 --:--:-- 0:00:05 --:--:-- 82800 $ sudo curl -LO https://github.com/rapiz1/rathole/ Killed

11.

The issue goes away immediately after restart, but then comes back ~90 seconds after the system comes online:

restart + immediate login

$ curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1248k 0 1248k 0 0 789k 0 --:--:-- 0:00:02 --:--:-- 2213k $ time while ./rathole --version >/dev/null 2>&1; sleep 1; end fish: Job 1, './rathole --version' terminated by signal SIGKILL (Forced quit)

real 1m22.365s user 0m5.817s sys 0m0.659s

Both curl and ./rathole commands work within this 90-second window; rathole does not have to be run first for curl to break:

restart + wait ~90s before login

$ curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)

12.

I was on my employer's VPN for the majority of this investgation, but I have disabled it and am seeing the exact same behavior.

To Reproduce

N/A, I have tried to reproduce this issue on other machines to no avail. Configuration

N/A Logs

N/A Environment

$ neofetch .-/+oossssoo+/-. @.*** :+ssssssssssssssssss+: --------------- -+ssssssssssssssssssyyssss+- OS: Ubuntu 22.04.4 LTS x86_6 .ossssssssssssssssssdMMMNysssso. Host: Precision 5570 /ssssssssssshdmmNNmmyNMMMMhssssss/ Kernel: 6.1.0-1035-oem +ssssssssshmydMMMMMMMNddddyssssssss+ Uptime: 1 hour, 30 mins /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/ Packages: 3671 (dpkg), 6 (fl .ssssssssdMMMNhsssssssssshNMMMdssssssss. Shell: bash 5.1.16 +sssshhhyNMMNyssssssssssssyNMMMysssssss+ Resolution: 1920x1200 ossyNMMMNyMMhsssssssssssssshmmmhssssssso WM: i3 ossyNMMMNyMMhsssssssssssssshmmmhssssssso Theme: Raleigh [GTK2], Arc-D +sssshhhyNMMNyssssssssssssyNMMMysssssss+ Icons: Papirus [GTK2], Papir .ssssssssdMMMNhsssssssssshNMMMdssssssss. Terminal: tmux /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/ CPU: 12th Gen Intel i9-12900 +sssssssssdmydMMMMMMMMddddyssssssss+ GPU: NVIDIA RTX A2000 8GB La /ssssssssssshdmNNNNmyNMMMMhssssss/ GPU: Intel Alder Lake-P .ossssssssssssssssssdMMMNysssso. Memory: 6113MiB / 63970MiB -+sssssssssssssssssyyyssss+- :+ssssssssssssssssss+: .-/+oossssoo+/-.

$ rustc --version rustc 1.76.0 (07dca489a 2024-02-04)

— Reply to this email directly, view it on GitHub https://github.com/rapiz1/rathole/issues/344, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEZMOZJ7UCZ3BOEXRATMZXLYYCKADAVCNFSM6AAAAABEUVXT3SVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE4DINRSGI2DOMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

rapiz1 commented 3 months ago

Also, it's possible that some anti virus software is running and thinking rathole release is a virus

On Fri, Mar 15, 2024, 20:10 Yujia Qiao @.***> wrote:

I suspect it's related to upx. Could you try upx the binary produced by cargo install? The upx version used by rathole can be found under.github

On Thu, Mar 14, 2024, 02:18 Ryan Lue @.***> wrote:

Describe the bug

I'm facing a very weird problem. I'm admittedly not even certain that rathole is the cause, but I've never faced an issue like this in 10+ years of using Linux, and can't even begin to suppose what else might be at fault. This is long but I promise it is puzzling beyond belief, and would really love some insight.

1.

The precompiled rathole binary from the releases page (v0.5.0) worked once, and since then always dies instantly with SIGKILL:

$ unzip rathole-x86_64-unknown-linux-gnu.zip $ ./rathole --genkey Private Key: vSB1qAcpJIcgTpcgqd62apd6spzPotDtC5/vK5i/4Fc=

Public Key: jj6CAi6VfaBixdCEKhWuwgWMRxlJ7zgb6McMlRt/23U=

$ ./rathole --genkey fish: Job 1, './rathole --genkey' terminated by signal SIGKILL (Forced quit)

$ ./rathole --version fish: Job 1, './rathole --version' terminated by signal SIGKILL (Forced quit)

$ ./rathole fish: Job 1, './rathole' terminated by signal SIGKILL (Forced quit)

2.

A version compiled using cargo install does not exhibit this problem:

$ cargo install rathole# ... $ rathole error: The following required arguments were not provided: <CONFIG|--genkey []>

USAGE: rathole [OPTIONS] <CONFIG|--genkey []>

For more information try --help $ rathole --version rathole Build Timestamp: 2024-03-13T15:10:15.426813843Z Build Version: 0.5.0 Commit SHA: None Commit Date: None Commit Branch: None cargo Target Triple: x86_64-unknown-linux-gnu cargo Profile: release cargo Features: base64,client,default,futures_core,futures_sink,hot_reload,noise,notify,server,snowstorm,tls,tokio_native_tls,tokio_tungstenite,tokio_util,websocket $ rathole --genkey Private Key: TXyPj6WAL/ugBZeLxd6vVIiGHK+AMieZ3Q72+sNmTKU=

Public Key: d4xIu16vRBBiOTMprNR/EEo6fj2sLBI33b8S5UpN1z0=

3.

At this point, no other executables on my system seemed to suffer from this issue. 4.

Now it gets weird: in the process of drafting the bug report, I tried to curl the latest release and started seeing the same instant SIGKILL issue:

$ curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)

5.

This issue does not happen for other URLs:

$ curl -LO https://github.com/sharkdp/fd/releases/download/v9.0.0/fd-v9.0.0-x86_64-unknown-linux-musl.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 1640k 100 1640k 0 0 770k 0 0:00:02 0:00:02 --:--:-- 3261k $ curl -LO https://github.com/BurntSushi/ripgrep/releases/download/14.1.0/ripgrep-14.1.0-x86_64-unknown-linux-musl.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 100 2468k 100 2468k 0 0 758k 0 0:00:03 0:00:03 --:--:-- 3371k

6.

Even weirder: If I take the failing curl command from Step 4 and remove characters from the end of the URL one at a time, eventually, it works:

...

$ curl -LO https://github.com/rapiz1/rathole/releas fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/relea fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/rele fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/rel fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/re fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/r fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole/ fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit) $ curl -LO https://github.com/rapiz1/rathole % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 371k 0 371k 0 0 312k 0 --:--:-- 0:00:01 --:--:-- 312k

7.

Weirder still: the same thing happens with wget, at the same threshold character:

$ wget https://github.com/rapiz1/rathole/relea fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole/rele --2024-03-13 https://github.com/rapiz1/rathole/rele--2024-03-13 10:11:27-- https://github.com/rapiz1/rathole/rele fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole/rel fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole/re --2024-03-13 https://github.com/rapiz1/rathole/re--2024-03-13 10:11:28-- https://github.com/rapiz1/rathole/re fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole/r fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole/ fish: Job 1, 'wget https://github.com/rapiz1/…' terminated by signal SIGKILL (Forced quit) $ wget https://github.com/rapiz1/rathole --2024-03-13 https://github.com/rapiz1/rathole--2024-03-13 10:11:33-- https://github.com/rapiz1/rathole Resolving github.com (github.com)... 140.82.112.3 Connecting to github.com (github.com)|140.82.112.3|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: ‘rathole.1’

rathole.1 [ <=> ] 371.02K 806KB/s in 0.5s

2024-03-13 10:11:33 (806 KB/s) - ‘rathole.1’ saved [379924]

8.

If I arbitrarily add different characters after this breakpoint, the instant SIGKILL goes away:

$ curl -LO https://github.com/rapiz1/ratholes-r-us/ curl: Remote file name has no length! curl: (23) Failed writing received data to disk/application $ curl -LO https://github.com/rapiz1/ratholes-r-us/foo % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 9 100 9 0 0 29 0 --:--:-- --:--:-- --:--:-- 29

9.

Another level of weirdness: if I run with sudo, it fails even before it prompts me for my password:

$ sudo curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip fish: Job 1, 'sudo curl -LO https://github.co…' terminated by signal SIGKILL (Forced quit) $ sudo ls [sudo] password for rlue: sudo: a password is required

10.

These issues are independent of the shell I'm running:

$ bash $ curl -LO https://github.com/rapiz1/rathole/ Killed $ curl -LO https://github.com/rapiz1/rathole % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 366k 0 366k 0 0 66589 0 --:--:-- 0:00:05 --:--:-- 82800 $ sudo curl -LO https://github.com/rapiz1/rathole/ Killed

11.

The issue goes away immediately after restart, but then comes back ~90 seconds after the system comes online:

restart + immediate login

$ curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1248k 0 1248k 0 0 789k 0 --:--:-- 0:00:02 --:--:-- 2213k $ time while ./rathole --version >/dev/null 2>&1; sleep 1; end fish: Job 1, './rathole --version' terminated by signal SIGKILL (Forced quit)

real 1m22.365s user 0m5.817s sys 0m0.659s

Both curl and ./rathole commands work within this 90-second window; rathole does not have to be run first for curl to break:

restart + wait ~90s before login

$ curl -LO https://github.com/rapiz1/rathole/releases/download/v0.5.0/rathole-x86_64-unknown-linux-gnu.zip fish: Job 1, 'curl -LO https://github.com/rap…' terminated by signal SIGKILL (Forced quit)

12.

I was on my employer's VPN for the majority of this investgation, but I have disabled it and am seeing the exact same behavior.

To Reproduce

N/A, I have tried to reproduce this issue on other machines to no avail. Configuration

N/A Logs

N/A Environment

$ neofetch .-/+oossssoo+/-. @.*** :+ssssssssssssssssss+: --------------- -+ssssssssssssssssssyyssss+- OS: Ubuntu 22.04.4 LTS x86_6 .ossssssssssssssssssdMMMNysssso. Host: Precision 5570 /ssssssssssshdmmNNmmyNMMMMhssssss/ Kernel: 6.1.0-1035-oem +ssssssssshmydMMMMMMMNddddyssssssss+ Uptime: 1 hour, 30 mins /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/ Packages: 3671 (dpkg), 6 (fl .ssssssssdMMMNhsssssssssshNMMMdssssssss. Shell: bash 5.1.16 +sssshhhyNMMNyssssssssssssyNMMMysssssss+ Resolution: 1920x1200 ossyNMMMNyMMhsssssssssssssshmmmhssssssso WM: i3 ossyNMMMNyMMhsssssssssssssshmmmhssssssso Theme: Raleigh [GTK2], Arc-D +sssshhhyNMMNyssssssssssssyNMMMysssssss+ Icons: Papirus [GTK2], Papir .ssssssssdMMMNhsssssssssshNMMMdssssssss. Terminal: tmux /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/ CPU: 12th Gen Intel i9-12900 +sssssssssdmydMMMMMMMMddddyssssssss+ GPU: NVIDIA RTX A2000 8GB La /ssssssssssshdmNNNNmyNMMMMhssssss/ GPU: Intel Alder Lake-P .ossssssssssssssssssdMMMNysssso. Memory: 6113MiB / 63970MiB -+sssssssssssssssssyyyssss+- :+ssssssssssssssssss+: .-/+oossssoo+/-.

$ rustc --version rustc 1.76.0 (07dca489a 2024-02-04)

— Reply to this email directly, view it on GitHub https://github.com/rapiz1/rathole/issues/344, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEZMOZJ7UCZ3BOEXRATMZXLYYCKADAVCNFSM6AAAAABEUVXT3SVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE4DINRSGI2DOMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

rlue commented 3 months ago

That last one is plausible. This is a work laptop, and I did get a slack message from IT asking about me running rathole that same morning. Still weird that 1) curl is broken, 2) rathole via cargo install works, and 3) they aren't blocked for the first 90 seconds after startup.

I suspect it's related to upx. Could you try upx the binary produced by cargo install? The upx version used by rathole can be found under.github

How? Sorry, I am not the smartest.

$ cargo install upx
    Updating crates.io index
error: could not find `upx` in registry `crates-io` with version `*`
$ fd -uuu upx ~/.cargo
$ cargo install --force rathole 2>&1 | grep upx

all turned up nothing. Not sure where else to look. Running the upx binary from their github releases page looks normal:

$ curl -L https://github.com/upx/upx/releases/download/v4.2.2/upx-4.2.2-amd64_linux.tar.xz | tar xJf -
$ upx-4.2.2-amd64_linux/upx --version
upx 4.2.2
NRV data compression library 0.84
UCL data compression library 1.03
zlib data compression library 1.3.0.1-motley
LZMA SDK version 4.43
Copyright (C) 1996-2024 Markus Franz Xaver Johannes Oberhumer
Copyright (C) 1996-2024 Laszlo Molnar
Copyright (C) 2000-2024 John F. Reiser
Copyright (C) 2002-2024 Jens Medoch
Copyright (C) 1995-2023 Jean-loup Gailly and Mark Adler
Copyright (C) 1999-2006 Igor Pavlov
UPX comes with ABSOLUTELY NO WARRANTY; for details type 'upx -L'.
rapiz1 commented 3 months ago

The released binary is compressed by upx. Self-decompressing program is often considered a sign of malware.

On Sat, Mar 16, 2024, 00:08 Ryan Lue @.***> wrote:

That last one is plausible. This is a work laptop, and I did get a slack message from IT asking about me running rathole that same morning. Still weird that 1) curl is broken, 2) rathole via cargo install works, and 3) they aren't blocked for the first 90 seconds after startup.

I suspect it's related to upx. Could you try upx the binary produced by cargo install? The upx version used by rathole can be found under.github

How? Sorry, I am not the smartest.

$ cargo install upx Updating crates.io index error: could not find upx in registry crates-io with version * $ fd -uuu upx ~/.cargo $ cargo install --force rathole 2>&1 | grep upx

all turned up nothing. Not sure where else to look. Running the upx binary from their github releases page looks normal:

$ curl -L https://github.com/upx/upx/releases/download/v4.2.2/upx-4.2.2-amd64_linux.tar.xz | tar xJf - $ upx-4.2.2-amd64_linux/upx --version upx 4.2.2 NRV data compression library 0.84 UCL data compression library 1.03 zlib data compression library 1.3.0.1-motley LZMA SDK version 4.43 Copyright (C) 1996-2024 Markus Franz Xaver Johannes Oberhumer Copyright (C) 1996-2024 Laszlo Molnar Copyright (C) 2000-2024 John F. Reiser Copyright (C) 2002-2024 Jens Medoch Copyright (C) 1995-2023 Jean-loup Gailly and Mark Adler Copyright (C) 1999-2006 Igor Pavlov UPX comes with ABSOLUTELY NO WARRANTY; for details type 'upx -L'.

— Reply to this email directly, view it on GitHub https://github.com/rapiz1/rathole/issues/344#issuecomment-1999985207, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEZMOZL56SQYF3A6DI3LIQ3YYMMI3AVCNFSM6AAAAABEUVXT3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJZHE4DKMRQG4 . You are receiving this because you commented.Message ID: @.***>

rlue commented 3 months ago

Wow you are 100% correct. Compressing the binary built with cargo breaks it.

$ curl -L https://github.com/upx/upx/releases/download/v4.2.2/upx-4.2.2-amd64_linux.tar.xz | tar xJf -
$ cd upx-4.2.2-amd64_linux
$ ./upx $HOME/.cargo/bin/rathole
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2024
UPX 4.2.2       Markus Oberhumer, Laszlo Molnar & John Reiser    Jan 3rd 2024

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   4597664 ->   1535008   33.39%   linux/amd64   rathole

Packed 1 file.
$ rathole
fish: Job 1, 'rathole' terminated by signal SIGKILL (Forced quit)

Case closed, thanks for all the help. :pray: