using nginx reverse proxy with frp and type = tcp

[zaalgol]

Bug Description

In our application, we create a frp tunnel from the local-server to the public server, so the public server can send data to the local-server. In the config we defined the type to be http, and defined a domain. The public server uses another proxy - nginx reverse proxy to route client requests to the local server address.. It works great.

We have a customer that wants to add his private proxy, that all data from public server to local-server should pass through it. Because frp supports http_proxy only with type=tcp, as a beginning I changed config to type=tcp, but because the type=tpc doesn’t support domains, I get a error in the public server frp: [http.go:92] do http proxy request error: no such domain: SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN.RAW SUB DOMAIN

frpc Version

0.35.1

frps Version

0.35.1

System Architecture

linux VM on a cloud

Configurations

frpc.ini:

[common]
server_addr = <PUBLIC SERVER ADDRESS>
server_port = 5223

#LOGGING
log_file = /var/log/supervisor/frpc.log
log_level = debug
log_max_days = 3
disable_log_color = false

#AUTHEXTRAS
authenticate_heartbeats = false
authenticate_new_work_conns = false
token = TOKEN

#set admin address for control frpc's action by http api such as reload
admin_addr = 0.0.0.0
admin_port = 7400

#connections will be established in advance, default value is zero
pool_count = 2

#if tcp stream multiplexing is used, default is true, it must be same with frps
tcp_mux = true

#decide if exit program when first login failed, otherwise continuous relogin to frps
login_fail_exit = false

#communication protocol used to connect to server
#now it supports tcp, kcp and websocket, default is tcp
protocol = tcp

#TLS
tls_enable = true
tls_cert_file = /certs/client.crt
tls_key_file = /certs/client.key

[<SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN>] 
type = http ### error when changed to tcp
remote_port = 55999 ### only for type=tcp
local_ip = gateway-service
local_port = 8081
#remote_port = 55998 ### for type = tcp
use_encryption = true
use_compression = true
subdomain = <SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN>

frps.ini:

[common]
bind_addr = 0.0.0.0
bind_port = 5223
vhost_http_port = 8080

dashboard_addr = 0.0.0.0
dashboard_port = 7400
dashboard_user =
dashboard_pwd =
enable_prometheus = false

#console or real logFile path like ./frps.log
log_file = /var/log/supervisor/frps.log
log_level = debug
log_max_days = 3
disable_log_color = false
detailed_errors_to_client = true

#auth token

token = TOKEN
tls_only = true
tls_cert_file = /certs/server.pem
tls_key_file = /certs/server.pem
tls_trusted_ca_file = /certs/ca.crt

#pool_count in each proxy will change to max_pool_count if they exceed the maximum value
max_pool_count = 10000
#max ports can be used for each client, default value is 0 means no limit
max_ports_per_client = 0

#if subdomain_host is not empty, you can set subdomain when type is http or https in frpc's configure file
#when subdomain is test, the host used by routing is test.frps.com
subdomain_host = <RAW SUB DOMAIN>

#if tcp stream multiplexing is used, default is true
tcp_mux = true

#custom 404 page for HTTP requests
custom_404_page = /ops/404.htmlbash-5

Nginx config:

location ~* ^/v1/api/<LOCAL SERVER ID>/(.*) {
    proxy_pass http://gtunnel-server.$kn.svc.cluster.local:8080/$1$is_args$args;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host <SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN.RAW SUB DOMAIN>;
}

Logs

[http.go:92] do http proxy request error: no such domain: <SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN.RAW SUB DOMAIN>

Steps to reproduce

3. ...

Affected area

• [ ] Docs
• [ ] Installation
• [ ] Performance and Scalability
• [X] Security
• [X] User Experience
• [ ] Test and Release
• [ ] Developer Infrastructure
• [ ] Client Plugin
• [ ] Server Plugin
• [ ] Extensions
• [X] Others

[Becods]

location ~* ^/v1/api/LOCAL SERVER ID/(.*) {
  proxy_pass http://127.0.0.1:8080/$1$is_args$args;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "Upgrade";
  proxy_set_header Host SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN.RAW SUB DOMAIN;
}

OR

location ~* ^/v1/api/LOCAL SERVER ID/(.*) {
  proxy_pass http://127.0.0.1:8081/$1$is_args$args;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "Upgrade";
  proxy_set_header Host SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN.RAW SUB DOMAIN;

}

[SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN]
type = tcp
remote_port = 8081
local_ip = gateway-service
local_port = 8081
use_encryption = true
use_compression = true
subdomain = SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN

[zaalgol]

Hi @Becods, Thanks for your response!

I tried the solution you suggested, and changed the 2 configurations.

The local server frp side runs OK. Logs:

2023/02/10 11:42:21 [I] [tcp.go:63] [4b6862736fea4562] [<SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN>] tcp proxy listen port [8081]
2023/02/10 11:42:21 [I] [control.go:446] [4b6862736fea4562] new proxy [<SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN>] success

Also the public server frp runs well. Logs:

2023/02/10 12:34:43 [I] [tcp.go:63] [5174c5f44d91f23c] [<SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN>] tcp proxy listen port [8081]
2023/02/10 12:34:43 [I] [control.go:446] [5174c5f44d91f23c] new proxy [<SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN>] success
2023/02/10 12:34:43 [D] [control.go:219] [5174c5f44d91f23c] new work connection registered

The problem is in the nginx reverse proxy.

If I set the port in proxy_pass to 8080, I get the following error, when sending a request from public server to local server:

127.0.0.1 - - [10/Feb/2023:11:53:29 +0000] "GET <REQUEST> HTTP/1.1" 403 162 "-" "python-requests/2.25.1"
<IP> - - [10/Feb/2023:11:53:29 +0000] "GET /v1/api/<LOCAL SERVER ID>/<REQUEST> HTTP/1.1" 403 162 "-" "python-requests/2.25.1"

If I set the port in proxy_pass to 8081, I get the following error, when sending a request from public server to local server: <IP> - - [10/Feb/2023:12:53:36 +0000] "GET /v1/api/<LOCAL SERVER ID><REQUEST> HTTP/1.1" 502 166 "-" "python-requests/2.25.1"

It is worth noting that in the old logic in which I used http, the log in nginx proxy was: <IP> - - [13/Feb/2023:09:09:38 +0000] "GET /v1/api/<SUBDOMAIN GENERATED FROM SECRET AND RAW SUBDOMAIN>/<REQUEST>HTTP/1.1" 200 51 "-" " python-requests/2.25.1" As you can see, it is directed at port 51, unlike the one I changed to tcp, which is directed at other ports.

Maybe need to make a change also in frps.ini? Or maybe need to do a different change in frpc.ini?