Trying to setup nginx ingress with FRPS server and connect from FRPC client outside of k8s cluster

[Emon46]

Bug Description

First of all thanks for building this cool tool! I am trying to set up the FRPS server with the nginx ingress controller inside eks cluster. I have deployed an ingress controller which created an NLB for me. and I can expose services in port 80 and 443. Now I want to set up my FRPS server on top of it. I have deployed the FRPS server and created a service and created an ingress for FRPS server port. here is my ingress and service yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/instance: tunnel-server
    app.kubernetes.io/name: tunnel-server
  name: tunnel-server
  namespace: tunnel-proxy
spec:
  ports:
  - name: bind
    port: 7000
    protocol: TCP
    targetPort: bind
  selector:
    app.kubernetes.io/instance: tunnel-server
    app.kubernetes.io/name: tunnel-server
  type: ClusterIP

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app.kubernetes.io/instance: tunnel-server
    app.kubernetes.io/name: tunnel-server
  name: tunnel-server
  namespace: tunnel-proxy
spec:
  ingressClassName: nginx
  rules:
  - host: p01-proxy.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: tunnel-server
            port:
              number: 7000
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - p01-proxy.mydomain.com
    secretName: tunnel-tls-cert

frpc Version

0.48.0

frps Version

0.48.0

System Architecture

linux/amd64

Configurations

frps.ini

[common]
bind_port = 7000
log_level = trace

frpc.ini

[common]
server_addr = p01-proxy.mydomain.com
server_port = 443
tls_enable = true
log_level = trace
tls_server_name = p01-proxy.mydomain.com
disable_custom_tls_first_byte = true
type = tcp

Logs

I didn't get any error message in frps server. In client i can only see it's tried to login but failed. client logs:

2023/05/16 00:28:47 [T] [root.go:212] start frpc service for config file [frpc_test.ini]
2023/05/16 00:28:48 [W] [service.go:134] login to server failed: EOF
2023/05/16 00:28:48 [T] [root.go:232] frpc service for config file [frpc_test.ini] stopped
EOF

Is the configuration correct? If not is there any work around we can make it to support with nginx controller in EKS cluster

Steps to reproduce

3. ...

Affected area

• [ ] Docs
• [ ] Installation
• [ ] Performance and Scalability
• [ ] Security
• [ ] User Experience
• [ ] Test and Release
• [ ] Developer Infrastructure
• [ ] Client Plugin
• [ ] Server Plugin
• [ ] Extensions
• [ ] Others

[fatedier]

Ingress is only for the HTTP protocol. You should expose port 7000 as a TCP port.

[aleksandr-orca]

@Emon46 by any chance did you resolve this issue?

[svyatoslavd-orca]

Hi @fatedier. Thanks for your awesome work and efforts! Could you please elaborate on 2 things:

1. What is the correct way to expose FRPS in K8S cluster using Nginx Ingress? Is there an example of a working configuration? I guess it's not the first time people are asking for this. Probably this deserves to be added to documentation.
2. What is the best practice for secure connections between FRPC and FRPS when using K8S Ingress? Is it possible to use TLS connection while exposing FRPS with Nginx Ingress?

[andrey-bondar]

@Emon46 Hi! I use ktunnel for this purpose in K8S. Please take a look

[julianbalog]

I got the frps working behind the nginx ingress controller in Kubernetes. To @fatedier's point, I could only make it work by exposing frps as a TCP service endpoint. (See the related concept here.) Here are the relevant configuration stubs to set up an frps/frpc tunnel secured with mTLS (client certificate authentication) in Kubernetes.

Disclaimer Some of the configuration below has been re-edited for brevity and it's possible I may have missed something. But, at a high level, it reflects closely the environment that works for me.

frps.ini:

[common]
bind_port = 7443
vhost_https_port = 7443
tls_only = false
tls_enable = true
tls_cert_file = /etc/frp/certs/server.crt
tls_key_file = /etc/frp/certs/server.key
tls_trusted_ca_file = /etc/frp/certs/ca.crt
log_level = trace

Create the kube secret required to mount the certificates into the frps pod. (You'll need to generate these certs first, following the procedure described here.)

kubectl create secret generic frps_certs \
    --from-file=ca.crt=certs/ca.crt \
    --from-file=server.crt=certs/server.crt \
    --from-file=server.key=certs/server.key

frps.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: frps
  name: frps
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: frps
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: frps
    spec:
      containers:
        - image: snowdreamtech/frps
          name: frps
          ports:
            - containerPort: 7443
          volumeMounts:
            - mountPath: /etc/frp/frps.ini
              name: frps-ini
            - mountPath: /etc/frp/certs
              name: frps-certs
      restartPolicy: Always
      volumes:
        - name: frps-ini
          hostPath:
            path: /etc/frp/frps.ini
            type: FileOrCreate
        - name: frps-certs
          secret:
            secretName: frps-certs
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: frps
  name: frps
spec:
  ports:
    - name: "https"
      port: 7443
      targetPort: 7443
  selector:
    app.kubernetes.io/name: frps
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: ingress-nginx
  name: ingress-nginx-controller
spec:
  ports:
  - name: frps
    port: 7443
    protocol: TCP
    targetPort: 7443
  selector:
    app.kubernetes.io/name: frps
  type: LoadBalancer
---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: frps
  name: frps
data:
  7443: "default/frps:7443"

Note the ConfigMap configuration above. We need to add it to the ingress-nginx-controller's --tcp-services-configmap argument.

kubectl edit deployment ingress-nginx-controller

Add the --tcp-services-configmap argument, as follows:

spec:
  containers:
  - args:
    - /nginx-ingress-controller
    - --tcp-services-configmap=default/frps

frpc.ini

[common]
server_addr = example.com
server_port = 7443
tls_enable = true
tls_cert_file = /etc/frp/certs/client.crt
tls_key_file = /etc/frp/certs/client.key
tls_trusted_ca_file = /etc/frp/certs/ca.crt
tls_server_name = example.com
log_level = trace

[ws-https]
type = https
custom_domains = ws.example.com
local_ip = ws-https
local_port = 5443

I hope this helps.

[github-actions[bot]]

Issues go stale after 30d of inactivity. Stale issues rot after an additional 7d of inactivity and eventually close.

[fatedier]

The next version will support websocket over TLS, which will help expose frps in the K8s cluster through TLS + HTTP routing rules.

I have tested it successfully in istio-gateway.

[ajavalekar]

Hi @fatedier Thanks for this great tool! As per your above comment where we can expose through http on using wss, can you provide what should be the frps and frpc configuration to use?

[ajavalekar]

Hi @fatedier

Can you please help me with what configuration you used to expose frps in the K8s cluster through TLS + HTTP routing rules? I need to open ssh channels to multiple servers on demand behind an ingress controller and want to do it over http routing instead of tcp. Please can you help on this?