request help: how to use APISIX Ingress proxy SMTP

[jensg-st]

Issue description

We are trying to replace nginx ingress controller with apisix but we are experiencing an issue with TCP streaming. We want to proxy SMTP via apisix and the behaviour is different from nginx ingress controller. It might be a apure apisix question or nginx configuration thing.

With nginx we configured the helm chart like that as an example:

ingress-nginx:
  tcp:
    2525: default/smtp-listener:2525

This worked perfectly fine.

Now we are configuring the ingress controller in regards to TCP like this:

gateway:
  type: LoadBalancer
  tls:
    enabled: true
    sslProtocols: "TLSv1 TLSv1.1 TLSv1.2"
  stream:
    enabled: true
    tcp:
    - 2525

This opens port 2525 on the gateway side and on apisix. With a route we are adding the backend service.

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: tcp-route
spec:
  stream:
    - name: tcp-route-rule1
      protocol: TCP
      match:
        ingressPort: 2525
      backend:
        serviceName: smtp-listener
        servicePort: 2525

The problem here is the behaviour of the proxy itself I'm assuming. With nginx ingress controller or port-forwarding to the service we are getting the following:

Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 localhost ESMTP Service Ready

With apisix it seems the connection "hangs" till the first data or command is set:

Trying 192.168.0.1...
Connected to computer.local.
Escape character is '^]'.
AUTH LOGIN
220 localhost ESMTP Service Ready
502 5.5.1 Please introduce yourself first.

As you can see, the ESMTP Service Ready comes AFTER sending the first data to the service. Is there a configuration I need to add so the data is getting send immediately? This breaks every SMTP client which considers this service not as SMTP service.

Environment

• your apisix-ingress-controller version (output of apisix-ingress-controller version --long):

Version: 1.6.0
Git SHA: no-git-module
Go Version: go1.19.5
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64

• your Kubernetes cluster version (output of kubectl version):

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.7+k3s1", GitCommit:"7af5b16788afe9ce1718d7b75b35eafac7454705", GitTreeState:"clean", BuildDate:"2022-10-25T19:31:34Z", GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.7+k3s1", GitCommit:"7af5b16788afe9ce1718d7b75b35eafac7454705", GitTreeState:"clean", BuildDate:"2022-10-25T19:31:34Z", GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"}

• if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):

5.19.0-32-generic #33~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Jan 30 17:03:34 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

[tao12345666333]

I don't know exactly what's going on here.

Can you provide more information? For example, what is your server?

In addition, I suggest that you can use tcpdump to capture packets to check whether there is any difference when using APISIX and NGINX.

FYI:

• https://www.cisco.com/c/en/us/support/docs/security/email-security-appliance/118016-troubleshoot-esa-00.html
• https://learn.microsoft.com/en-us/exchange/mail-flow/test-smtp-telnet?view=exchserver-2019
• https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113423-asa-esmtp-smtp-inspection.html

[jensg-st]

Thank you for replying. I kind of figured out what the issue is/was. We have some problems with Ubuntu hosts and the centos7 image you are using (#1602) and we don't need Hong Kong timezone for our images.

Because of the DNS issue we had with the image we did build our own image based on Ubuntu 22.04. I don't know what the issue with that base image is but it had this TCP "hang". We decided now to use Rocky Linux as base image for the ingress controller because it is basically one of the successors of Centos. That works like a charm. Thanks again.

[ps-19]

Use the stream_timeout option to configure Apisix to send the SMTP Service response after the connection is established. _Default value of streamtimeout is 10 sec.

gateway:
  type: LoadBalancer
  tls:
    enabled: true
    sslProtocols: "TLSv1 TLSv1.1 TLSv1.2"
  stream:
    enabled: true
    tcp:
    - 2525
    timeout: 2