search

docker-flow / docker-flow-proxy

Docker Flow Proxy
https://docker-flow.github.io/docker-flow-proxy/
MIT License
317 stars 189 forks source link

docker-flow TCP service stops right away. #89

Closed rong0312 closed 2 years ago

rong0312 commented 5 years ago

Hey all, Based on the wiki, i would like to verify that this solution uses the ingress- routing mesh of the swarm?

SOURCE: The Flow Explained We'll go over the flow of a request to one of the services in the Swarm cluster. A user or a service sends a request to our DNS (e.g. acme.com). The request is usually HTTP on the port 80 or HTTPS on the port 443. DNS resolves the domain to one of the servers inside the cluster. We do not need to register all the nodes. A few is enough (more than one in the case of a failure). The Docker's routing mesh inspects which containers are running on a given port and re-sends the request to one of the instances of the proxy. It uses round robin load balancing so that all instances share the load (more or less) equally. The proxy network performs load balancing among all the instances of the destination service, and re-sends the request to one of them. A similar logic is used for the destination services. The proxy does not need to do load balancing. Docker networking does that for us. The only thing it needs is the name of the service and that both belong to the same network. As a result, there is no need to reconfigure the proxy every time a new release is made or when a service is scaled.

The issue: Iv noticed that the routing mesh is extremely slow and not built for production (any counter perspective is welcome. I found a bypass, attaching my service to the host's network:

smtp: image: namshi/smtp:latest networks: ports:

But in case routing mesh is a must here, i lose my solution (which i originally tried to bypass with this image- via HAproxy)

Any ideas? is the routing mesh really in use of the proxies?

Cheers, Ron

rong0312 commented 5 years ago

Just to save some time, this is my full yaml file:

version: '3.7' services:

proxy: image: dockerflow/docker-flow-proxy networks:

networks: proxy-net: driver: overlay

led to this log (proxy):

[WARNING] 006/143723 (101) : config : frontend 'services' has no 'bind' directive. Please declare it as a backend if this was intended. 2019/01/07 14:37:23 HAPRoxy: Proxy services started. 2019/01/07 14:37:23 HAPRoxy: Proxy tcpFE_25 started. 2019/01/07 14:37:23 HAPRoxy: Stopping frontend services in 0 ms. 2019/01/07 14:37:23 HAPRoxy: Stopping frontend tcpFE_25 in 0 ms. 2019/01/07 14:37:23 HAPRoxy: Proxy services stopped (FE: 0 conns, BE: 0 conns). 2019/01/07 14:37:23 HAPRoxy: Proxy tcpFE_25 stopped (FE: 0 conns, BE: 0 conns).

any ideas what went wrong?

thomasjpfan commented 5 years ago

Since the routing mesh is having performance issues, you can disable the routing mesh on DFP by: https://docs.docker.com/engine/swarm/ingress/#bypass-the-routing-mesh

This would send all request to a given node to the DFP running on that node. DFP would still route the request to your service, smtp, on the proxy-net overlay network.

rong0312 commented 5 years ago

Thank you for your fast reply @thomasjpfan , When attaching smtp service to the host's network, then port 25 will be allocated to smtp. How can i force the request go through the LB?

Anyhow, when i look at one of my proxy containers log:

stderr: [WARNING] 007/061722 (87) : config : frontend 'services' has no 'bind' directive. Please declare it as a backend if this was intended.

2019/01/08 06:17:22 HAPRoxy: Stopping frontend services in 0 ms. 2019/01/08 06:17:22 HAPRoxy: Stopping frontend tcpFE_25 in 0 ms. 2019/01/08 06:17:22 HAPRoxy: Proxy services stopped (FE: 0 conns, BE: 0 conns). 2019/01/08 06:17:22 HAPRoxy: Proxy tcpFE_25 stopped (FE: 0 conns, BE: 0 conns).

"bind" meant for HTTP request isn't it? anything i missed in my yaml file? any idea how to solve this issue?

For the sake of debugging, this is my yaml after host network attachment:

version: '3.7' services:

proxy: image: dockerflow/docker-flow-proxy networks:

networks: proxy-net: driver: overlay

I can connect to SMTP service on each host via port 25. not via LB (same error logs as above).

So, when i tried to add BIND_PORTS=25 & removing the port section from my smtp service, i managed to 'fix' the stderr msg from the log, but the service still stops!

2019/01/08 07:02:38 Validating configuration Configuration file is valid 2019/01/08 07:02:38 HAPRoxy: Proxy services started. 2019/01/08 07:02:38 HAPRoxy: Proxy tcpFE_25 started. 2019/01/08 07:02:38 HAPRoxy: Stopping frontend services in 0 ms. 2019/01/08 07:02:38 HAPRoxy: Stopping frontend tcpFE_25 in 0 ms. 2019/01/08 07:02:38 HAPRoxy: Proxy services stopped (FE: 0 conns, BE: 0 conns). 2019/01/08 07:02:38 HAPRoxy: Proxy tcpFE_25 stopped (FE: 0 conns, BE: 0 conns).

any ideas? Thanks alot!

thomasjpfan commented 5 years ago

There looks to be a bug with DFP, when only a TCP service is proxied. I will look into this.

rong0312 commented 5 years ago

Hey @thomasjpfan , is there something i can do in the meantime? In case that i use docker-flow with 2+ services, the solution will work (conceptually)?

thomasjpfan commented 5 years ago

The "simplest" solution, would be to add a normal HTTP service that does nothing. To confirm my hypothesis about the TCP bug, can you get the haproxy configuration and post it here?

rong0312 commented 5 years ago

Hey again, I tried to add 'hello-world' container:

services:
  proxy:
    image: dockerflow/docker-flow-proxy
    networks:
      - proxy-net
    ports:
      - 80:80
      - 443:443
    environment:
      - LISTENER_ADDRESS=swarm-listener
      - DEBUG=true
      - MODE=swarm
      - BIND_PORTS=8888
.
.
.
hello-world:
    image: strm/helloworld-http
    networks:
      - proxy-net
    deploy:
      replicas: 1
      labels:
        - com.df.notify=true
        - com.df.port=8888
        - com.df.srcPort=80
        - com.df.reqMode=http

However the looks seems to be the same:

smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:31:58 Starting HAProxy
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:31:58 Starting "Docker Flow: Proxy"
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:03 Got configuration from http://swarm-listener:8080.
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:03 Creating configuration for the service smtp_smtp
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:03 Reloading the proxy
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:03 Validating configuration
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | Configuration file is valid
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:03 HAPRoxy: Proxy services started.
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:03 HAPRoxy: Proxy tcpFE_25 started.
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:04 Proxy config was reloaded
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 Sending distribution request to http://10.0.4.15:8080/v1/docker-flow-proxy/reconfigure?distribute=false&port=25&reqMode=tcp&serviceName=smtp_smtp&srcPort=25
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 Creating configuration for the service smtp_smtp
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 Reloading the proxy
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 Validating configuration
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | Configuration file is valid
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 HAPRoxy: Proxy services started.
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 HAPRoxy: Proxy tcpFE_25 started.
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 HAPRoxy: Stopping frontend services in 0 ms.
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 HAPRoxy: Stopping frontend tcpFE_25 in 0 ms.
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 HAPRoxy: Proxy services stopped (FE: 0 conns, BE: 0 conns).
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 HAPRoxy: Proxy tcpFE_25 stopped (FE: 0 conns, BE: 0 conns).
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 Proxy config was reloaded
smtp_proxy.1.z44f6emjkblp@BE-SMTP1-DEV    | 2019/01/22 07:32:05 Sending distribution request to http://10.0.4.14:8080/v1/docker-flow-proxy/reconfigure?distribute=false&port=25&reqMode=tcp&serviceName=smtp_smtp&srcPort=25

PS: I cant see that my hello world is even recognised, am i right?

thomasjpfan commented 5 years ago

Open up port 8080 and query the haproxy configuration on DFP: https://proxy.dockerflow.com/usage/#config

Is hello-world listening on port 8888? Your labels for hello-world tells DFP to listening to com.df.srcPort=80, and direct it to hello-world at com.df.port=8888.

On a side node, if you want to set BIND_PORTS=8888 in DFP, DFP should have port 8888 open.

rong0312 commented 5 years ago

This hello-world needs port 80, my bad i made it the opposite way. I changed the config and went into the container to see HAproxy config:

frontend services
    mode http
    option  forwardfor

    option httplog
    log global
    bind *:80
    bind *:8888

frontend tcpFE_25
    bind *:25
    mode tcp
    option tcplog
    log global
    default_backend smtp_smtp-be25_0

which seems ok to me (?). how ever the logs remain the same.

This is my compose. Does it fix what you intended to explain to me? 


proxy:
    image: dockerflow/docker-flow-proxy
    networks:
      - proxy-net
    ports:
      - 80:80
      - 443:443
    environment:
      - LISTENER_ADDRESS=swarm-listener
      - DEBUG=true
      - MODE=swarm
      - BIND_PORTS=8888,80
    deploy:
      replicas: 2
      restart_policy:
        condition: on-failure
.
.
.

  hello-world:
    image: strm/helloworld-http
    networks:
      - proxy-net
    deploy:
      replicas: 1
      labels:
        - com.df.notify=true
        - com.df.port=80
        - com.df.srcPort=8888
        - com.df.reqMode=http
      restart_policy:
        condition: on-failure
rong0312 commented 5 years ago

And about that config query you asked me to do-

curl -i 172.72.1.111:80/v1/docker-flow-proxy/config
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html>
  <head>
    <!-- Bootstrap -->
    <link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
    <style>
      body {
        padding-top: 50px
      }
    </style>
  </head>
  <body>
  <div class="container">
      <div class="panel panel-warning">
        <div class="panel-heading">
          <h3 class="panel-title">Docker Flow Proxy: 503 Service Unavailable</h3>
        </div>
      <div class="panel-body">
        No server is available to handle this request.
      </div>
  </div>
</body>
</html>
thomasjpfan commented 5 years ago

BIND_PORTS configures haproxy to listen to those ports for HTTP requests. The docker service needs these ports open. The following snippet opens up the needed ports, including 8080.

proxy:
    image: dockerflow/docker-flow-proxy
    networks:
      - proxy-net
    ports:
      - 80:80
      - 443:443
      - 8888:8888
      - 8080:8080
      - 25:25
    environment:
      - LISTENER_ADDRESS=swarm-listener
      - DEBUG=true
      - BIND_PORTS=8888
    deploy:
      replicas: 2
      restart_policy:
        condition: on-failure
  ...
  hello-world:
    image: strm/helloworld-http
    networks:
      - proxy-net
    deploy:
      replicas: 1
      labels:
        - com.df.notify=true
        - com.df.port=80
        - com.df.srcPort=8888
        - com.df.reqMode=http
      restart_policy:
        condition: on-failure

With port 8080 open you can run the following to get the haproxy config.

curl -i 172.72.1.111:8080/v1/docker-flow-proxy/config
rong0312 commented 5 years ago

First of all, thanks alot for you replies as they help me.

Your snippet works! I managed to ssh to my smtp nodes via the proxy (saw that on DFP logs and SMTP logs). The weird part: when i removed that dummy service (hello-world) I could ssh to my smtp but with the following logs (DFP):

smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | 2019/01/23 07:22:16 HAPRoxy: Proxy services started.
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | 2019/01/23 07:22:16 HAPRoxy: Proxy tcpFE_25 started.
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | 2019/01/23 07:22:16 HAPRoxy: Stopping frontend services in 0 ms.
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | 2019/01/23 07:22:16 HAPRoxy: Stopping frontend tcpFE_25 in 0 ms.
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | 2019/01/23 07:22:16 HAPRoxy: Proxy services stopped (FE: 0 conns, BE: 0 conns).
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | 2019/01/23 07:22:16 HAPRoxy: Proxy tcpFE_25 stopped (FE: 0 conns, BE: 0 conns).
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | The configuration file is valid, but there still may be a misconfiguration somewhere that will give unexpected results, please verify:
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | stdout:
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    |
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | stderr:
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | [WARNING] 022/072216 (87) : config : frontend 'services' has no 'bind' directive. Please declare it as a backend if this was intended.
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    |
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    |
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | 2019/01/23 07:22:17 Proxy config was reloaded
smtp_proxy.1.kqkcevqhte5u@BE-SMTP1-DEV    | 2019/01/23 07:23:17 HAPRoxy: 10.255.0.4:50638 [23/Jan/2019:07:23:12.244] tcpFE_25 smtp_smtp-be25_0/smtp_smtp 1/0/5651 157 -- 1/1/0/0/0 0/0

Logs are missleading, I am about to check the functionality and efficiency of the proxy, will update shortly.

Thanks alot mate.

rong0312 commented 5 years ago

Ok so the solution seems to work very good now! there is just that log issue that is a-little bit confusing but i will use this solution for my SMTP needs. Thanks alot @thomasjpfan !!

thomasjpfan commented 5 years ago

Yes, removing the hello world service would result in an error. The original intention of DFP was to support HTTP and TCP support was added later. Support for only TCP services is on my bucket list of features to add to DFP.

lle0x commented 2 years ago

Dear @rong0312

If this issue is still relevant, please feel free to leave a comment here.

lle0x commented 2 years ago

Closed due to inactivity